From 9f0c99baf7e5fcb0327b2a70aa156ce3c8747daf Mon Sep 17 00:00:00 2001 From: said Date: Sat, 11 Jul 2026 16:46:59 +0100 Subject: [PATCH 01/30] add new wrapper plan checklist --- docs/maintainer/roadmap/index.md | 4 +- .../stage-boundary-enforcement-checklist.md | 449 --------- .../wrapper-plan-migration-checklist.md | 954 ++++++++++++++++++ 3 files changed, 956 insertions(+), 451 deletions(-) delete mode 100644 docs/maintainer/roadmap/stage-boundary-enforcement-checklist.md create mode 100644 docs/maintainer/roadmap/wrapper-plan-migration-checklist.md diff --git a/docs/maintainer/roadmap/index.md b/docs/maintainer/roadmap/index.md index 1228b6973..40f84e712 100644 --- a/docs/maintainer/roadmap/index.md +++ b/docs/maintainer/roadmap/index.md @@ -2,7 +2,7 @@ title: Roadmap audience: maintainers prerequisites: user language support, developer documentation -related: ../../user/language-support/planned-features.md, stage-boundary-enforcement-checklist.md, semantic-pyi-wrapper-checklist.md, native-array-handle-checklist.md, documentation-content-checklist.md +related: ../../user/language-support/planned-features.md, wrapper-plan-migration-checklist.md, semantic-pyi-wrapper-checklist.md, native-array-handle-checklist.md, documentation-content-checklist.md status: active-roadmap --- @@ -13,7 +13,7 @@ maintainers. Public support status remains in User documentation. ## Planned Features -- [Stage boundary enforcement checklist](stage-boundary-enforcement-checklist.md) +- [Wrapper plan migration checklist](wrapper-plan-migration-checklist.md) - [Semantic `.pyi` wrapper checklist](semantic-pyi-wrapper-checklist.md) - [Native array handle checklist](native-array-handle-checklist.md) - [Documentation content checklist](documentation-content-checklist.md) diff --git a/docs/maintainer/roadmap/stage-boundary-enforcement-checklist.md b/docs/maintainer/roadmap/stage-boundary-enforcement-checklist.md deleted file mode 100644 index af35b9b7b..000000000 --- a/docs/maintainer/roadmap/stage-boundary-enforcement-checklist.md +++ /dev/null @@ -1,449 +0,0 @@ ---- -title: Stage Boundary Enforcement Checklist -audience: maintainers -prerequisites: pipeline map, semantic IR, ownership policy -related: ../internal-architecture/pipeline-map.md, ../../user/reference/semantic-ir.md, semantic-pyi-wrapper-checklist.md, index.md -status: active-roadmap ---- - -# Stage Boundary Enforcement Checklist - -This checklist tracks the architectural work required to make every wrapper -pipeline transition explicit, one-way, and mechanically enforced. It is not -enough for the main build path to call functions in the intended order. Each -stage must accept only the preceding stage's completed representation, reject -invalid state without repairing it, and leave all earlier-stage decisions -unchanged. - -The target pipeline is: - -```text -parsed source model - -> semantic IR draft - -> policy-completed semantic IR - -> readiness-approved semantic IR - -> codegen IR - -> bridge IR - -> binding IR - -> emitted sources - -> compilation and link result -``` - -Stages are dependent but monotonic. A later stage may translate or validate the -preceding output. It must not reach backward into parser facts, secretly rerun -an earlier stage, infer a missing semantic decision, or mutate completed policy. - -## Incremental Implementation Protocol - -This checklist is the implementation prompt and the authoritative progress -record. Do not maintain a second copy of its requirements in a separate prompt. - -The normal request for continuing this work is: - -> Implement the next coherent group of unchecked items from the stage boundary -> enforcement checklist. - -A request may also name a phase or a smaller set of items when a particular -boundary should be handled first. - -For every implementation turn: - -1. Read `AGENTS.md` and this entire checklist before selecting work. -2. Inspect the live files and existing changes; do not assume an unchecked item - is still unimplemented or that a checked item is still correct. -3. Select the smallest dependency-closed group that produces a verifiable - architectural improvement. Do not select unrelated boxes merely because - they are nearby. -4. State which checklist items are in scope before editing. -5. Update the relevant maintained architecture or public contract docs before - executable code when behavior or ownership changes. -6. Implement the selected group completely across code, diagnostics, tests, - documentation, and call sites. Do not add compatibility paths for an old - stage API. -7. Run focused verification for the selected boundary, followed by every - repository-required static check for the files changed. -8. Mark an item complete only when its full acceptance criterion has direct - evidence. Leave partially implemented items unchecked and add a short note - identifying the remaining work instead of treating partial progress as - completion. -9. Under the completed item or phase, record the owning files, tests, and exact - verification evidence needed by the next maintainer to audit the claim. -10. Stop at a coherent stage boundary. Report the next dependency-ready group, - but do not begin it unless it was part of the stated scope. - -Each implementation summary must include the changed-stage breakdown required -by `AGENTS.md`, the checklist items completed, tests added or updated, focused -and static verification results, remaining unchecked dependencies, and any -unrelated pre-existing failure. The checklist is complete only when every -non-negotiable requirement and phase acceptance item has direct evidence. - -## Non-Negotiable Contract - -- [ ] Semantic draft, policy-completed IR, readiness-approved IR, and codegen IR - are mechanically distinguishable representations. A mutable metadata boolean - is not the stage boundary. -- [ ] Policy completion returns a new completed representation and does not - mutate the caller's semantic draft. -- [ ] Every semantic decision required by lowering, bridge generation, binding - generation, printing, or build integration is complete before `ir2ast.py`. -- [ ] Completed policy cannot be replaced through a public mutable metadata - dictionary. -- [ ] Readiness accepts only policy-completed IR, does not invoke policy - completion, and does not mutate its input. -- [ ] Successful readiness returns a distinct readiness-approved representation. -- [ ] Lowering accepts only readiness-approved IR and cannot invoke readiness or - policy completion. -- [ ] Bridge and binding generators consume codegen decisions and cannot inspect - parser models or semantic drafts. -- [ ] Printers emit the representation they receive and cannot invoke semantic - policy completion or readiness. -- [ ] High-level orchestration invokes every stage explicitly in the documented - order, with no hidden fallback or compatibility path. - -## Boundary Immutability Model - -The implementation rule is: - -```text -private mutable stage builder - -> validate the stage result - -> deeply freeze the stage output - -> hand the immutable output to the next stage -``` - -Immutability applies to values crossing stage boundaries, not to every local -object used while constructing them. Parsers, converters, lowerers, generators, -symbol tables, scopes, caches, and compiler-command builders may use mutation -inside their owning stage. Those mutable builders must remain private and must -never be accepted as another stage's input. - -- [ ] Parsed-project output is immutable before semantic conversion receives it. -- [ ] Semantic-draft output is immutable before policy completion receives it. -- [ ] Policy-completed output is deeply immutable before readiness receives it. -- [ ] Readiness-approved output is deeply immutable before lowering receives it. -- [ ] Codegen IR is frozen after lowering and before bridge generation. -- [ ] Bridge IR is frozen after bridge generation and before binding generation. -- [ ] Binding IR is frozen after binding generation and before printing. -- [ ] Native build plans and build results are immutable records; execution - state remains private to the compiler/build runner. -- [ ] Generated source payloads cross their boundary as immutable text and - immutable artifact descriptions. -- [ ] Runtime handles are explicitly outside the pipeline-freezing rule because - allocation, association, owner state, and `close()` state are intentionally - mutable at execution time. -- [ ] A frozen dataclass containing a mutable list, set, dictionary, metadata - dictionary, or mutable nested semantic object does not satisfy this contract. -- [ ] Boundary collections use tuples, frozensets, deeply copied read-only - mappings, or equivalently immutable structures. -- [ ] Mutable backing mappings are not retained or exposed after constructing a - read-only view. -- [ ] Completed policy is represented through typed read-only fields or a typed - immutable policy bundle rather than replaceable string-key metadata entries. -- [ ] Mutation of a draft, builder, or source metadata object after a transition - cannot change the frozen output already handed to the next stage. -- [ ] Stage types cannot be forged by setting a marker in a metadata dictionary; - construction flows through the owning transition function. -- [ ] Tests treat Python's normal supported API as the enforcement boundary; - deliberate `object.__setattr__`-style interpreter bypasses are not supported - mutation paths. - -## Phase 0 — Live Boundary Audit - -- [ ] Inventory every production caller of parsing, source-to-IR conversion, - policy completion, readiness, lowering, bridge generation, binding - generation, printing, and compilation. -- [ ] Record the input and output representation of every stage entrypoint. -- [ ] Inventory every mutation of semantic modules, declarations, semantic - types, metadata, policy decisions, export lists, and readiness blockers. -- [ ] Inventory every call to `complete_semantic_policies()` and classify it as - the explicit pipeline transition or a hidden stage invocation to remove. -- [ ] Inventory all `_raise_for_*` and blocker construction in `ir2ast.py` and - classify each check as semantic validity, policy validity, readiness/backend - support, or a genuine lowering invariant. -- [ ] Audit every bridge/binding branch that reads datatype, `intent`, rank, - shape, `is_alias`, memory handling, dotted-variable form, nullability, - storage, ownership, or policy fields. -- [ ] For each audited bridge/binding branch, record whether it is completed - policy dispatch, permitted backend-local mechanics, or policy inference that - must move upstream. -- [ ] Confirm the audit includes source-driven builds, semantic `.pyi` builds, - readiness-only inspection, `.pyi` emission, manifest replay, and Makefile - generation. - -## Phase 1 — Documented Stage-State Model - -- [ ] Update the maintained pipeline map with the exact stage-state types and - allowed transitions. -- [ ] Document which stage owns parser validity, semantic contract validity, - policy validity, readiness/backend support, lowering invariants, and - compilation failures. -- [ ] Document the distinction between semantic policy and backend-local emitted - helper storage. -- [ ] Document which transformations are allowed to be lossy, such as entry - export pruning, and which source representation remains available for - diagnostics. -- [ ] Document whether completed and readiness-approved representations are - immutable snapshots, wrappers around immutable data, or another design with - equivalent mechanical guarantees. -- [ ] Update source-navigation docs so contributors enter each concern through - its owning stage rather than a downstream generator. -- [ ] For every stage package, document what it owns, what it consumes, what it - returns, which internal builders may mutate, what it must never infer, and - which downstream package may import its public output. -- [ ] Document the runtime-handle exception so pipeline immutability is not - incorrectly applied to intentionally stateful native runtime objects. - -## Phase 2 — Explicit Stage Representations - -- [ ] Introduce a semantic draft representation produced by source-to-IR and - semantic `.pyi` conversion. -- [ ] Introduce a policy-completed representation that cannot be confused with - the draft type. -- [ ] Introduce a readiness-approved/wrappable representation that cannot be - constructed by merely setting metadata. -- [ ] Preserve a separate codegen representation for lowering output. -- [ ] Introduce distinct bridge and binding handoff representations when those - stages currently share a mutable model that either stage can rewrite. -- [ ] Give each stage representation a narrow public construction path owned by - its transition function; keep mutable builders private to the stage package. -- [ ] Replace mutable policy metadata storage with an immutable completed policy - bundle or an equivalently sealed representation. -- [ ] Make nested collections and policy maps immutable enough that downstream - code cannot replace a decision indirectly. -- [ ] Ensure mutation of the original draft after completion cannot affect the - completed representation. -- [ ] Remove `POLICY_COMPLETION_PREPARED_METADATA` if the new type makes it - redundant, or limit it to serialized diagnostic provenance rather than using - it as authority. -- [ ] Do not add aliases, coercions, adapters, or compatibility wrappers that - allow an old mutable `SemanticModule` to bypass the new boundary. - -The intended transition API should be equivalent to this shape, although exact -names may follow the settled package design: - -```python -def build_semantic_draft(parsed: ParsedProject) -> SemanticDraft: ... - -def complete_policies(draft: SemanticDraft) -> PolicyCompletedIR: ... - -def validate_readiness(completed: PolicyCompletedIR) -> WrappableSemanticIR: ... - -def lower_to_codegen(wrappable: WrappableSemanticIR) -> CodegenIR: ... - -def generate_bridge(codegen: CodegenIR) -> BridgeIR: ... - -def generate_binding(bridge: BridgeIR) -> BindingIR: ... -``` - -- [ ] Each transition rejects every other stage representation rather than - coercing it or running a missing earlier transition. -- [ ] Each transition returns a new object and leaves its input observably - unchanged. -- [ ] Stage results expose immutable diagnostic/source provenance without - retaining a mutable reference to the preceding builder. -- [ ] Equality or stable fingerprints allow tests to prove that validation and - downstream generation did not mutate an earlier result. - -## Phase 3 — Post-IR Policy Completion - -Policy completion must own all semantic choices needed downstream, including -choices currently reconstructed from raw facts during lowering or codegen. - -- [ ] Complete object kind for every argument, result, field, module variable, - callback boundary, class instance, and hidden native value. -- [ ] Complete ownership, transfer, destruction, borrowed state, target owner, - and release responsibility. -- [ ] Complete mutability, native mutation, writeback, assignment mode, and - replacement behavior. -- [ ] Complete nullability and distinguish omitted arguments from explicit - present-but-null descriptor values. -- [ ] Complete output projection and hidden/identity/copy result behavior. -- [ ] Complete contract-value and boundary storage modes (`stack`, `heap`, or - `alias`). -- [ ] Complete Python barrier action and native barrier action. -- [ ] Complete getter behavior, native setter assignment, and Python setter - exposure for every field and module variable. -- [ ] Complete descriptor/data-buffer array interoperability and all required ABI - selector facts. -- [ ] Complete pass-by-value, pass-by-address, and call-local address behavior. -- [ ] Complete native array handle kind, ownership, operations, extraction, - descriptor interop, and build requirements. -- [ ] Complete callback argument/result ownership and barrier decisions. -- [ ] Complete entry export reachability before ownership decisions for the - retained declarations. -- [ ] Reject attempts to run policy completion on an already-completed - representation. -- [ ] Ensure policy completion produces path-aware blockers for missing or - contradictory facts rather than inserting a downstream fallback. - -## Phase 4 — Readiness As A Mandatory Gate - -- [ ] Make the prepared readiness API accept only policy-completed IR. -- [ ] Remove automatic policy completion from readiness APIs. -- [ ] Make readiness validation non-mutating. -- [ ] Move semantic support and backend-capability blockers out of `ir2ast.py` - into readiness. -- [ ] Keep policy contradictions in policy completion rather than readiness. -- [ ] Keep only genuine target-AST representability invariants in lowering. -- [ ] Return a distinct readiness-approved representation only when there are no - blockers. -- [ ] Preserve a structured blocker report when readiness fails. -- [ ] Require both source and semantic `.pyi` wrapper builds to pass readiness - before lowering. -- [ ] Require manifest replay and Makefile generation to use the same readiness - gate as direct builds. - -## Phase 5 — Mechanical IR-To-AST Lowering - -- [ ] Change `semantic_ir_to_codegen_ast()` to accept only readiness-approved - semantic IR. -- [ ] Add one recursive entry validator that verifies every required completed - policy category before visiting any declaration. -- [ ] Replace optional `.get(...)` access for required getter, setter, result, - class, array-handle, storage, barrier, and interoperability decisions with - required typed access. -- [ ] Move pass-by-value selection from parser-origin inspection into completed - policy. -- [ ] Move descriptor versus data-buffer interoperability selection into - completed policy. -- [ ] Audit array category, source shape, target/addressability, optionality, - projection, and layout conversion so only mechanical representation lowering - remains. -- [ ] Remove readiness decisions and unsupported-contract policy from lowering. -- [ ] Prove lowering does not mutate the readiness-approved input. -- [ ] Preserve backend-local creation of scopes, names, temporaries, imports, - statements, and expressions. - -## Phase 6 — Bridge And Binding Dispatch - -- [ ] Ensure bridge and binding modules cannot import or call - `OwnershipPolicyResolver`, `default_ownership_policy`, policy completion, or - readiness. -- [ ] Route semantic behavior through explicit dispatchers keyed by completed - codegen actions or typed completed policy selectors. -- [ ] Reject missing dispatcher combinations without choosing a default. -- [ ] Remove policy inference based on datatype, `intent`, rank/shape alone, - `is_alias`, local memory handling, dotted variables, parser origin, or missing - policy. -- [ ] Limit branches inside selected implementation methods to emitted-code - mechanics. -- [ ] Represent backend-local helper temporary storage separately from semantic - `OwnershipDecision` so local implementation details cannot be mistaken for - contract policy. -- [ ] Prove bridge generation does not mutate or replace completed policy carried - by codegen IR. -- [ ] Prove binding generation does not mutate or replace completed policy - carried by bridge/codegen IR. -- [ ] Keep low-level printers free of ownership-aware behavior selection. - -## Phase 7 — Printers And Build Orchestration - -- [ ] Remove hidden policy completion from `emit_module_stubs()` and other - printer entrypoints. -- [ ] Make `.pyi` emission orchestration explicitly select the required semantic - stage before invoking the printer. -- [ ] Make direct source builds visibly call parse, semantic conversion, policy - completion, readiness, lowering, bridge/binding generation, and compilation - in order. -- [ ] Make semantic `.pyi` builds visibly call `.pyi` parsing/conversion, native - contract validation, policy completion, readiness, lowering, bridge/binding - generation, and compilation in order. -- [ ] Ensure inspection-only CLI stages stop at their declared representation - and do not mutate it for a later report in the same command. -- [ ] Ensure native array build requirements consume completed/readiness-approved - policy rather than reconstructing descriptor needs. -- [ ] Remove legacy entrypoints or permissive stage coercions instead of keeping - compatibility paths. - -## Phase 8 — Mechanical Architecture Tests - -- [ ] Raw semantic draft is rejected by prepared-readiness entrypoints. -- [ ] Raw semantic draft is rejected by lowering. -- [ ] Policy-completed but readiness-unvalidated IR is rejected by lowering. -- [ ] Removing each required policy category is detected by the recursive stage - validator before lowering starts. -- [ ] Missing getter, setter, return, class-instance, class-self, array-handle, - storage, barrier, and interoperability decisions cannot become `None`. -- [ ] Policy completion does not mutate its input draft. -- [ ] Mutating the draft after completion cannot affect completed IR. -- [ ] Readiness does not mutate completed IR or any completed decision. -- [ ] Lowering does not mutate readiness-approved IR or any completed decision. -- [ ] Bridge generation does not mutate or recompute policy. -- [ ] Binding generation does not mutate or recompute policy. -- [ ] Policy completion rejects already-completed input. -- [ ] Wrapper build orchestration invokes stages in the exact documented order. -- [ ] Every stage transition returns a different object from its input and the - input retains the same stable fingerprint after the call. -- [ ] Parsed-project, semantic-draft, completed, wrappable, codegen, bridge, - binding, build-plan, and build-result boundary collections reject ordinary - supported mutation operations. -- [ ] Frozen stage results contain no reachable mutable list, set, dictionary, - mutable metadata dictionary, or mutable semantic child object. -- [ ] Mutable parser, semantic, lowering, bridge, binding, and build builders are - not exported from their package's public stage API. -- [ ] Structural AST tests reject prohibited resolver/completion imports from - lowering, bridge, binding, and printer packages. -- [ ] Structural AST tests reject prohibited raw-fact policy inference in bridge - and binding code. -- [ ] Structural tests allow narrowly identified backend-local helper planning - without allowing semantic policy construction in generators. -- [ ] Structural dependency tests enforce the permitted package direction: - parsing cannot import semantics; semantics cannot import lowering/codegen; - readiness cannot import lowering; lowering cannot import bridge/binding; - bridge/binding cannot import parsers or semantic policy resolvers; printers - cannot invoke earlier stage transitions. -- [ ] Structural dependency tests allow the high-level pipeline orchestrator to - import and compose stage entrypoints without making orchestration policy - authority. -- [ ] Runtime-handle tests continue to prove intentional allocation, - association, ownership, and close-state mutation despite immutable pipeline - artifacts. -- [ ] Missing dispatcher combinations fail explicitly. -- [ ] Export pruning remains before readiness/lowering, and omitted declarations - never reach codegen. -- [ ] Semantic `.pyi` round-tripping preserves the editable contract. -- [ ] Source, generated-contract, and modified-contract runtime behavior remains - unchanged unless an explicitly documented blocker moves earlier. - -## Phase 9 — Focused Evidence - -- [ ] `tests/semantics/policy/` covers immutable completed - policy, complete recursive decision sets, and strict dispatcher behavior. -- [ ] `tests/semantics/readiness/` covers the completed-to- - validated transition and non-mutating readiness. -- [ ] `tests/lowering/test_semantic_ir.py` covers validated-only lowering and absence - of policy/readiness inference. -- [ ] `tests/codegen/printers/` covers printer-only emission without - hidden policy completion. -- [ ] `tests/parsing/pyi/` covers semantic draft construction and - contract round-tripping. -- [ ] `tests/wrapper/fortran/edit_pyi_contracts/` proves edited policy remains - authoritative through runtime behavior. -- [ ] `tests/architecture/test_dependency_boundaries.py` enforces - dependency and inference restrictions mechanically. -- [ ] Source and semantic `.pyi` build-mode tests prove the mandatory stage - sequence. -- [ ] Focused wrapper tests cover scalar, string, array, descriptor, derived - type, callback, module-variable, and optional-argument boundaries. -- [ ] LAPACK remains excluded from local verification unless separately - authorized. - -## Phase 10 — Verification And Completion Record - -- [ ] Focused semantic, `.pyi`, codegen-structure, build-mode, and wrapper tests - pass. -- [ ] `python3 -m ruff check .` passes. -- [ ] `python3 -m ruff format --check .` passes. -- [ ] Bandit passes with the repository configuration. -- [ ] Vulture passes. -- [ ] The blocking Radon policy passes with an explicit base fallback when local - CI SHA variables are unavailable. -- [ ] Full Radon complexity and maintainability reports are run and recorded as - advisory output. -- [ ] `git diff --check` passes. -- [ ] No compatibility shim, fallback path, or legacy stage entrypoint remains. -- [ ] The final implementation report lists every blocker moved, every stage - representation introduced, every downstream inference removed, and every - remaining limitation. -- [ ] This checklist is moved from active work to completed evidence only after - every requirement above has direct test or structural evidence. diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md new file mode 100644 index 000000000..f0ab91dbc --- /dev/null +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -0,0 +1,954 @@ +--- +title: Wrapper Plan Migration Checklist +audience: maintainers +prerequisites: pipeline map, semantic IR, ownership policy +related: ../internal-architecture/pipeline-map.md, ../../user/reference/semantic-ir.md, semantic-pyi-wrapper-checklist.md, index.md +status: active-roadmap +--- + +# Wrapper Plan Migration Checklist + +This checklist replaces the broad "freeze every stage" idea with a smaller, +more useful target: build wrappers from an explicit, readable wrapper plan +instead of sending semantic IR through the current generic lowering/codegen +model first. + +This file is the canonical implementation prompt for the migration. Route +granularity, plan structure, supported-lane definitions, validation rules, +emitter ownership, and cutover state must be recorded here before the related +code is changed. A phase heading is not by itself an implementation-ready +specification; the expansion rules below apply to the broader later phases. + +The long-term target is: + +```text +semantic IR + -> post-IR policy completion + -> wrapper-generation route selection + -> wrapper-plan route + -> build and validate wrapper plan + -> binding emitter + bridge emitter + -> complete generated wrapper artifact set + -> temporary legacy route + -> semantic_ir_to_codegen_ast() + -> existing bridge/binding generation + -> complete generated wrapper artifact set + -> shared compilation and link orchestration + -> importable extension +``` + +The route selects the complete runtime-wrapper generation path. It does not +select only argument lowering, one function, the binding, or the bridge. Both +routes must produce everything the shared build orchestration needs for one +extension: generated bridge source, generated C/CPython binding source and +headers, module initialization, required imports/includes/runtime support, +generated source compile requirements, and the names and files passed to the +existing compilation/link stage. + +Semantic `.pyi` generation and printing are not part of this route and remain +on their current path. The wrapper plan may consume semantic IR loaded from a +`.pyi` contract, but it does not replace or alter `.pyi` emission. + +The wrapper plan is the contract between the Python binding, the generated +native bridge, and the native call. A maintainer should be able to open the plan +for one wrapper function and see: + +- the Python-visible arguments and result order, +- decorator effects such as `@native_call`, `@bind`, `@raises`, overload + metadata, hidden native literals, and reordered native arguments, +- the binding action selected for each Python argument/result, +- the bridge handoff produced by binding and consumed by bridge, +- the bridge action selected for the native call, +- copy-in, copy-out, writeback, cleanup, and result projection phases, +- the exact dispatch key that selects each binding and bridge implementation + method. + +The plan should be simple enough for maintainers to read and transform, but +complete enough that invalid edits fail before any C or Fortran source is +emitted. + +## Migration Source Of Truth + +This is a representation and ownership migration, not a wrapper-behavior +redesign. The current production path through `semantic_ir_to_codegen_ast()`, +the existing bridge and binding generators, the source printers, and the build +orchestration is the behavioral source of truth for each lane until that lane +has completed parity evidence. + +- Derive each new plan action and handler from an audited current code path and + its wrapper tests. Do not invent a new conversion, ABI, call order, cleanup + order, public/generated-symbol naming contract, runtime helper protocol, + error behavior, or build artifact layout merely because the new + representation could support one. +- Preserve settled semantic/public contracts and previously documented bug + fixes when they are stricter than accidental current implementation behavior. + Any other intentional behavior change is separate work: document and test it + independently before changing the migration baseline. +- Copy the minimum dependency-closed set of existing CPython and NumPy API + models, C and Fortran declaration/statement nodes, datatype/literal models, + scope/name mechanisms, and source-printer behavior needed by each lane into + the isolated new generator. These are second independent copies, not imports, + aliases, subclasses, or adapters around legacy model classes. Copy the current + behavior first, record its origin and parity evidence, and modify only the new + copy. Do not make the new route import legacy codegen primitives. +- Shared policy-completed semantic input, stable runtime APIs, the generated- + artifact handoff, and compilation/link orchestration may remain common. If a + new route needs changed runtime behavior, add a separately named helper used + only by that route until cutover; do not change the meaning of a helper still + used by the legacy generator. +- Keep the legacy route runnable and behaviorally unchanged while a lane is + introduced. Production route selection is explicit, and tests must be able to + invoke both routes for an eligible generation unit without a public + compatibility flag. +- A lane is not migrated merely because the new route compiles. It must match + the existing route's Python-visible results, native argument/result order, + mutation and writeback, exceptions, ownership/lifetime behavior, generated + artifact requirements, and success/failure cleanup behavior. +- Generated source text does not need to be byte-for-byte identical when the + same behavior and ABI are preserved. Backend-local names and equivalent + mechanical control flow may differ. Material differences must be explained by + the new mechanical organization, not by an unplanned semantic change. +- Until final cutover, reverting a lane means deliberately routing its whole + generation unit through the still-maintained legacy route. It does not mean + catching a new-route failure and silently retrying legacy generation. + +## Route Selection Contract + +The initial migration route is atomic for one extension generation unit. In +the current build orchestration that unit is the merged semantic module used to +build one importable extension. + +- Route selection runs after post-IR policy completion and before + `semantic_ir_to_codegen_ast()`. +- The support check recursively inspects every runtime-visible or + runtime-required element in the merged module: functions, arguments, + results, hidden native arguments/results, decorator effects, module + variables, classes, constructors, properties, methods, overload dispatch, + cleanup, and build requirements. +- The wrapper-plan route is selected only when every such element is covered by + implemented plan actions, validators, binding handlers, and bridge handlers. +- No accepted decorator, native-call projection kind, or implicit call behavior + may be ignored by the support check. It must be implemented by a completed + lane or reported as an unsupported owner path that selects the legacy route. +- If any element is not covered, the whole generation unit uses the temporary + legacy route. Initially, one extension must not mix plan-generated functions + with legacy-generated functions, or a plan-generated binding with a + legacy-generated bridge. +- The route decision returns a structured support report with the selected + route and owner paths/reasons for every unsupported element. It must be + inspectable in tests and maintainer diagnostics. +- Unsupported migration coverage may select the legacy route. Invalid or + inconsistent completed policy must fail before route selection; an error in + plan construction, validation, emission, or compilation after the plan route + is selected must fail the build and must not silently retry the legacy route. +- Source-driven and semantic-`.pyi`-driven extension builds use the same route + selector and the same wrapper-plan builder after producing the merged, + policy-completed semantic module. + +Function-level or mixed-route generation can be considered only after the +module-level migration is complete and only if a concrete need justifies its +extra composition and validation rules. It is not part of this checklist. + +## Design Rules + +- Post-IR policy completion decides semantic actions. Binding and bridge emitters + consume those actions; they do not reconstruct them from datatype, `intent`, + `is_alias`, local memory handling, dotted-variable shape, or missing policy. +- The wrapper plan records action keys, handoff expectations, and native-call + ordering. It does not contain raw generated C, Fortran, or CPython text. +- Binding and bridge emitters are dispatch tables plus small implementation + methods. Reuse completed policy actions directly when they already identify + the behavior, such as `PythonBarrierAction.SCALAR_VALUE` and + `NativeBarrierAction.PASS_VALUE`; do not create a parallel action vocabulary + merely to rename them in the plan. +- Each action has exactly one registered handler for its emitter. Validation and + plan rendering resolve that registry to the exact method name; emitters do not + dynamically construct method names or run a second policy-selection tree. +- Initial copied handlers keep the current dispatch method names when practical, + so a plan action can be traced directly to the audited legacy method. Add a + new plan-only action only when no completed policy action expresses the + required semantic step; first confirm that the missing distinction does not + belong in post-IR policy completion. +- Dispatch should be by semantic lane and action, not by every concrete dtype. + For example, `Float64`, `Int32`, and `Bool` can share scalar-value handlers + while dtype remains plan data. +- Every binding output that the bridge consumes is represented by a handoff + spec. Every bridge native argument/result that the native call consumes is + represented by a native-call spec. +- Validation checks producer/consumer consistency before emission: + `binding.produces` must match `bridge.expects`, bridge output must match the + native call slot, and writeback/result plans must consume values that were + actually produced. +- Local variable names may be generated by a plan context, but the plan should + carry stable symbolic roles such as `value`, `descriptor`, `shape`, `status`, + `message`, or `writeback`. +- The old lowering/codegen path may remain temporarily for unsupported lanes, + but the route must be explicit and tracked by lane and owner path. Do not hide + a semantic fallback inside binding, bridge, or printer code. +- Plan construction and validation are separate from source emission. Emitters + may create backend-local names and helper temporaries, but they may not add, + remove, reorder, or reinterpret semantic actions from the validated plan. +- Backend-specific resource mechanics do not belong in the wrapper plan. + CPython borrowed/new/stolen-reference rules, `Py_INCREF`/`Py_DECREF`, and + partial-failure reference cleanup are private binding-emitter mechanics. + Equivalent compiler/runtime details remain private to their backend. +- A plan may record backend-neutral lifetime relationships such as owned, + borrowed, transferred, retained owner, destruction responsibility, or + call-scoped cleanup. The selected backend method mechanically maps those + relationships and the target API's contract to its local resource handling; + doing so is implementation, not a new semantic decision. +- Isolation begins after post-IR policy completion. The new generator must not + import `x2py.codegen`, and legacy `x2py.codegen` modules must not import the + new generator. High-level pipeline orchestration is the only owner allowed to + select between them. +- Copy backend primitives incrementally by lane, not as a wholesale clone of + the legacy codegen package. Each copied slice includes only the dependency + closure needed to generate and print that lane, so its representation can + evolve without changing the legacy route. +- Maintainer edits use an explicit immutable plan transformation between plan + construction and validation. The transformed plan is validated exactly like + a generated plan; there is no trusted-edit bypass. +- The binding and bridge emitters both consume the same validated plan. Their + generated runtime data flow is Python binding -> bridge -> native call, but + one emitter is not the semantic-policy input to the other. + +## Intended Ownership And Files + +The exact names may evolve during Phase 0, but responsibilities should remain +separated along these lines: + +```text +x2py/wrapper_codegen/ + scope.py copied scope and deterministic name-allocation behavior + types.py copied datatype/literal primitives needed by migrated lanes + + plan/ + models.py frozen readable plan records + actions.py action, handoff, and native-pass enums/specs + build.py completed semantic IR -> WrapperPlan + validate.py cross-boundary consistency validation + support.py whole-generation-unit route support report + render.py stable maintainer-readable plan rendering + + c/ + nodes.py copied C declarations/statements needed by migrated lanes + cpython_api.py copied CPython API primitives needed by migrated lanes + numpy_api.py copied NumPy C API primitives needed by migrated lanes + concepts.py copied C/CPython helper concepts needed by migrated lanes + context.py new-route local values and cleanup state + binding.py validated plan -> complete CPython binding representation + printer.py isolated C/CPython source emission + + fortran/ + nodes.py copied Fortran declarations/statements needed by lanes + context.py new-route Fortran names and scopes + bridge.py validated plan -> complete native bridge representation + printer.py isolated Fortran source emission + + artifacts.py complete new-route generated-wrapper artifact result + +x2py/pipeline/ + shared route selection and compile/link orchestration +``` + +This is a permanent responsibility-based package name, not a temporary +`new_codegen` package that would need another migration after cutover. Files +under `c/` and `fortran/` are added only as a migrated lane needs them; the +layout does not authorize copying the entire legacy package up front. + +Within the generation boundary there is no shared model identity between the +routes. If a migrated scalar handler needs a legacy `Variable`, datatype, +literal, CPython call model, NumPy call model, scope, or printer-dispatch base, +copy that item's complete required dependency slice into +`x2py.wrapper_codegen` first. The copied class may then evolve independently +without changing or importing its legacy counterpart. + +`ir2ast.py` remains the entry to the temporary legacy route; it must not choose +the route or partially invoke the new emitters. The source-driven and `.pyi`- +driven build entrypoints should call one shared orchestration helper rather +than implement different support rules. + +The two routes should return one pipeline-owned internal generated-artifact +result shape so the existing compiler/link orchestration does not need to +understand plan actions. That result is an internal build handoff, not a +compatibility API and not a second semantic model. + +The isolated C and Fortran layers are thin mechanical backend layers. For each +primitive, first copy the proven legacy behavior and tests needed by the lane; +then simplify or modify the new copy only as required by plan-driven emission. +The original node, printer, scope, and generator implementations remain +untouched and runnable. Shared compilation receives generated files through the +artifact result and does not import either route's internal models. + +## Node Construction And Printing Contract + +Emitters construct backend nodes; printers render backend nodes. Do not mix +these responsibilities. + +```text +validated WrapperPlan + -> Fortran bridge emitter + -> isolated Fortran module/function/declaration/statement nodes + -> isolated Fortran printer + -> bridge source text + -> CPython binding emitter + -> isolated C module/function/declaration/statement nodes + -> isolated C source/header printers + -> binding source and header text + -> artifact assembler + -> complete generated-wrapper artifact result +``` + +- Both emitters consume the same validated wrapper plan and its explicit bridge + ABI specification. The binding emitter must not consume the generated + Fortran AST, and the bridge emitter must not discover information that the C + binding needs. If either occurs, the shared plan or ABI specification is + incomplete. +- Each dispatched emitter method returns backend-node fragments such as + declarations, setup statements, call arguments, result statements, + success/failure cleanup, and produced symbolic values. A module assembler + combines those fragments into a complete C or Fortran module node. +- Emitters do not concatenate source text. CPython conversion and reference- + counting operations are represented by isolated C/API call nodes; Fortran + declarations, assignments, calls, and control flow use isolated Fortran + nodes. +- Printers accept only their backend nodes. They own syntax, indentation, + punctuation, fixed syntax templates, and mechanical rendering of represented + includes/imports. They must not accept `WrapperPlan`, inspect plan actions, + choose conversions, add lifecycle behavior, or repair incomplete modules. +- Includes, imports, public/generated symbols, function signatures, and header + declarations are selected by plan-driven emission and represented as nodes + before printing. A printer may deduplicate or order them mechanically. +- C source/header and Fortran source printers are independently testable against + the copied baseline nodes before plan emitters use them. +- `plan/` does not import `c/` or `fortran/`; the C and Fortran backends do not + import each other; and backend printers import their nodes/types but not plan + builders, actions, validators, emitters, or pipeline routing. Add structural + tests for these internal dependency directions with the package skeleton. + +## Core Plan Shape + +The exact class names can evolve, but the first implementation should stay close +to this shape: + +```python +@dataclass(frozen=True) +class WrapperPlan: + extension_name: str + module: ModulePlan + requirements: WrapperArtifactRequirements + + +@dataclass(frozen=True) +class ModulePlan: + public_name: str + functions: tuple[FunctionPlan, ...] + variables: tuple[VariablePlan, ...] + classes: tuple[ClassPlan, ...] + + +@dataclass(frozen=True) +class FunctionPlan: + public_name: str + native_name: str + decorators: DecoratorPlan + python_arguments: tuple[ArgumentPlan, ...] + bridge_abi: BridgeAbiPlan + native_call: NativeCallPlan + results: tuple[ResultPlan, ...] + writebacks: tuple[WritebackPlan, ...] + + +@dataclass(frozen=True) +class ArgumentPlan: + public_name: str + semantic_type: object + python_position: int | None + binding: BindingStep + bridge: BridgeStep + native: NativeArgumentSpec + writeback: WritebackPlan | None = None + + +@dataclass(frozen=True) +class BindingStep: + action: PythonBarrierAction + produces: tuple[HandoffSpec, ...] + + +@dataclass(frozen=True) +class BridgeStep: + action: NativeBarrierAction + expects: tuple[HandoffSpec, ...] + produces: tuple[NativeArgumentSpec, ...] + + +@dataclass(frozen=True) +class NativeCallPlan: + native_name: str + arguments: tuple[NativeArgumentRef, ...] + results: tuple[NativeResultRef, ...] +``` + +The plan validator owns consistency diagnostics. Binding and bridge emitters +should be able to trust a validated plan and focus on emitted-code mechanics. + +The first implementation also needs two non-semantic orchestration records: + +- a route support report naming the generation unit, selected route, covered + lanes, and unsupported owner paths/reasons; +- a generated wrapper artifact result naming the complete bridge/binding + sources, headers, imports/includes/runtime requirements, generated source + compilation requirements, and extension initialization name. + +These records must not duplicate the native object/library/link plan already +owned by build orchestration. + +## Worked Scalar Trace + +For a Python-visible scalar procedure equivalent to: + +```python +def f(x: Float64) -> None: ... +``` + +assume post-IR policy completion produces the existing actions +`PythonBarrierAction.SCALAR_VALUE` and +`NativeBarrierAction.PASS_VALUE`. The maintainer-visible plan rendering should +stay approximately this small: + +```text +function f(x: Float64) -> None + argument x + binding action scalar_value + binding handler CPythonBindingEmitter._convert_python_scalar_value_argument + produces x.value : Float64 + bridge ABI f_bridge.x consumes x.value + bridge action pass_value + bridge handler FortranBridgeEmitter._convert_native_value_argument + native slot f argument 0 <- x.value + result none +``` + +The exact top-level call path should be equally direct: + +```text +complete_semantic_policies(module) existing semantic stage +build_wrapper_plan(module) completed policy -> WrapperPlan +validate_wrapper_plan(plan) handoff/ABI/handler validation +generate_wrapper_artifacts(plan) + CPythonBindingEmitter.emit_function(f) + -> _convert_python_scalar_value_argument(x) + -> isolated C node fragments + FortranBridgeEmitter.emit_function(f) + -> _convert_native_value_argument(x) + -> isolated Fortran node fragments + CPythonCodePrinter.doprint(c_module) nodes -> C source/header + FCodePrinter.doprint(fortran_module) nodes -> Fortran source +create_shared_library(...) existing compilation/link entrypoint +``` + +At runtime, the generated CPython binding converts the Python argument into the +scalar C handoff, calls the generated bridge symbol, and the bridge invokes the +native procedure using the completed `PASS_VALUE` behavior. CPython reference +counting, concrete temporary names, C declarations, Fortran declarations, and +printer formatting are deliberately absent from the rendered plan. + +A scalar plan that requires a maintainer to inspect backend nodes or printer +code to discover either selected handler has failed the readability goal. The +rendered plan is the normal trace; backend nodes and printers are inspected only +when debugging how a selected handler emits source. + +## Decorator And Native-Projection Coverage + +Decorator and projection handling is part of route support, not an emitter +detail. Maintain a matrix in this section as implementation proceeds. Each row +must eventually name its exact plan representation, validation rules, binding +handler, bridge handler, and focused tests. + +| Contract effect | Owning phase | Initial route rule | +| --- | --- | --- | +| Direct scalar call with implicit native order | Phase 1 | Eligible after scalar input actions are complete | +| `@bind(...)` native symbol selection | Phase 1 | Legacy route until symbol selection is explicit in `NativeCallPlan` | +| `@external` native target selection | Phase 1 | Legacy route until target/source-language requirements are explicit | +| `@hold_gil` call behavior | Phase 1 | Legacy route until GIL behavior is an explicit binding call phase | +| `@native_call` scalar `Arg(...)` reordering and `Addr(Arg(...))` | Phase 1 | Legacy route until every native slot and address handoff validates | +| `@native_call` typed numeric/logical hidden literals | Phase 1 | Legacy route until literal type, value, and native slot validate | +| `@native_call` scalar `Return(...)`, `Work(...)`, and direct native result projection | Phase 2 | Legacy route until result/workspace production and consumption validate | +| `@raises(...)` status/message projection | Phase 2 | Legacy route until status, message, success rule, and Python error path validate | +| Optionality and `IsPresent(...)` | Phase 3 | Legacy route until omitted, explicit `None`, present, and presence-token paths validate | +| String `Len(...)` and typed string literals | Phase 5 | Legacy route until the expanded string sub-lanes are complete | +| Array shape/stride/size/itemsize and conversion projections | Phase 6 | Legacy route until the expanded ordinary-array sub-lanes are complete | +| `Allocatable(...)` and `Pointer(...)` native projections | Phase 7 | Legacy route until the expanded descriptor/handle sub-lanes are complete | +| Derived/native type metadata | Phases 8-9 | Legacy route until the relevant derived-type and class sub-lanes are complete | +| `Pass()`, methods, constructors, properties, and `@overload(...)` | Phase 9 | Legacy route until the expanded class sub-lanes are complete | +| Callback decorators, adapters, and trampoline behavior | Phase 10 | Legacy route until the expanded callback sub-lanes are complete | + +When the live parser or semantic model accepts an effect missing from this +matrix, add it before implementing or routing that case. Do not treat the table +as proof that every current syntax spelling has already been audited; Phase 0 +owns that live inventory. + +## Incremental Protocol + +For each lane: + +1. Audit the current lowering, binding, bridge, printer, runtime-helper, and + build paths for the lane, and record the observed behavior and focused tests + that make it the migration baseline. +2. Expand this checklist with the lane's exact scope, exclusions, source-path + baseline, copied dependencies, plan fields, and validation invariants. +3. Copy the minimum dependency-closed legacy backend primitives required by the + lane into `x2py.wrapper_codegen`, prove copied baseline behavior, then add + or adapt isolated node/printer tests. +4. Implement the plan objects, action registries, ABI/handoff specs, validator, + and support-report coverage for that lane. +5. Generate the plan from policy-completed semantic IR, dispatch binding and + bridge handlers into isolated node fragments, assemble complete backend + modules, and print complete internal artifacts. +6. Compile the internal artifacts before changing production route selection. +7. Run the same eligible fixtures through both routes and compare compiled + runtime behavior, failure paths, native-call mapping, and artifact + requirements. +8. Extend the whole-module support predicate so a generation unit uses the + wrapper-plan route only when all its elements belong to completed lanes. + Keep the old route for generation units containing unsupported lanes. +9. Mark the lane complete only when focused parity tests pass and every + intentional difference from the baseline is separately documented. + +Do not start a later lane by guessing. Each lane must define the handoff specs +and consistency checks it needs. + +## Mandatory Expansion Gate For Broad Phases + +Phases 5 through 10 are roadmap envelopes, not complete implementation +checklists. Before implementation starts on one of them, update this file and +split that phase into dependency-ordered sub-lanes. The expansion must be based +on an audit of the live semantic models, completed policies, existing +bridge/binding behavior, decorators, and focused wrapper tests. + +Each expanded sub-lane must state: + +- the exact included and excluded semantic cases; +- the completed-policy fields it consumes and any decisions that still need to + move into post-IR policy completion; +- plan records, action keys, handoff specs, native-call slots, lifecycle phases, + and required generated artifacts; +- binding and bridge handler names and which backend-local helper values they + may create; +- validation invariants across Python input/result, binding handoff, bridge + handoff, native call, writeback, cleanup, ownership, and release; +- the whole-module support-predicate change that makes the sub-lane eligible; +- focused generation/runtime tests and parity evidence against the legacy + route; +- the exact legacy source path for every copied primitive, its dependency + closure, baseline evidence, modifications in the isolated copy, and a reason + for every entirely new backend primitive; +- dependencies on earlier lanes and the legacy behavior that can be removed + when the sub-lane is complete. + +Do not mark a broad phase complete from its current envelope items. Mark its +expanded sub-lanes individually, then close the phase only after all live cases +in its audited support matrix are either migrated or explicitly removed from +the product contract. + +## Required Execution Order + +The checklist order is mandatory. Do not wire the new route into production +build selection while its scalar artifacts exist only as models or uncompiled +source. Complete each subphase and its focused evidence before starting the +next one: + +```text +0A current-behavior baseline + -> 0B isolated package and dependency boundary + -> 0C copied scalar backend nodes and printers + -> 0D wrapper-plan core and validation + -> 1A internal scalar plan emission + -> 1B compiled dual-route parity + -> 1C production route selection + -> later semantic lanes in numbered order +``` + +Within later lanes, follow the same order: audit current behavior, expand the +lane checklist, copy required backend dependencies, add plan/actions and +validators, emit nodes, print internally, compile and compare both routes, and +only then widen production route eligibility. + +## Phase 0 — Foundation Before Production Routing + +### Phase 0A — Current Scalar Baseline + +- [ ] Inventory the current end-to-end wrapper behavior and implementation paths + for the first scalar lane, including lowering branches, binding/bridge helper + methods, CPython/NumPy API primitives, source printers, generated artifacts, + build integration, and focused runtime fixtures. +- [ ] Create a maintained baseline matrix mapping each first-lane current code + path and observable behavior to its proposed plan action, handler, copied + backend dependencies, and parity evidence. Do not define an action from a + hypothetical implementation. +- [ ] Audit every decorator, native-call projection kind, implicit call + behavior, and generated module/class feature accepted by the live semantic + model; reconcile the coverage matrix with that audit. +- [ ] Confirm the legacy generator's independent entrypoint and baseline tests; + do not modify legacy lowering, nodes, generators, or printers in this phase. +- [ ] Update the maintained wrapper-plan contract with the audited Python + surface, binding handoff, bridge ABI, native call, result, cleanup, writeback, + node-construction, and printing phases. + +### Phase 0B — Isolated Package Boundary + +- [ ] Create the `x2py.wrapper_codegen` package skeleton without connecting it + to production build selection. +- [ ] Define and enforce the package boundary: `x2py.wrapper_codegen` cannot + import `x2py.codegen`, legacy `x2py.codegen` cannot import + `x2py.wrapper_codegen`, and only pipeline orchestration may eventually import + both route entrypoints. +- [ ] Add dependency tests for that boundary before copied backend code is + introduced. +- [ ] Define the pipeline-owned generated-wrapper artifact result shared by both + routes without duplicating native object/library/link-plan ownership. +- [ ] Keep runtime helper APIs shared only when their behavior is unchanged. Add + separately named new-route helpers when different behavior is required, and + record their generated callers and cleanup contract. +- [ ] Document that CPython reference counting and API ownership conventions are + binding-emitter-local mechanics and are absent from plan models, rendered + plans, and cross-backend plan validation. + +### Phase 0C — Copied Scalar Backend Foundation + +- [ ] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, + datatype/literal models, CPython and NumPy API primitives, scopes/naming + behavior, helper concepts, and printer behavior required for Phase 1. Record + each legacy source path and baseline test before copying it. +- [ ] Copy those primitives into `x2py.wrapper_codegen` without importing, + aliasing, subclassing, or adapting legacy model classes. +- [ ] Keep each initial copy behaviorally equivalent to its legacy source before + modifying it for plan emission. +- [ ] Add isolated node/printer tests proving representative scalar C source, + C headers, and Fortran source render equivalently to the baseline. +- [ ] Verify the copied C/Fortran printers consume only isolated backend nodes + and cannot import or inspect wrapper-plan models. + +### Phase 0D — Wrapper Plan Core + +- [ ] Define the first frozen plan data classes and tuple collections. +- [ ] Define `BridgeAbiPlan`, native-call refs, handoff specs, handler + registries, and validation errors around existing completed + `PythonBarrierAction` and `NativeBarrierAction` values. Do not add duplicate + plan action enums for behavior already represented by completed policy. +- [ ] Add a plan validator that catches mismatched binding/bridge handoffs, + missing bridge/native-call slots, unknown action handlers, duplicate symbolic + roles, and writebacks/results that consume unavailable values. +- [ ] Define the whole-generation-unit support report, including stable + owner-path reasons for unsupported elements, without changing production + route selection yet. +- [ ] Define a small immutable plan-transformation API so maintainers can alter + actions, ordering, or handoffs before validation without mutating the + policy-completed semantic IR. +- [ ] Add deterministic plan rendering that includes symbolic owner paths, + dispatch handler names, handoffs, bridge ABI slots, native slots, and + lifecycle order without backend nodes or CPython-specific mechanics. +- [ ] Verify generated and maintainer-transformed plans pass through the same + validator and produce owner-path diagnostics before node emission. + +## Phase 1 — Scalar Function Inputs + +Scope: free functions with scalar numeric/logical arguments that are +Python-visible inputs. Scalar call-target decorators and scalar native-call +argument projections are included as separate checklist items; unsupported +projection kinds keep the whole module on the legacy route. + +### Phase 1A — Internal Plan Emission + +- [ ] Generate plans for scalar value arguments such as `f(x: Float64)`. +- [ ] Represent Python argument position and `@native_call` native argument order + explicitly, including reordered arguments. +- [ ] Represent implicit native order, `@bind(...)`, `@external`, and + `@hold_gil` explicitly; none may be inferred or ignored by the emitters. +- [ ] Represent `Arg(...)`, `Addr(Arg(...))`, and typed numeric/logical hidden + literals as native argument sources with exact native positions. +- [ ] Reject duplicate, missing, or out-of-range Python/native positions and + bridge/native-call slots during plan validation. +- [ ] Add binding actions for scalar Python object to scalar value/storage. +- [ ] Add bridge actions for scalar pass-by-value, pass-by-address, and + call-local address where already supported by completed policy. +- [ ] Validate binding-produced scalar handoffs against bridge expectations and + the shared `BridgeAbiPlan`. +- [ ] Make each scalar binding/bridge handler return isolated node fragments; + assemble complete C and Fortran module nodes outside individual handlers. +- [ ] Print complete scalar-only bridge source, C/CPython binding source/header, + module initialization, and generated-source requirements through the isolated + printers and artifact assembler. + +### Phase 1B — Internal Compilation And Parity + +- [ ] Provide test-only orchestration that sends the same eligible module + directly through legacy and wrapper-plan routes without a public + compatibility option or production selector change. +- [ ] Compile and import new-route scalar artifacts through the shared compiler + and linker before making any production module eligible. +- [ ] Compare both routes for Python calls/results, native argument order, + pass-by-value/address behavior, conversion failures, exception state, + backend-local cleanup, generated artifact requirements, compilation, import, + and runtime behavior. +- [ ] Resolve every unexplained parity difference or document a separately + approved behavior correction before proceeding. + +### Phase 1C — Production Route Integration + +- [ ] Add one shared pipeline selector used by source-driven and semantic-`.pyi`- + driven builds after policy completion and before any legacy + `semantic_ir_to_codegen_ast()` call. +- [ ] Select the wrapper-plan route only for complete generation units whose + recursively inspected elements are all covered by completed scalar-input + actions, validators, emitters, printers, and parity evidence. +- [ ] Route a generation unit containing any unsupported element entirely + through the existing path. +- [ ] Add route-selector tests proving one module cannot mix plan and legacy + functions, bindings, bridges, nodes, or printers. +- [ ] Prove plan construction, validation, emission, printing, compilation, or + linking failures do not silently retry the legacy route. +- [ ] Keep explicit internal selection of the legacy route available for + rollback and dual-route tests. + +## Phase 2 — Scalar Results And Hidden Outputs + +Scope: scalar direct returns, hidden scalar outputs, and scalar projected +results. + +- [ ] Audit and record the legacy result, hidden-output, result-packaging, + `@raises`, cleanup, printer, and runtime paths that define this lane's + baseline. +- [ ] Copy and baseline-test the additional result variables, CPython creation + calls, C/Fortran statements, and printer behavior required by this lane. +- [ ] Represent direct native return, hidden output, identity output, and + projected result lanes in `ResultPlan`. +- [ ] Add bridge actions for scalar result assignment and hidden scalar output + storage without relying on `is_alias`. +- [ ] Add binding actions for scalar Python result creation. +- [ ] Validate that every Python result consumes a native result or writeback + that the bridge produces. +- [ ] Cover `@raises` status/message outputs so runtime-status validation is + represented in the plan, not rediscovered in binding. +- [ ] Emit and print complete result-capable C/Fortran modules internally, then + compile and compare both routes for values, status/error paths, cleanup, ABI, + and artifact requirements. +- [ ] Widen whole-module route eligibility to scalar results only after that + parity evidence passes. + +## Phase 3 — Scalar Inout, Optional, And Descriptor-Like Scalars + +Scope: scalar copy-in/copy-out, optional arguments, present-but-null descriptor +values, and scalar allocatable/pointer descriptor boundaries. + +- [ ] Audit and record the legacy copy-in/out, optional presence, nullable + scalar descriptor, cleanup, and failure-path behavior for this lane. +- [ ] Copy and baseline-test the additional optional/descriptor nodes, API + primitives, local-state helpers, and printer behavior required by this lane. +- [ ] Represent copy-in, native mutation, copy-out, and cleanup as explicit + writeback phases. +- [ ] Preserve the three-state optional rule: omitted argument, explicit `None`, + and present concrete value are distinct when the native ABI needs them. +- [ ] Represent scalar descriptor presence tokens and nullable value handoffs in + the plan. +- [ ] Validate that a writeback consumes an existing binding/bridge handoff and + writes to a Python-visible target or result slot. +- [ ] Emit and print complete inout/optional/descriptor-capable modules + internally, then compile and compare both routes for all three presence + states, mutation, writeback, cleanup, ABI, and failures. +- [ ] Widen whole-module route eligibility to this lane only after parity, and + complete it before moving arrays or handles to the plan path. + +## Phase 4 — Scalar Module Variables + +Scope: scalar module variables. Derived-type fields remain in Phases 8 and 9 +because their wrapper instance, owner, and property lifecycle must already be +represented before field access can use the plan route. + +- [ ] Audit and record the legacy scalar module-variable getter, setter, + rejected replacement, module initialization, and attribute-routing behavior. +- [ ] Copy and baseline-test the additional module/type nodes, getter/setter API + primitives, initialization nodes, and printer behavior required by this lane. +- [ ] Represent getter behavior, setter exposure, native setter assignment, and + rejected replacement behavior in module-variable plans. +- [ ] Add binding actions for Python attribute get/set around scalar values. +- [ ] Add bridge actions for scalar module-variable read/write. +- [ ] Validate getter/setter pair consistency: a Python setter cannot exist + without a compatible bridge setter handoff. +- [ ] Keep ordinary Python module-name rebinding semantics separate from native + module-variable storage. +- [ ] Emit and print complete module-variable-capable modules internally, then + compile and compare both routes for get/set behavior, rejection paths, + initialization, cleanup, ABI, and generated artifacts. +- [ ] Widen whole-module route eligibility to scalar module variables only after + that parity evidence passes. + +## Phase 5 — Strings + +Scope: scalar character values, fixed-length strings, deferred-length strings, +and string copy/writeback. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate, separating at least value/storage, fixed/deferred length, + input/result/inout, optionality, and ownership/lifetime cases found in the + live contract. + +- [ ] Define string handoff specs for value strings, storage strings, fixed + length, deferred length, and mutable buffers. +- [ ] Add binding and bridge actions for string value input, string storage + input, string result, and string writeback. +- [ ] Validate length/source expectations before emission. +- [ ] Keep string behavior separate from numeric scalar behavior even when both + are rank-zero. + +## Phase 6 — Ordinary Arrays + +Scope: NumPy data-buffer arrays that do not require native descriptor handles. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate, separating at least input/result/inout, hidden outputs, rank/shape + forms, dtype families, C/Fortran order and striding, optionality, copy versus + view behavior, and writeability cases found in the live contract. + +- [ ] Define array handoff specs for data pointer, dtype, rank, shape, strides, + contiguity/order, itemsize, and writeability. +- [ ] Add binding actions for NumPy validation and data-buffer extraction. +- [ ] Add bridge actions for array data-buffer passing and shape/stride + forwarding. +- [ ] Represent hidden array outputs and array copy-out/writeback explicitly. +- [ ] Validate dtype/rank/shape/order expectations in the plan before emitted C + checks are generated. + +## Phase 7 — Native Array Handles And Descriptors + +Scope: `Allocatable[T[...]]`, `Pointer[T[...]]`, descriptor-backed handoffs, and +runtime native array handle objects. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate and reconcile it with the maintained native-array-handle checklist. + Split allocatable and pointer behavior only after extracting their shared + descriptor, presence, ownership, release, module/field, argument, and result + sub-lanes. + +- [ ] Define descriptor handoff specs for CFI descriptors, descriptor ownership, + optional-absent handles, owner retention, extraction policy, and required + headers. +- [ ] Move bridge-created `ArrayInteropPolicy` decisions into plan generation or + completed policy for source/contract values. +- [ ] Replace bridge-created semantic `OwnershipDecision` values for generated + helper temporaries with explicit bridge-local helper specs. +- [ ] Validate descriptor/data-buffer mismatches before emission. +- [ ] Keep generated helper storage local to bridge/binding implementation + methods; do not represent helper temporaries as semantic ownership policy. + +## Phase 8 — Derived Types And Snapshots + +Scope: opaque derived-type wrappers, borrowed views, wrapper-owned instances, +and snapshot copies. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate, separating origin, owned/borrowed/aliased/snapshot lifetime, + input/result/field/module-state use, destruction, owner retention, and + recursive member cases found in the live contract. + +- [ ] Define derived-type handoff specs for wrapper address, owned instance, + borrowed instance, and snapshot copy. +- [ ] Represent destruction/release responsibility in the result/writeback plan. +- [ ] Add binding and bridge actions for derived-type input, result, field + access, and snapshot creation. +- [ ] Represent scalar and later non-scalar field getter/setter behavior only + after the owning derived wrapper and class lifecycle are available. +- [ ] Validate owner-retention and release expectations before emission. + +## Phase 9 — Classes, Constructors, Properties, And Methods + +Scope: generated Python classes, keyword constructors, explicit constructor +bindings, properties, methods, static methods, overloads, and direct +`@bind(func_name)` constructor cases. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate. Inventory class creation and destruction, constructor categories, + instance/static/type-bound methods, properties, overloads, inheritance or + type relationships, decorator effects, and module initialization needs + before defining the sub-lanes. + +- [ ] Represent class layout, constructor candidates, explicit constructor + bindings, default constructor behavior, and property/method plans. +- [ ] Keep `.pyi` constructor text, runtime constructor behavior, and bridge + calls aligned through the same class plan. +- [ ] Validate that direct constructor bindings are not confused with overload + dispatch. +- [ ] Validate property setter/getter exposure against bridge handoffs. + +## Phase 10 — Callbacks And Trampolines + +Scope: callback argument conversion, callback result conversion, adapter +procedures, C trampolines, callback context setup/cleanup, and error/abort +paths. + +- [ ] Before implementation, expand this phase under the mandatory expansion + gate. First inventory callback signature categories, context lifetime, + re-entry/GIL behavior, exception propagation, abort paths, recursion, and the + scalar/string/array/derived argument-result combinations actually supported + by the product contract. + +- [ ] Define callback handoff specs for Python callable validation, callback + context, native adapter arguments, trampoline arguments, and callback results. +- [ ] Represent callback setup and cleanup as call-scoped plan phases. +- [ ] Add binding/bridge actions for scalar, array, string, and derived callback + arguments/results only after those lanes are stable for ordinary calls. +- [ ] Validate callback result and argument handoffs across binding, bridge, + adapter, and trampoline steps before emission. + +## Phase 11 — Cutover And Removal + +- [ ] Track which lanes still use the old `semantic_ir_to_codegen_ast()` path. +- [ ] Track route support at whole-generation-unit granularity and keep + unsupported owner-path diagnostics stable until the corresponding lane is + migrated. +- [ ] Keep completed legacy lanes available for deliberate rollback until all + live lanes have parity evidence and the final cutover is approved; do not + delete old handlers incrementally merely because one fixture uses the plan + route. +- [ ] Do not move modified isolated nodes or printers back into the legacy + package during migration. After final cutover, remove the legacy package + pieces proven unused and keep `x2py.wrapper_codegen` as the canonical + generator rather than performing a second package rename. +- [ ] Remove old lowering/codegen code only after every live wrapper lane has a + wrapper-plan route and focused verification. +- [ ] Remove the temporary legacy route and its route diagnostics after every + live generation unit is supported; do not replace it with compatibility + shims or per-function fallback. +- [ ] Keep source printers only for the remaining generated source fragments they + still own, or replace them with narrower emitters once the model layer is no + longer needed. + +## Verification + +- [ ] Documentation-only changes run + `python3 -m pytest -q tests/docs/test_examples.py tests/docs/test_structure.py` + and `git diff --check`. +- [ ] Wrapper-plan code changes run focused plan validation tests, focused + wrapper generation tests, and the required static-analysis suite from + `AGENTS.md`. +- [ ] Runtime wrapper tests are required when generated behavior changes. +- [ ] Every migrated lane runs eligible fixtures through both routes. Compare + behavior and ABI-relevant call mapping; do not require byte-identical source + when mechanical organization differs. +- [ ] Structural dependency tests prove complete generator isolation: no + imports from `x2py.wrapper_codegen` to `x2py.codegen` or in the reverse + direction. +- [ ] LAPACK remains excluded from local verification unless separately + authorized. + +## Completion Record + +- [ ] The final report for each lane names the plan actions added, the binding + and bridge handlers they dispatch to, and the handoff specs validated. +- [ ] The final report lists old lowering/codegen paths still used by unsupported + lanes. +- [ ] The final report includes focused verification commands and results. +- [ ] The final report includes the changed-stage breakdown required by + `AGENTS.md` and names every test file added or updated with the behavior it + covers. From 948e91544702ccd034a82115df3b712a4b606c87 Mon Sep 17 00:00:00 2001 From: said Date: Sat, 11 Jul 2026 20:25:37 +0100 Subject: [PATCH 02/30] add the migration plan --- .github/workflows/quality.yml | 84 +- docs/developer/quality-assurance.md | 31 +- docs/developer/testing-strategy.md | 12 +- .../wrapper-plan-migration-checklist.md | 939 +++++++++++++++--- tests/README.md | 7 +- 5 files changed, 808 insertions(+), 265 deletions(-) diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index cce04b4ff..bcc575b83 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -23,7 +23,6 @@ on: env: X2PY_GFORTRAN_BINARY: gfortran-13 X2PY_GFORTRAN_PACKAGE: gfortran-13 - X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH: .pytest_cache/x2py/real-library-native jobs: static-analysis: @@ -66,67 +65,10 @@ jobs: continue-on-error: true run: python -m radon mi c_parser fortran_parser semantics x2py -s - real-library-native-cache: - name: Real-Library Native Cache - if: ${{ !inputs.static_analysis_only }} - needs: static-analysis - runs-on: ubuntu-24.04 - permissions: - contents: read - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 2 - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.12" - - name: Install test dependencies - run: | - python -m pip install --upgrade pip - python -m pip install -e ".[qa]" - - name: Install pinned GFortran - shell: bash - run: | - if ! command -v "$X2PY_GFORTRAN_BINARY" >/dev/null 2>&1; then - sudo apt-get update - sudo apt-get install --yes "$X2PY_GFORTRAN_PACKAGE" - fi - compiler_dir="$RUNNER_TEMP/x2py-gfortran" - mkdir -p "$compiler_dir" - ln -sf "$(command -v "$X2PY_GFORTRAN_BINARY")" "$compiler_dir/gfortran" - echo "$compiler_dir" >> "$GITHUB_PATH" - "$compiler_dir/gfortran" --version - - name: Capture native cache key facts - id: native-cache-facts - shell: bash - run: | - echo "gfortran_hash=$(gfortran --version | head -n 1 | sha256sum | cut -d' ' -f1)" >> "$GITHUB_OUTPUT" - - name: Restore real-library native cache - id: restore-real-library-native-cache - uses: actions/cache/restore@v4 - with: - path: ${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH }} - key: real-library-native-${{ runner.os }}-${{ runner.arch }}-${{ steps.native-cache-facts.outputs.gfortran_hash }}-${{ hashFiles('tests/data/fortran/blas/**', 'tests/data/fortran/lapack/**', 'tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py', 'tools/warm_real_library_native_cache.py') }} - restore-keys: | - real-library-native-${{ runner.os }}-${{ runner.arch }}-${{ steps.native-cache-facts.outputs.gfortran_hash }}- - - name: Warm real-library native cache - env: - PYTHONPATH: . - X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR: ${{ github.workspace }}/${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH }} - run: python tools/warm_real_library_native_cache.py - - name: Save real-library native cache - if: steps.restore-real-library-native-cache.outputs.cache-hit != 'true' - uses: actions/cache/save@v4 - with: - path: ${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH }} - key: real-library-native-${{ runner.os }}-${{ runner.arch }}-${{ steps.native-cache-facts.outputs.gfortran_hash }}-${{ hashFiles('tests/data/fortran/blas/**', 'tests/data/fortran/lapack/**', 'tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py', 'tools/warm_real_library_native_cache.py') }} - test: name: Tests (Python ${{ matrix.python-version }}) if: ${{ !inputs.static_analysis_only }} - needs: [static-analysis, real-library-native-cache] + needs: static-analysis runs-on: ubuntu-24.04 permissions: contents: read @@ -159,26 +101,17 @@ jobs: ln -sf "$(command -v "$X2PY_GFORTRAN_BINARY")" "$compiler_dir/gfortran" echo "$compiler_dir" >> "$GITHUB_PATH" "$compiler_dir/gfortran" --version - - name: Capture native cache key facts - id: native-cache-facts - shell: bash - run: | - echo "gfortran_hash=$(gfortran --version | head -n 1 | sha256sum | cut -d' ' -f1)" >> "$GITHUB_OUTPUT" - - name: Restore real-library native cache - uses: actions/cache/restore@v4 - with: - path: ${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH }} - key: real-library-native-${{ runner.os }}-${{ runner.arch }}-${{ steps.native-cache-facts.outputs.gfortran_hash }}-${{ hashFiles('tests/data/fortran/blas/**', 'tests/data/fortran/lapack/**', 'tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py', 'tools/warm_real_library_native_cache.py') }} - restore-keys: | - real-library-native-${{ runner.os }}-${{ runner.arch }}-${{ steps.native-cache-facts.outputs.gfortran_hash }}- - name: Run tests shell: bash env: PYTHONPATH: . HYPOTHESIS_PROFILE: ci X2PY_COVERAGE_REQUESTED: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-coverage')) || (github.event_name == 'workflow_call' && inputs.coverage) }} - X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR: ${{ github.workspace }}/${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_PATH }} run: | + pytest_args=( + --ignore=tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py + ) + if [[ "${{ matrix.python-version }}" == "3.12" ]]; then test_paths=(tests) else @@ -212,7 +145,6 @@ jobs: tests/wrapper/fortran/runtime_behavior tests/wrapper/fortran/scalars tests/wrapper/fortran/strings - "tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_library[blas]" tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py ) fi @@ -221,11 +153,13 @@ jobs: COVERAGE_PROCESS_START="${{ github.workspace }}/pyproject.toml" \ python -m coverage run -m pytest -q --randomly-seed=1 \ -o junit_family=legacy \ - --junitxml="$RUNNER_TEMP/pytest-results.xml" "${test_paths[@]}" + --junitxml="$RUNNER_TEMP/pytest-results.xml" \ + "${pytest_args[@]}" "${test_paths[@]}" else python -m pytest -q --randomly-seed=1 \ -o junit_family=legacy \ - --junitxml="$RUNNER_TEMP/pytest-results.xml" "${test_paths[@]}" + --junitxml="$RUNNER_TEMP/pytest-results.xml" \ + "${pytest_args[@]}" "${test_paths[@]}" fi - name: Upload coverage data if: matrix.python-version == '3.12' && ((github.event_name == 'push' && github.ref == 'refs/heads/main') || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-coverage')) || (github.event_name == 'workflow_call' && inputs.coverage)) diff --git a/docs/developer/quality-assurance.md b/docs/developer/quality-assurance.md index c1fda151c..fa649931c 100644 --- a/docs/developer/quality-assurance.md +++ b/docs/developer/quality-assurance.md @@ -67,11 +67,11 @@ python -m coverage report For subprocess coverage investigations, mirror that command shape before deciding a fix. A plain local coverage run can miss subprocess data. -GitHub Actions runs ordinary PR tests without coverage overhead. Python 3.10 -and 3.11 run the regular suite, BLAS real-library wrapper test, and native -bundle tests; the full LAPACK real-library wrapper test runs only on Python -3.12. Pushes to `main` always run the Python 3.12 test job under coverage and -publish the coverage report. Add the `run-coverage` PR label, or pass +GitHub Actions runs ordinary PR tests without coverage overhead. During the +wrapper-plan migration, every Python version excludes the full BLAS/LAPACK +real-library wrapper test while retaining general native-bundle coverage. +Pushes to `main` always run the remaining Python 3.12 test job under coverage +and publish the coverage report. Add the `run-coverage` PR label, or pass `coverage: true` to the reusable workflow, to request the same coverage gate outside the main branch. @@ -228,21 +228,12 @@ reports advisory/manual. issues, Ruff formatting drift, Vulture unused test parameters, and the too-strict Radon policy. -**Native artifact cache:** the Quality workflow pins the test runner to -`ubuntu-24.04`, installs `gfortran-13`, and warms -`.pytest_cache/x2py/real-library-native` in a dedicated pre-matrix job. The -Python matrix restores that exact cache before pytest and sets -`X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR` to the restored path. Requested coverage -runs collect Python 3.12 coverage data; a final coverage job combines that -artifact and uploads the XML report. This cache holds the full BLAS/LAPACK -object files, archives, and shared libraries used by the real-library wrapper -tests. Cache keys include the runner OS, runner -architecture, pinned `gfortran` version, BLAS/LAPACK source content, and native -cache helper code. Native object files are not portable across different -platforms, compilers, compiler flags, or source revisions; a key change -intentionally rebuilds them. Cold object builds compile independent sources in -parallel after required module sources; set `X2PY_REAL_LIBRARY_NATIVE_JOBS` to -override the bounded worker count. +**Native artifact cache:** the full BLAS/LAPACK native-cache preparation is +disabled with the deferred real-library wrapper test during wrapper-plan +migration. Restore the cache job and matrix environment only when Phase 12 of +the migration checklist explicitly re-enables both corpora. Requested coverage +runs still collect Python 3.12 coverage data; a final coverage job combines +that artifact and uploads the XML report. **Failure reporting:** each pytest matrix invocation writes `pytest-results.xml`; the final failure-only step runs diff --git a/docs/developer/testing-strategy.md b/docs/developer/testing-strategy.md index cb39f9e3b..27837cc51 100644 --- a/docs/developer/testing-strategy.md +++ b/docs/developer/testing-strategy.md @@ -58,14 +58,18 @@ modes execute one shared behavioral assertion body. Modified-contract behavior stays in the same feature subject but uses its intentionally different assertions. -Run all Fortran wrapper subjects except real-library runtime work with: +During the wrapper-plan migration, run all Fortran wrapper subjects except the +deferred full BLAS/LAPACK corpus with: ```bash -python3 -m pytest -q tests/wrapper/fortran --ignore=tests/wrapper/fortran/real_libraries +python3 -m pytest -q tests/wrapper/fortran \ + --ignore=tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py ``` -Do not run LAPACK runtime tests locally unless the task explicitly requests -them. BLAS-only evidence may be selected separately when relevant. +Do not run the full BLAS or LAPACK real-library wrapper tests locally or in +GitHub Actions during migration. Re-enable both only after every other +wrapper-plan migration row is complete. General native-bundle tests remain +active. ## Fixtures and generated expectations diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index f0ab91dbc..2a78cbfdc 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -26,7 +26,8 @@ semantic IR -> post-IR policy completion -> wrapper-generation route selection -> wrapper-plan route - -> build and validate wrapper plan + -> mechanically project completed policy into wrapper plan + -> validate wrapper plan structure and handoffs -> binding emitter + bridge emitter -> complete generated wrapper artifact set -> temporary legacy route @@ -63,9 +64,12 @@ for one wrapper function and see: - the exact dispatch key that selects each binding and bridge implementation method. -The plan should be simple enough for maintainers to read and transform, but -complete enough that invalid edits fail before any C or Fortran source is -emitted. +The plan should be simple enough for maintainers to read and reproduce by hand. +The supported way to change generated behavior is to change the semantic +contract or completed policy and rebuild the plan, not to patch the plan or add +a backend exception. Validation must fail before any C or Fortran source is +emitted when a completed policy or its mechanically derived plan is +inconsistent. ## Migration Source Of Truth @@ -84,13 +88,24 @@ has completed parity evidence. fixes when they are stricter than accidental current implementation behavior. Any other intentional behavior change is separate work: document and test it independently before changing the migration baseline. -- Copy the minimum dependency-closed set of existing CPython and NumPy API - models, C and Fortran declaration/statement nodes, datatype/literal models, - scope/name mechanisms, and source-printer behavior needed by each lane into - the isolated new generator. These are second independent copies, not imports, - aliases, subclasses, or adapters around legacy model classes. Copy the current - behavior first, record its origin and parity evidence, and modify only the new - copy. Do not make the new route import legacy codegen primitives. +- Recreate the minimum dependency-closed behavior needed by each lane inside the + isolated generator. For each CPython/NumPy API model, C/Fortran node, + datatype/literal, naming mechanism, or printer case, either copy a small + already-suitable implementation or rewrite a smaller class containing only + fields and methods the new emitters/printers use. These are independent + implementations, not imports, aliases, subclasses, or adapters around legacy + model classes. +- The legacy implementation is the behavioral source, not the required class + design. Record the legacy origin, consumed fields, emitted source behavior, + and parity evidence before simplifying. Do not preserve unused constructors, + properties, inheritance, mutation APIs, lookup categories, or future-facing + fields merely because the legacy class has them. +- Reuse the smallest coherent legacy method-body snippets and helper logic when + they remain straightforward after switching to the validated plan and + isolated context. When the legacy method depends on unrelated class state or + abstractions, rewrite a smaller direct handler from the recorded behavior + instead of copying that structure. Preserve observable call order, error + handling, cleanup, and generated-call behavior either way. - Shared policy-completed semantic input, stable runtime APIs, the generated- artifact handoff, and compilation/link orchestration may remain common. If a new route needs changed runtime behavior, add a separately named helper used @@ -112,6 +127,43 @@ has completed parity evidence. generation unit through the still-maintained legacy route. It does not mean catching a new-route failure and silently retrying legacy generation. +### Legacy Replay Procedure + +The working legacy pipeline makes this a reproduction exercise rather than a +greenfield generator project. Use it as an executable oracle every time a test +or semantic lane is migrated. + +1. Select an existing passing `tests/wrapper` test and run its complete + generation unit through the legacy route with generated artifacts retained. +2. Record the generated C binding source/header, Fortran bridge source, + generated artifact names and requirements, native-call order, runtime + behavior, and failure behavior relevant to the lane. +3. Trace those artifacts back through the current Python lowering, binding, + bridge, node/API-model, printer, runtime-helper, and build-orchestration + methods. Record the exact source methods and consumed state in this + checklist before porting them. +4. Complete any semantic decisions exposed by that trace in post-IR policy. + Do not move an old local decision into the planner or emitter merely because + that is the quickest textual copy. +5. Copy a small proven implementation unchanged when all of it is needed, or + rewrite the smallest independent equivalent when the old implementation + carries unrelated state. Start from the old method bodies and generated + artifacts; do not invent a replacement mechanism from memory. +6. Generate the same complete artifact set through the new plan route, inspect + differences against the retained legacy artifacts, then compile and run the + same existing assertions through both routes. + +Generated source is diagnostic evidence, not necessarily a byte-for-byte +golden file. Equivalent backend-local names or mechanical formatting are +allowed, but every ABI, conversion, ownership, cleanup, call-order, and build- +artifact difference must be explained. When a parity failure occurs, rerun and +inspect the working legacy route before changing the new implementation. + +Most wrapper tests share conversion, node, printer, and build mechanisms. Once +one mechanism has been reproduced and validated, later tests should reuse the +same new handler or primitive and add only their completed policy/plan mapping +or genuinely new mechanical behavior. + ## Route Selection Contract The initial migration route is atomic for one extension generation unit. In @@ -149,11 +201,61 @@ Function-level or mixed-route generation can be considered only after the module-level migration is complete and only if a concrete need justifies its extra composition and validation rules. It is not part of this checklist. +## Policy Authority And Planner Boundary + +The wrapper planner receives policy-completed semantic owners and converts them +into a wrapper plan. It is not another policy stage. + +```text +semantic contract + semantic datatype facts + -> post-IR policy completion + -> complete module/function/class/variable/argument/result policies + -> WrapperPlanner.build(policy-completed module) + -> frozen WrapperPlan +``` + +- Datatype states what an object is: scalar family, precision, rank, shape, or + other representation facts. Completed policy states what wrapper generation + must do: Python conversion action, bridge/native action, ownership, transfer, + destruction, mutability/writeback, nullability, output projection, release, + storage mode, getter/setter behavior, native-call order, and lifecycle order. +- The planner copies those completed decisions and datatype facts into readable + plan records, assigns stable owner paths and symbolic references, and wires + already-decided producers to consumers. It may traverse owners, preserve + declared order, and build deterministic tuples and lookup references. +- The planner must not select an action, ownership mode, ABI behavior, call + order, writeback, cleanup, result projection, or handler from datatype, + `intent`, decorators, raw metadata, `is_alias`, dotted-variable shape, local + memory checks, or a missing policy field. Such a branch means policy + completion is incomplete and must be moved there. +- Planner complexity is presumed to indicate an incomplete policy boundary. + A planner branch is allowed only for structural traversal or deterministic + wiring that cannot change wrapper semantics. Any other exception requires a + concrete reason recorded in this checklist before implementation and a + focused test proving that it is structural rather than a hidden decision. +- Every runtime-visible or runtime-required owner must expose a complete typed + policy before planning. If an argument, result, function, class, module + variable, decorator effect, or native projection is still represented only + by scattered facts that the planner would need to interpret, extend post-IR + policy completion with a typed completed-policy record first. +- The plan retains the completed policy values or stable typed references to + them so validation and rendering can show why each action was selected. Plan + records must not use untyped `object` or free-form dictionaries for policy. +- Completed policy is the customization authority. A maintainer who wants + different generated behavior changes the contract/policy, reruns post-IR + policy completion when applicable, and rebuilds the plan. There is no direct + plan-transformation API and no backend-specific customization path. +- The validator checks that the plan is a faithful and structurally consistent + projection of completed policy. It diagnoses missing policy, unsupported + completed actions, broken handoffs, and producer/consumer mismatches; it does + not fill defaults or choose replacement behavior. + ## Design Rules -- Post-IR policy completion decides semantic actions. Binding and bridge emitters - consume those actions; they do not reconstruct them from datatype, `intent`, - `is_alias`, local memory handling, dotted-variable shape, or missing policy. +- Post-IR policy completion decides semantic actions. The planner projects those + decisions into the plan, and binding and bridge emitters consume them. None of + these stages reconstruct policy from datatype, `intent`, `is_alias`, local + memory handling, dotted-variable shape, or missing policy. - The wrapper plan records action keys, handoff expectations, and native-call ordering. It does not contain raw generated C, Fortran, or CPython text. - Binding and bridge emitters are dispatch tables plus small implementation @@ -164,21 +266,27 @@ extra composition and validation rules. It is not part of this checklist. - Each action has exactly one registered handler for its emitter. Validation and plan rendering resolve that registry to the exact method name; emitters do not dynamically construct method names or run a second policy-selection tree. -- Initial copied handlers keep the current dispatch method names when practical, +- Initial isolated handlers keep the current dispatch method names when practical, so a plan action can be traced directly to the audited legacy method. Add a - new plan-only action only when no completed policy action expresses the - required semantic step; first confirm that the missing distinction does not - belong in post-IR policy completion. + new completed policy action in post-IR policy completion when no existing + action expresses the required semantic step; never invent a plan-only action + to avoid completing policy. - Dispatch should be by semantic lane and action, not by every concrete dtype. For example, `Float64`, `Int32`, and `Bool` can share scalar-value handlers while dtype remains plan data. - Every binding output that the bridge consumes is represented by a handoff - spec. Every bridge native argument/result that the native call consumes is - represented by a native-call spec. + spec in one end-to-end argument/result transfer record. Every bridge native + argument/result that the native call consumes is represented in that same + record and the function's `BridgeAbiPlan`/`NativeCallPlan`. - Validation checks producer/consumer consistency before emission: - `binding.produces` must match `bridge.expects`, bridge output must match the - native call slot, and writeback/result plans must consume values that were - actually produced. + the transfer's Python-side action must produce its handoff, its bridge ABI + slot must consume that handoff, its native action must satisfy the native-call + slot, and writeback/result plans must consume values actually produced. +- Keep one readable `ArgumentTransferPlan` for the complete Python argument -> C + handoff -> bridge ABI -> native argument path. Do not force maintainers to + join separate binding and bridge subplans. C and Fortran emitters remain + separate implementations that consume different fields of the same transfer + record. - Local variable names may be generated by a plan context, but the plan should carry stable symbolic roles such as `value`, `descriptor`, `shape`, `status`, `message`, or `writeback`. @@ -201,17 +309,176 @@ extra composition and validation rules. It is not part of this checklist. import `x2py.codegen`, and legacy `x2py.codegen` modules must not import the new generator. High-level pipeline orchestration is the only owner allowed to select between them. -- Copy backend primitives incrementally by lane, not as a wholesale clone of - the legacy codegen package. Each copied slice includes only the dependency +- Add backend primitives incrementally by lane, not as a wholesale clone of the + legacy codegen package. Each isolated slice includes only the dependency closure needed to generate and print that lane, so its representation can evolve without changing the legacy route. -- Maintainer edits use an explicit immutable plan transformation between plan - construction and validation. The transformed plan is validated exactly like - a generated plan; there is no trusted-edit bypass. - The binding and bridge emitters both consume the same validated plan. Their generated runtime data flow is Python binding -> bridge -> native call, but one emitter is not the semantic-policy input to the other. +### Hierarchical Plan Ownership + +The plan mirrors semantic ownership. It is hierarchical rather than one flat +list and does not rely on one large planner method. + +```text +WrapperPlan + ModulePlan + VariablePlan ... + FunctionPlan + ArgumentPlan -> ArgumentTransferPlan ... + ResultPlan ... + BridgeAbiPlan + NativeCallPlan + WritebackPlan/CleanupPlan ... + ClassPlan + ConstructorPlan ... + MethodPlan ... + PropertyPlan ... + VariablePlan ... +``` + +- Create one frozen plan record for each runtime-visible or runtime-required + semantic owner. The record carries its completed typed policy, datatype facts + where applicable, stable owner path, and references to its child records. +- The module planner preserves declared member order and visits module + variables, functions, and classes. The function planner visits arguments and + results and assembles the already-completed call, ABI, writeback, cleanup, and + projection records. The class planner does the same for constructors, + methods, properties, and fields. +- Hidden native arguments/results, decorator projections, reordered native + slots, and function-local lifecycle steps belong under their owning + `FunctionPlan`. They are not detached module-level plan objects. +- Each planner visitor method returns one plan node. Only the owning parent + assembles child nodes into ordered tuples; child visitors do not mutate a + shared plan or inspect sibling backend output. +- Cross-level and cross-backend relationships use typed references or stable + owner paths. Do not duplicate a child plan to make it available to both + emitters. +- Owners that do not affect runtime wrapper generation may be absent only when + the support analyzer explicitly classifies them as ignorable. An unsupported + runtime owner selects the whole legacy route before planning. + +### Visitor And Class Ownership + +Tree traversal in the isolated generator is class-based and follows one +visitor protocol. Production behavior is owned by named classes and methods, +not a collection of module-level orchestration functions. + +- Implement a minimal independent `wrapper_codegen.visitor.ClassVisitor` with + deterministic MRO dispatch and a configurable method prefix. Audit the + existing visitor algorithm as the behavioral source, but include only the + behavior the isolated planner, validators, emitters, and printers need. +- `WrapperPlanner(ClassVisitor)` traverses policy-completed semantic owners with + methods such as `_visit_SemanticModule`, `_visit_SemanticFunction`, + `_visit_SemanticClass`, and `_visit_SemanticVariable`. These methods only + copy completed policy/datatype facts and assemble deterministic plan records. +- `WrapperPlanSupportAnalyzer`, `WrapperPlanValidator`, and + `WrapperPlanRenderer` use the same visitor protocol for semantic or plan-node + traversal. The C binding and Fortran bridge emitters use visitors to assemble + module/function/argument/result structure. +- Isolated printers use the same protocol with + `visitor_method_prefix = "_print"` and explicit `_print_` methods. + Unsupported node types fail through the visitor's explicit default path. +- Visitor dispatch answers "which model/node type is this?" Completed-action + registries answer "which already-decided mechanical implementation runs?" + Do not replace action registries with `isinstance` ladders inside visitor + methods, and do not make visitor dispatch another policy selector. +- New production modules expose class APIs: `WrapperPlanner.build(...)`, + `WrapperPlanSupportAnalyzer.analyze(...)`, + `WrapperPlanValidator.validate(...)`, `WrapperPlanRenderer.render(...)`, and + `WrapperCodeGenerator.generate(...)`. Do not add equivalent module-level + builder, validator, renderer, support, emitter, printer, or generator + functions. +- A module-level production function is permitted only when a concrete Python + protocol or external entrypoint requires it and the reason is recorded in + this checklist before implementation. Dataclasses, enums, constants, test + functions, and a tool script's `main()` are not orchestration alternatives. +- Structural checks reject undeclared module-level functions in + `x2py.wrapper_codegen` and reject top-level `isinstance`/`match` traversal + ladders that bypass the visitor protocol. + +### Dispatch Size And Splitting + +- A primary emitter dispatcher is keyed by a completed action such as + `SCALAR_VALUE` or `PASS_VALUE`. Its selected method should implement one + understandable mechanical case and may delegate repeated node construction + to small private helpers. +- If a selected method grows because scalar families genuinely use different + APIs or generated control flow, add a visible secondary dispatcher keyed by + an explicit backend datatype family such as logical, integer, real, or + complex. Keep all precisions of a family together when precision changes only + type data or a conversion-table entry. +- Split again by precision or concrete datatype only when the emitted API, + range/error handling, declarations, or control flow actually differs. Do not + create one handler per dtype merely to keep methods short. +- Handler registries and plan rendering expose the complete selected chain, for + example `_convert_python_scalar_value_argument -> _convert_python_real_value`. + Missing primary or secondary combinations fail validation/emission rather + than falling back to a general method. +- When a method becomes difficult to follow, first separate independent phases + such as declaration, conversion, call argument, and cleanup into node-fragment + helpers. Use another dispatcher only when there is a real behavioral axis to + dispatch on. + +### Emitter Traceability And Complexity Gates + +The repository-wide Ruff/Radon limits protect legacy code but are too permissive +for the new emitter contract. The general changed-block Radon limit is 20 and +the Ruff McCabe limit is 45; neither means a maintainer can reproduce an emitter +step easily. Add a stricter static checker for `x2py.wrapper_codegen` before the +first emitter handlers are implemented. + +- Every registered primary handler, secondary handler, and private + node-fragment helper in `c/binding.py` or `fortran/bridge.py` has Radon + cyclomatic complexity at most 5 (grade A), at most 25 statements, and control- + flow nesting depth at most 2. +- Every `WrapperPlanner` visitor method has the same complexity, statement, and + nesting limits. It may use an explicit completed-policy variant only to + choose the corresponding plan-record shape. It may not dispatch on datatype, + decorators, or raw metadata to derive behavior; completed policy already + contains the selected actions and ordering. +- A registered handler accepts one validated transfer/result plan plus an + explicit backend emission context and returns an `EmissionFragment`. It must + not require a maintainer to recreate hidden mutable emitter state before + calling it in a focused test. +- Module/function assemblers and the top-level artifact orchestrator may have + cyclomatic complexity at most 10 and at most 50 statements because they + combine already-selected fragments; they may not perform policy or datatype + dispatch. +- New node, API-model, naming, and context methods are also kept at Radon + complexity at most 5 and contain only behavior required by current migrated + lanes. Copied printer methods may remain outside the stricter handler limit + when preserving complex legacy formatting; they remain subject to the + repository Radon non-regression policy and baseline rendering tests. +- Do not add a permanent complexity allowlist for a registered emitter handler. + A copied legacy handler that exceeds a limit may exist during isolated + baseline work, but the new route cannot become eligible for that action until + the handler is split and passes the strict gate. +- Complexity is a trigger for review, not an instruction to add a dispatcher + blindly. If complexity comes from independent emission phases, extract small + fragment helpers. If it comes from different datatype-family behavior, add a + secondary family dispatcher. If precision only selects a type/API table + value, keep one family handler and use data rather than another dispatcher. +- The checker also verifies that every registry target exists, every registered + handler is measured, no handler calls a printer, and every secondary + dispatcher has a total explicit mapping for the supported plan combinations. + It also enforces the declared module-level-function and visitor rules for the + isolated package. +- Plan rendering resolves and prints the full primary/secondary handler chain. + Registry checks verify that every rendered handler exists and is covered by a + supported plan combination. Compiled parity through existing wrapper tests is + the default behavioral proof; direct handler tests are reserved for + mechanical failures that existing fixtures cannot isolate. + +Implement this as a dedicated checker such as +`python3 tools/check_wrapper_codegen_complexity.py`. It may reuse Radon's +`cc_visit` and the repository's existing Radon helper code, while adding AST- +based statement count, nesting, registry, and forbidden-printer-call checks. +Run it as a blocking static check for every wrapper-codegen implementation +change. + ## Intended Ownership And Files The exact names may evolve during Phase 0, but responsibilities should remain @@ -219,32 +486,34 @@ separated along these lines: ```text x2py/wrapper_codegen/ - scope.py copied scope and deterministic name-allocation behavior - types.py copied datatype/literal primitives needed by migrated lanes + visitor.py minimal independent class visitor used by isolated models + names.py minimal deterministic NameAllocator; no legacy Scope copy + types.py minimal datatype/literal nodes needed by migrated lanes plan/ models.py frozen readable plan records - actions.py action, handoff, and native-pass enums/specs - build.py completed semantic IR -> WrapperPlan - validate.py cross-boundary consistency validation - support.py whole-generation-unit route support report - render.py stable maintainer-readable plan rendering + specs.py handoff, bridge-ABI, native-slot, and structural specs + build.py WrapperPlanner visitor: completed policy -> WrapperPlan + validate.py WrapperPlanValidator visitor + support.py WrapperPlanSupportAnalyzer visitor and route report + render.py WrapperPlanRenderer visitor c/ - nodes.py copied C declarations/statements needed by migrated lanes - cpython_api.py copied CPython API primitives needed by migrated lanes - numpy_api.py copied NumPy C API primitives needed by migrated lanes - concepts.py copied C/CPython helper concepts needed by migrated lanes - context.py new-route local values and cleanup state - binding.py validated plan -> complete CPython binding representation - printer.py isolated C/CPython source emission + nodes.py minimal C declarations/statements needed by migrated lanes + cpython_api.py minimal CPython API nodes needed by migrated lanes + numpy_api.py minimal NumPy C API nodes needed by migrated lanes + concepts.py minimal C/CPython helper nodes needed by migrated lanes + context.py module/function names, local values, and cleanup state + binding.py CPythonBindingEmitter visitor + printer.py CPythonCodePrinter visitor with _print_ methods fortran/ - nodes.py copied Fortran declarations/statements needed by lanes - context.py new-route Fortran names and scopes - bridge.py validated plan -> complete native bridge representation - printer.py isolated Fortran source emission + nodes.py minimal Fortran declarations/statements needed by lanes + context.py module/function names and local values + bridge.py FortranBridgeEmitter visitor + printer.py FCodePrinter visitor with _print_ methods + generate.py WrapperCodeGenerator class and artifact orchestration artifacts.py complete new-route generated-wrapper artifact result x2py/pipeline/ @@ -257,11 +526,37 @@ under `c/` and `fortran/` are added only as a migrated lane needs them; the layout does not authorize copying the entire legacy package up front. Within the generation boundary there is no shared model identity between the -routes. If a migrated scalar handler needs a legacy `Variable`, datatype, -literal, CPython call model, NumPy call model, scope, or printer-dispatch base, -copy that item's complete required dependency slice into -`x2py.wrapper_codegen` first. The copied class may then evolve independently -without changing or importing its legacy counterpart. +routes. If a migrated handler needs behavior currently provided by a legacy +`Variable`, datatype, literal, API model, scope, or printer-dispatch base, audit +which parts are actually consumed and implement the smallest independent class +that reproduces those parts. Copy a class unchanged only when it is already +small and all of it is required. + +### Naming Without Legacy Scope + +Do not copy the general legacy `Scope`. The wrapper plan already resolves +public names, native names, bridge symbols, argument positions, and ABI slots; +the new backend only needs deterministic collision-free names for generated +modules, functions, and local temporaries. + +- `NameAllocator` owns `reserve(name)` and `new_name(base)` with deterministic + suffixing. It does not store semantic variables, classes, decorators, + symbolic aliases, loops, dotted symbols, imports, or original-name lookup. +- A C or Fortran module emission context owns one module-level allocator for + generated functions, module objects, helper symbols, and public/ABI names + reserved from the validated plan. +- Each emitted function receives an explicit function context with its own + local allocator for arguments, result storage, temporaries, and cleanup + values. It may reserve module symbols but does not search a parent semantic + scope. +- Later class support may add a class emission context only when Phase 9 proves + it is needed. Do not add class/loop/program scope categories during scalar + phases. +- Public and ABI names come from the plan and cannot be renamed by the backend + allocator. The allocator handles only backend-local collisions. +- Focused naming tests copy the relevant legacy collision cases and prove stable + names across repeated generation, but the new allocator API and internal data + structures remain minimal. `ir2ast.py` remains the entry to the temporary legacy route; it must not choose the route or partially invoke the new emitters. The source-driven and `.pyi`- @@ -274,11 +569,11 @@ understand plan actions. That result is an internal build handoff, not a compatibility API and not a second semantic model. The isolated C and Fortran layers are thin mechanical backend layers. For each -primitive, first copy the proven legacy behavior and tests needed by the lane; -then simplify or modify the new copy only as required by plan-driven emission. -The original node, printer, scope, and generator implementations remain -untouched and runnable. Shared compilation receives generated files through the -artifact result and does not import either route's internal models. +primitive, preserve the proven legacy behavior and tests needed by the lane +while choosing the smallest new representation that can express it. The +original node, printer, scope, and generator implementations remain untouched +and runnable. Shared compilation receives generated files through the artifact +result and does not import either route's internal models. ## Node Construction And Printing Contract @@ -304,6 +599,17 @@ validated WrapperPlan Fortran AST, and the bridge emitter must not discover information that the C binding needs. If either occurs, the shared plan or ABI specification is incomplete. +- Preserve the legacy high-level pattern in isolated form: + `WrapperCodeGenerator.generate(plan)` constructs complete `fortran_module` + and `c_module` objects, then passes them to isolated + `FCodePrinter.doprint(...)` and `CPythonCodePrinter.doprint(...)` + implementations. Preserve this orchestration and the required module/header + behavior from legacy codegen rather than inventing a second source-writing + mechanism. +- Do not copy the legacy sequential dependency where the binding generator + learns its ABI by consuming the bridge generator's AST. In the new route, + both complete module trees are independently constructed from the same + validated `WrapperPlan` and `BridgeAbiPlan`. - Each dispatched emitter method returns backend-node fragments such as declarations, setup statements, call arguments, result statements, success/failure cleanup, and produced symbolic values. A module assembler @@ -320,7 +626,10 @@ validated WrapperPlan declarations are selected by plan-driven emission and represented as nodes before printing. A printer may deduplicate or order them mechanically. - C source/header and Fortran source printers are independently testable against - the copied baseline nodes before plan emitters use them. + the baseline nodes before plan emitters use them. Each isolated printer + implements only the `_print_` cases required by currently migrated + nodes and fails explicitly for unsupported node types; do not copy unused + printer methods in anticipation of later lanes. - `plan/` does not import `c/` or `fortran/`; the C and Fortran backends do not import each other; and backend printers import their nodes/types but not plan builders, actions, validators, emitters, or pipeline routing. Add structural @@ -342,6 +651,8 @@ class WrapperPlan: @dataclass(frozen=True) class ModulePlan: public_name: str + owner_path: OwnerPath + policy: CompletedModulePolicy functions: tuple[FunctionPlan, ...] variables: tuple[VariablePlan, ...] classes: tuple[ClassPlan, ...] @@ -351,7 +662,8 @@ class ModulePlan: class FunctionPlan: public_name: str native_name: str - decorators: DecoratorPlan + owner_path: OwnerPath + policy: CompletedFunctionPolicy python_arguments: tuple[ArgumentPlan, ...] bridge_abi: BridgeAbiPlan native_call: NativeCallPlan @@ -362,25 +674,21 @@ class FunctionPlan: @dataclass(frozen=True) class ArgumentPlan: public_name: str - semantic_type: object + owner_path: OwnerPath + datatype: SemanticType + policy: OwnershipDecision python_position: int | None - binding: BindingStep - bridge: BridgeStep - native: NativeArgumentSpec + transfer: ArgumentTransferPlan writeback: WritebackPlan | None = None @dataclass(frozen=True) -class BindingStep: - action: PythonBarrierAction - produces: tuple[HandoffSpec, ...] - - -@dataclass(frozen=True) -class BridgeStep: - action: NativeBarrierAction - expects: tuple[HandoffSpec, ...] - produces: tuple[NativeArgumentSpec, ...] +class ArgumentTransferPlan: + python_action: PythonBarrierAction + handoff: HandoffSpec + bridge_slot: BridgeArgumentRef + native_action: NativeBarrierAction + native_slot: NativeArgumentRef @dataclass(frozen=True) @@ -392,6 +700,12 @@ class NativeCallPlan: The plan validator owns consistency diagnostics. Binding and bridge emitters should be able to trust a validated plan and focus on emitted-code mechanics. +`CompletedModulePolicy` and `CompletedFunctionPolicy` stand for typed post-IR +policy records, not plan-owned decisions. Phase 0D must replace these sketch +names with the actual completed semantic policy types and define equivalent +typed policy fields for results, variables, and classes before those owners are +migrated. If policy completion cannot provide such a record, that semantic +stage must be completed before the corresponding planner visitor is written. The first implementation also needs two non-semantic orchestration records: @@ -430,13 +744,13 @@ function f(x: Float64) -> None result none ``` -The exact top-level call path should be equally direct: +The exact class-owned call path should be equally direct: ```text -complete_semantic_policies(module) existing semantic stage -build_wrapper_plan(module) completed policy -> WrapperPlan -validate_wrapper_plan(plan) handoff/ABI/handler validation -generate_wrapper_artifacts(plan) +completed = complete_semantic_policies(module) existing semantic stage +plan = WrapperPlanner().build(completed[0]) completed policy -> WrapperPlan +WrapperPlanValidator().validate(plan) structural consistency only +WrapperCodeGenerator().generate(plan) CPythonBindingEmitter.emit_function(f) -> _convert_python_scalar_value_argument(x) -> isolated C node fragments @@ -448,6 +762,13 @@ generate_wrapper_artifacts(plan) create_shared_library(...) existing compilation/link entrypoint ``` +`WrapperPlanner._visit_SemanticFunction(f)` visits `x`, copies its `Float64` +datatype and completed `SCALAR_VALUE`/`PASS_VALUE` policy actions, and wires the +pre-decided positions into `ArgumentTransferPlan`. It does not decide that a +`Float64` should use those actions. A different valid completed policy produces +a different plan through the same visitor without changing planner or emitter +code. + At runtime, the generated CPython binding converts the Python argument into the scalar C handoff, calls the generated bridge symbol, and the bridge invokes the native procedure using the completed `PASS_VALUE` behavior. CPython reference @@ -489,32 +810,124 @@ matrix, add it before implementing or routing that case. Do not treat the table as proof that every current syntax spelling has already been audited; Phase 0 owns that live inventory. +## Existing Wrapper Suite As The Migration Queue + +`tests/wrapper` is the behavioral source and final acceptance suite for this +migration. Migrate its existing generation units one by one; do not create a +parallel wrapper suite or new native source fixtures merely to make the new +route easier to exercise. + +- Phase 0A adds a maintained migration matrix to this file covering every + Python test node under `tests/wrapper`. Each row records whether the test + generates a wrapper, the source/contract generation unit it uses, the lanes + that currently block the wrapper-plan route, and one status: + `not-applicable`, `deferred-real-library`, `legacy`, `dual-route`, or + `wrapper-plan`. +- Existing source files, contract fixtures, build helpers, runtime assertions, + failure assertions, and ABI assertions are reused as written whenever they + already cover the migrated behavior. Do not copy their behavior into a new + test with a smaller invented source. +- A new native source or contract fixture is allowed only when the audit proves + that accepted production behavior has no existing test. Record that coverage + gap and its owning semantic lane here before adding the fixture; migration + convenience is not sufficient justification. +- Whole-generation-unit routing still applies. An existing test moves to + `dual-route` only when every runtime-required feature in its module is + supported. If a nominally scalar fixture also contains results, strings, + arrays, decorators, module state, or classes, leave it on the legacy route + until those lanes are complete rather than carving out a narrower fixture. +- Dual-route parity reuses the same existing fixture and assertion function for + legacy and wrapper-plan builds through internal test orchestration. Do not + add a public route flag, duplicate the behavioral assertions, or require + byte-identical generated source. +- Once parity passes and production eligibility is widened, that existing test + moves to `wrapper-plan`. Keep deliberate legacy execution only in the + migration parity harness until final cutover. +- The final target is not merely that `tests/wrapper` passes. Every test in the + suite must be represented in the migration matrix, and every test that + generates a runtime wrapper must use the wrapper-plan route after cutover. + Tests that only inspect documentation, layout, parsing, or `.pyi` generation + may be `not-applicable` but must still pass. +- During active migration, every local and GitHub Actions pytest invocation + excludes + `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py`. Mark its BLAS + and LAPACK rows `deferred-real-library`; do not use either corpus for lane + parity or general migration verification. General native-bundle tests in + `test_stage7_native_bundles.py` remain active because they test linker/build + mechanics independently of the full BLAS/LAPACK corpora. + +### Intermediate Test Contract + +Add intermediate tests only where they protect a stable boundary or a failure +that the existing compiled wrapper tests cannot isolate. + +- Test policy completeness for each newly migrated owner: missing or blocked + policy fails before planning. +- Test policy-to-plan projection with an existing semantic fixture. Assert the + owner hierarchy, completed action keys, call/ABI order, handoffs, and one + concise maintainer rendering for the lane; do not snapshot every plan field. +- Test validator failures with small directly constructed plans. Use + table-driven cases for invariant categories such as missing/duplicate slots, + incompatible producer/consumer handoffs, unavailable results/writebacks, and + unsupported completed actions. Do not add one test per implementation branch. +- Test whole-module support and route selection with existing generation units: + one fully supported unit and representative units blocked by later lanes. + Prove that no unit mixes routes and no new-route failure silently retries + legacy generation. +- Test the isolated visitor, name allocator, dependency boundary, registry + completeness, and complexity checker directly because they are standalone + contracts not observable through wrapper runtime behavior. +- Do not require one direct unit test per emitter handler or helper. Registry + checks prove that the rendered handler exists; the reused compiled wrapper + tests prove the selected chain's behavior. Add a direct emitter test only for + a failure or mechanical contract that cannot be reached through an existing + wrapper fixture. +- Do not add full generated C/Fortran source snapshots. Keep or add focused + source assertions only when exact source structure is the observable ABI or + build contract and runtime behavior cannot prove it. +- Add focused node/printer tests only for nontrivial mechanical behavior such as + precedence, escaping, declaration syntax, or header/source partitioning. + Do not create one test per node or copied printer method. + +The governing rule is one intermediate test per stable contract or failure +category, not one test per class, method, node, or branch. + ## Incremental Protocol For each lane: -1. Audit the current lowering, binding, bridge, printer, runtime-helper, and - build paths for the lane, and record the observed behavior and focused tests - that make it the migration baseline. -2. Expand this checklist with the lane's exact scope, exclusions, source-path - baseline, copied dependencies, plan fields, and validation invariants. -3. Copy the minimum dependency-closed legacy backend primitives required by the - lane into `x2py.wrapper_codegen`, prove copied baseline behavior, then add - or adapt isolated node/printer tests. -4. Implement the plan objects, action registries, ABI/handoff specs, validator, - and support-report coverage for that lane. -5. Generate the plan from policy-completed semantic IR, dispatch binding and +1. Select and run an existing passing `tests/wrapper` generation unit through + the legacy route. Retain and inspect its complete generated artifact set. +2. Trace the current lowering, binding, bridge, node/API-model, printer, + runtime-helper, and build paths that produced those artifacts, and record the + observed behavior and existing tests that make them the migration baseline. +3. Expand this checklist with the lane's exact scope, exclusions, source-path + baseline, required backend behavior, plan fields, and validation invariants. +4. Complete every policy field required by the lane in post-IR policy + completion; do not start its planner while semantic decisions remain + scattered or implicit. +5. Implement the lane's hierarchical plan records, planner visitors, + action registries, ABI/handoff specs, validator, renderer, and support-report + coverage. +6. Implement the minimum dependency-closed backend slice in + `x2py.wrapper_codegen`: copy small suitable pieces, rewrite oversized legacy + classes as minimal equivalents, and add only the intermediate tests required + by the contract above. +7. Generate the plan from policy-completed semantic IR, dispatch binding and bridge handlers into isolated node fragments, assemble complete backend modules, and print complete internal artifacts. -6. Compile the internal artifacts before changing production route selection. -7. Run the same eligible fixtures through both routes and compare compiled - runtime behavior, failure paths, native-call mapping, and artifact - requirements. -8. Extend the whole-module support predicate so a generation unit uses the +8. Compare the new generated artifacts with the retained legacy artifacts and + explain every material difference before compilation. +9. Compile the internal artifacts before changing production route selection. +10. Run the same eligible existing fixtures and assertions through both routes + and compare compiled runtime behavior, failure paths, native-call mapping, + and artifact requirements. +11. Extend the whole-module support predicate so a generation unit uses the wrapper-plan route only when all its elements belong to completed lanes. Keep the old route for generation units containing unsupported lanes. -9. Mark the lane complete only when focused parity tests pass and every - intentional difference from the baseline is separately documented. +12. Update every affected `tests/wrapper` migration-matrix row and mark the lane + complete only when the reused parity tests pass and every intentional + difference from the baseline is separately documented. Do not start a later lane by guessing. Each lane must define the handoff specs and consistency checks it needs. @@ -539,11 +952,11 @@ Each expanded sub-lane must state: - validation invariants across Python input/result, binding handoff, bridge handoff, native call, writeback, cleanup, ownership, and release; - the whole-module support-predicate change that makes the sub-lane eligible; -- focused generation/runtime tests and parity evidence against the legacy - route; -- the exact legacy source path for every copied primitive, its dependency - closure, baseline evidence, modifications in the isolated copy, and a reason - for every entirely new backend primitive; +- existing `tests/wrapper` nodes that cover the sub-lane, their migration-matrix + status changes, and dual-route parity evidence against the legacy route; +- the exact legacy source path and consumed behavior for every isolated + primitive, whether it is copied or rewritten, its minimal dependency closure, + baseline evidence, and a reason for every behavior with no legacy source; - dependencies on earlier lanes and the legacy behavior that can be removed when the sub-lane is complete. @@ -560,32 +973,93 @@ source. Complete each subphase and its focused evidence before starting the next one: ```text -0A current-behavior baseline +0A tests/wrapper inventory and current-behavior baseline -> 0B isolated package and dependency boundary - -> 0C copied scalar backend nodes and printers - -> 0D wrapper-plan core and validation + -> 0C complete scalar policy + -> 0D hierarchical wrapper-plan core and validation + -> 0E minimal scalar backend nodes, names, and printers -> 1A internal scalar plan emission -> 1B compiled dual-route parity -> 1C production route selection -> later semantic lanes in numbered order + -> 11 cross-cutting tests/wrapper completion + -> 12 cutover and legacy removal ``` -Within later lanes, follow the same order: audit current behavior, expand the -lane checklist, copy required backend dependencies, add plan/actions and -validators, emit nodes, print internally, compile and compare both routes, and +Within later lanes, follow the same dependency order: identify the existing +`tests/wrapper` coverage, audit current behavior, expand the lane checklist, +complete policy, add plan records/planner/validation, implement only the +required backend behavior, emit and print nodes internally, compile and compare +the same existing tests through both routes, update the migration matrix, and only then widen production route eligibility. -## Phase 0 — Foundation Before Production Routing +### Session Continuation Protocol + +Progress is evidence-driven, not calendar-driven. Do not use a time target to +skip a prerequisite, weaken parity, reduce validation, invent a smaller test +fixture, or mark partially completed work as complete. + +The minimal user prompt for a new implementation session is: + +```text +Continue implementing the wrapper-plan migration checklist. +``` + +That prompt means the agent must follow this resume procedure before editing: + +1. Read this checklist and the repository instructions, then inspect the dirty + worktree, recent relevant commits, current checklist state, and wrapper-test + migration matrix. Work with existing user changes; do not reset them. +2. Audit the first unchecked dependency-closed group whose prerequisites are + genuinely complete. Reconcile stale checkbox state against live code and + tests before choosing work. +3. If the next item is a broad phase or lacks exact policy fields, legacy source + paths, plan records, handlers, invariants, and existing test coverage, expand + it in this file before implementation. +4. Use the legacy replay procedure for the selected existing generation unit. + Start from its passing test, retained generated artifacts, and traced Python + implementation rather than designing from memory. +5. Implement one coherent group through its required intermediate checks. Do + not stop after adding models or copied code when the group's next required + validation can be completed in the same session. +6. Run the focused existing `tests/wrapper` nodes, required intermediate + contract tests, wrapper-codegen checker, static-analysis commands, and other + verification required by `AGENTS.md` for the files changed. +7. Update checkboxes and migration-matrix rows only for behavior proven by the + required evidence. A legacy test passing does not prove the new route; a row + becomes `wrapper-plan` only when route diagnostics and parity requirements + prove it. +8. End the session with the exact completed group, changed pipeline stages, + legacy paths reused, tests and command results, remaining unsupported paths, + and the next dependency-ordered unchecked item. + +Do not search for a shortcut around an unmet gate. Missing policy goes back to +post-IR completion, unsupported mechanics remain on the explicit legacy route, +and failed parity is investigated against retained legacy artifacts. If the new +route starts recreating most of legacy codegen without materially improving +policy traceability, plan readability, or handler size, stop for an explicit +value review rather than continuing automatically. -### Phase 0A — Current Scalar Baseline +## Phase 0 — Foundation Before Production Routing +### Phase 0A — Wrapper Test Inventory And Current Scalar Baseline + +- [ ] Enumerate every Python test node under `tests/wrapper` and add the + migration matrix required above. Do not begin implementation with untracked + wrapper tests. +- [ ] Classify each test as non-generating or map it to its complete wrapper + generation unit and all semantic lanes needed before that unit can use the + wrapper-plan route. +- [ ] Select the first existing generation unit whose coverage best matches the + initial scalar lane. Record every additional feature in that unit that delays + atomic route eligibility; do not replace it with a new narrower fixture. - [ ] Inventory the current end-to-end wrapper behavior and implementation paths for the first scalar lane, including lowering branches, binding/bridge helper methods, CPython/NumPy API primitives, source printers, generated artifacts, build integration, and focused runtime fixtures. - [ ] Create a maintained baseline matrix mapping each first-lane current code - path and observable behavior to its proposed plan action, handler, copied - backend dependencies, and parity evidence. Do not define an action from a + path and observable behavior to its proposed plan action, handler, required + backend behavior, and parity evidence. Do not define an action from a hypothetical implementation. - [ ] Audit every decorator, native-call projection kind, implicit call behavior, and generated module/class feature accepted by the live semantic @@ -604,8 +1078,19 @@ only then widen production route eligibility. import `x2py.codegen`, legacy `x2py.codegen` cannot import `x2py.wrapper_codegen`, and only pipeline orchestration may eventually import both route entrypoints. -- [ ] Add dependency tests for that boundary before copied backend code is +- [ ] Add dependency tests for that boundary before isolated backend code is introduced. +- [ ] Implement and test the minimal independent `ClassVisitor` used throughout + the package, including deterministic MRO lookup, configurable method prefixes, + and explicit unsupported-node failure. +- [ ] Add structural checks requiring visitor-based traversal and rejecting + undeclared module-level production functions in `x2py.wrapper_codegen`. +- [ ] Add the blocking wrapper-codegen complexity/traceability checker before + emitter handlers are introduced. Cover Radon complexity, statement count, + nesting depth, registry completeness, secondary-dispatch completeness, and + forbidden printer calls from handlers. +- [ ] Add focused tests for the checker, including one failure fixture for each + enforced limit and registry/dependency rule. - [ ] Define the pipeline-owned generated-wrapper artifact result shared by both routes without duplicating native object/library/link-plan ownership. - [ ] Keep runtime helper APIs shared only when their behavior is unchanged. Add @@ -615,42 +1100,89 @@ only then widen production route eligibility. binding-emitter-local mechanics and are absent from plan models, rendered plans, and cross-backend plan validation. -### Phase 0C — Copied Scalar Backend Foundation - -- [ ] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, - datatype/literal models, CPython and NumPy API primitives, scopes/naming - behavior, helper concepts, and printer behavior required for Phase 1. Record - each legacy source path and baseline test before copying it. -- [ ] Copy those primitives into `x2py.wrapper_codegen` without importing, - aliasing, subclassing, or adapting legacy model classes. -- [ ] Keep each initial copy behaviorally equivalent to its legacy source before - modifying it for plan emission. -- [ ] Add isolated node/printer tests proving representative scalar C source, - C headers, and Fortran source render equivalently to the baseline. -- [ ] Verify the copied C/Fortran printers consume only isolated backend nodes - and cannot import or inspect wrapper-plan models. - -### Phase 0D — Wrapper Plan Core +### Phase 0C — Complete Scalar Policy + +- [ ] Audit the selected existing scalar generation unit and list every + module/function/argument/result/decorator/native-projection decision the + planner would otherwise need to infer. +- [ ] Define or complete typed post-IR policy records for the owners needed by + the first scalar lane. Do not use free-form metadata or planner defaults as a + substitute for a completed policy field. +- [ ] Move any remaining scalar action, call/ABI order, ownership, lifecycle, + projection, writeback, or cleanup decisions into + `complete_semantic_policies(...)` before implementing the planner. +- [ ] Verify the policy-completed module contains every datatype fact and + completed policy value needed to reproduce the audited legacy behavior + without reading bridge, binding, or backend-local state. +- [ ] Add only the intermediate policy tests required by the contract above, + reusing the selected existing semantic fixture and covering missing/blocked + policy failure before planning. +- [ ] Do not add plan models, backend nodes, emitters, printers, compilation, or + production route selection in this phase. + +### Phase 0D — Hierarchical Wrapper Plan Core - [ ] Define the first frozen plan data classes and tuple collections. -- [ ] Define `BridgeAbiPlan`, native-call refs, handoff specs, handler - registries, and validation errors around existing completed +- [ ] Implement `WrapperPlanner(ClassVisitor)` with explicit `_visit_` + methods that each return one plan record, recursively visit only that owner's + children, and perform only deterministic structural wiring. Keep each method + within the strict planner complexity gate. +- [ ] Define one `ArgumentTransferPlan` containing the existing completed Python + action, one binding-to-bridge handoff, bridge ABI slot, completed native + action, and native-call slot. Do not create separate binding and bridge + subplans for one argument. +- [ ] Define `BridgeAbiPlan`, native-call refs, handoff specs, primary/secondary + handler registries, and validation errors around existing completed `PythonBarrierAction` and `NativeBarrierAction` values. Do not add duplicate plan action enums for behavior already represented by completed policy. -- [ ] Add a plan validator that catches mismatched binding/bridge handoffs, - missing bridge/native-call slots, unknown action handlers, duplicate symbolic - roles, and writebacks/results that consume unavailable values. +- [ ] Add a plan validator that catches inconsistent transfer handoffs, + missing bridge/native-call slots, unknown primary or secondary handlers, + duplicate symbolic roles, and writebacks/results that consume unavailable + values. - [ ] Define the whole-generation-unit support report, including stable owner-path reasons for unsupported elements, without changing production route selection yet. -- [ ] Define a small immutable plan-transformation API so maintainers can alter - actions, ordering, or handoffs before validation without mutating the - policy-completed semantic IR. +- [ ] Implement class-owned `WrapperPlanSupportAnalyzer`, + `WrapperPlanValidator`, and `WrapperPlanRenderer` visitor APIs; do not add + equivalent module-level functions. - [ ] Add deterministic plan rendering that includes symbolic owner paths, - dispatch handler names, handoffs, bridge ABI slots, native slots, and - lifecycle order without backend nodes or CPython-specific mechanics. -- [ ] Verify generated and maintainer-transformed plans pass through the same - validator and produce owner-path diagnostics before node emission. + completed policy values, dispatch handler names, handoffs, bridge ABI slots, + native slots, and lifecycle order without backend nodes or CPython-specific + mechanics. +- [ ] Verify the planner fails on missing/incomplete policy rather than deriving + defaults, and the validator produces owner-path diagnostics before node + emission for policy/plan inconsistency. +- [ ] Add one policy-to-plan projection/rendering test for the selected existing + scalar fixture and table-driven validator tests by invariant category. Do not + add one test per planner method or plan field. +- [ ] Do not add backend nodes, emitters, printers, compilation, or production + route selection in this phase. + +### Phase 0E — Minimal Scalar Backend Foundation + +- [ ] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, + datatype/literal behavior, CPython and NumPy API primitives, naming behavior, + helper concepts, and printer cases required for Phase 1. Record each legacy + source path and consumed field/method before implementation. +- [ ] Implement a minimal `NameAllocator` and module/function emission contexts; + do not copy legacy `Scope` or its semantic lookup/categories. +- [ ] For every required node/API/helper class, choose explicitly between a + small unchanged copy and a rewritten minimal class. Each new field and method + must have a current Phase 1 emitter or printer consumer. +- [ ] Do not import, alias, subclass, or adapt legacy model classes. Preserve + required behavior through the isolated implementation and later compiled + parity evidence. +- [ ] Add focused tests only for nontrivial naming/node/printer mechanics that + the selected existing wrapper fixture cannot isolate. Do not add exhaustive + node tests or full generated-source snapshots. +- [ ] Reproduce the legacy module/header assembly and + `module -> doprint(module)` orchestration needed to create complete + `c_module` and `fortran_module` objects before writing source. +- [ ] Implement only the C/Fortran printer cases needed by the isolated scalar + nodes. Verify structurally that the printers consume only isolated nodes and + cannot import or inspect wrapper-plan models. +- [ ] Do not add scalar emitters, compilation, or production route selection in + this phase. ## Phase 1 — Scalar Function Inputs @@ -662,6 +1194,8 @@ projection kinds keep the whole module on the legacy route. ### Phase 1A — Internal Plan Emission - [ ] Generate plans for scalar value arguments such as `f(x: Float64)`. +- [ ] Populate one end-to-end `ArgumentTransferPlan` per scalar argument rather + than constructing separate binding and bridge plan objects. - [ ] Represent Python argument position and `@native_call` native argument order explicitly, including reordered arguments. - [ ] Represent implicit native order, `@bind(...)`, `@external`, and @@ -670,30 +1204,49 @@ projection kinds keep the whole module on the legacy route. literals as native argument sources with exact native positions. - [ ] Reject duplicate, missing, or out-of-range Python/native positions and bridge/native-call slots during plan validation. -- [ ] Add binding actions for scalar Python object to scalar value/storage. -- [ ] Add bridge actions for scalar pass-by-value, pass-by-address, and - call-local address where already supported by completed policy. -- [ ] Validate binding-produced scalar handoffs against bridge expectations and - the shared `BridgeAbiPlan`. +- [ ] Register the isolated `_convert_python_scalar_value_argument` binding + handler for `SCALAR_VALUE` and isolated native handlers for `PASS_VALUE`, + `PASS_CALL_LOCAL_ADDRESS`, or other scalar actions already supported by + completed policy. +- [ ] Reuse audited legacy method-body snippets where they remain simple; + otherwise write smaller direct handlers that reproduce the baseline behavior + using the isolated nodes and explicit contexts. +- [ ] If scalar-family mechanics make a handler difficult to follow, introduce + a secondary logical/integer/real/complex dispatcher; keep precision as data + unless it changes emitted APIs, checks, declarations, or control flow. +- [ ] Keep every scalar emitter handler/helper within the strict complexity, + statement-count, and nesting limits; expose any secondary handler chain in + plan rendering and registry checks. +- [ ] Validate the transfer handoff, bridge ABI slot, and native-call slot as one + chain before either emitter runs. - [ ] Make each scalar binding/bridge handler return isolated node fragments; assemble complete C and Fortran module nodes outside individual handlers. -- [ ] Print complete scalar-only bridge source, C/CPython binding source/header, - module initialization, and generated-source requirements through the isolated - printers and artifact assembler. +- [ ] Construct complete `c_module` and `fortran_module` objects, then pass them + to isolated `CPythonCodePrinter.doprint(...)` and `FCodePrinter.doprint(...)` + implementations to produce the scalar binding source/header and bridge + source. +- [ ] Assemble module initialization and generated-source requirements with the + printed files into the complete new-route artifact result. ### Phase 1B — Internal Compilation And Parity -- [ ] Provide test-only orchestration that sends the same eligible module +- [ ] Select existing `tests/wrapper` nodes whose complete generation units are + now covered; do not introduce a new scalar source fixture for parity. +- [ ] Provide test-only orchestration that sends each selected existing module directly through legacy and wrapper-plan routes without a public compatibility option or production selector change. - [ ] Compile and import new-route scalar artifacts through the shared compiler and linker before making any production module eligible. +- [ ] Reuse the selected tests' existing build helpers and assertions for both + routes; do not duplicate their behavioral assertions in a new test file. - [ ] Compare both routes for Python calls/results, native argument order, pass-by-value/address behavior, conversion failures, exception state, backend-local cleanup, generated artifact requirements, compilation, import, and runtime behavior. - [ ] Resolve every unexplained parity difference or document a separately approved behavior correction before proceeding. +- [ ] Change the selected tests' migration-matrix status from `legacy` to + `dual-route` only after both compiled routes pass. ### Phase 1C — Production Route Integration @@ -711,6 +1264,8 @@ projection kinds keep the whole module on the legacy route. linking failures do not silently retry the legacy route. - [ ] Keep explicit internal selection of the legacy route available for rollback and dual-route tests. +- [ ] Move every newly eligible existing test from `dual-route` to + `wrapper-plan` in the migration matrix after production selection passes. ## Phase 2 — Scalar Results And Hidden Outputs @@ -720,8 +1275,9 @@ results. - [ ] Audit and record the legacy result, hidden-output, result-packaging, `@raises`, cleanup, printer, and runtime paths that define this lane's baseline. -- [ ] Copy and baseline-test the additional result variables, CPython creation - calls, C/Fortran statements, and printer behavior required by this lane. +- [ ] Add or rewrite only the additional result variables, CPython creation + calls, C/Fortran statements, and printer cases required by this lane, with + baseline tests. - [ ] Represent direct native return, hidden output, identity output, and projected result lanes in `ResultPlan`. - [ ] Add bridge actions for scalar result assignment and hidden scalar output @@ -744,8 +1300,9 @@ values, and scalar allocatable/pointer descriptor boundaries. - [ ] Audit and record the legacy copy-in/out, optional presence, nullable scalar descriptor, cleanup, and failure-path behavior for this lane. -- [ ] Copy and baseline-test the additional optional/descriptor nodes, API - primitives, local-state helpers, and printer behavior required by this lane. +- [ ] Add or rewrite only the additional optional/descriptor nodes, API + primitives, local-state helpers, and printer cases required by this lane, + with baseline tests. - [ ] Represent copy-in, native mutation, copy-out, and cleanup as explicit writeback phases. - [ ] Preserve the three-state optional rule: omitted argument, explicit `None`, @@ -768,8 +1325,9 @@ represented before field access can use the plan route. - [ ] Audit and record the legacy scalar module-variable getter, setter, rejected replacement, module initialization, and attribute-routing behavior. -- [ ] Copy and baseline-test the additional module/type nodes, getter/setter API - primitives, initialization nodes, and printer behavior required by this lane. +- [ ] Add or rewrite only the additional module/type nodes, getter/setter API + primitives, initialization nodes, and printer cases required by this lane, + with baseline tests. - [ ] Represent getter behavior, setter exposure, native setter assignment, and rejected replacement behavior in module-variable plans. - [ ] Add binding actions for Python attribute get/set around scalar values. @@ -901,8 +1459,53 @@ paths. - [ ] Validate callback result and argument handoffs across binding, bridge, adapter, and trampoline steps before emission. -## Phase 11 — Cutover And Removal - +## Phase 11 — Cross-Cutting Wrapper Suite Completion + +Scope: existing wrapper tests whose generation units combine completed semantic +lanes or exercise build and runtime behavior rather than introducing one new +datatype lane. + +- [ ] Reconcile every remaining `legacy` or `dual-route` matrix row by owning + test area: `build_from_source`, `build_from_pyi`, `edit_pyi_contracts`, + `external_routines`, `multiple_files`, `naming`, `runtime_behavior`, and + `real_libraries`. +- [ ] Group remaining rows into dependency-ordered waves by their actual + unsupported owner paths. Do not implement a broad test directory as one + special case and do not add per-test backend fallbacks. +- [ ] For every newly discovered semantic or backend gap, expand the applicable + earlier lane or add an explicit sub-lane here, then follow the complete + policy -> plan -> backend -> emission -> compiled parity -> route sequence. +- [ ] Prove source-driven and semantic-`.pyi`-driven builds use the same route + selector and wrapper planner while retaining their existing build assertions. +- [ ] Prove edited-policy contracts, external symbols, multiple-source builds, + naming/generic interfaces, runtime policies, recursion, OpenMP, and real + library-independent native bundles preserve their existing assertions + through the wrapper-plan route. +- [ ] Keep non-wrapper-generating tests, including layout and generated-`.pyi` + checks, marked `not-applicable` to route selection but passing in the same + suite. +- [ ] Run every `tests/wrapper` test except + `test_real_blas_lapack.py` locally and in CI as the pre-cutover gate. +- [ ] Finish this phase only when every nondeferred matrix row is either + `wrapper-plan` or justified `not-applicable`; no nondeferred row may remain + `legacy` or `dual-route`. BLAS/LAPACK rows remain + `deferred-real-library` until Phase 12. + +## Phase 12 — Cutover And Removal + +- [ ] Re-audit collected Python test nodes under `tests/wrapper` and reconcile + them with the migration matrix. No test may be missing from the matrix. +- [ ] After every other migration row is complete, restore the full + `test_real_blas_lapack.py` run and any required native-cache preparation in + local opt-in verification and GitHub Actions. +- [ ] Run both BLAS and LAPACK generation units through legacy and wrapper-plan + routes using their existing assertions. Resolve parity before changing their + matrix rows from `deferred-real-library` to `wrapper-plan`. +- [ ] Require every wrapper-generating test row to be `wrapper-plan`; no row may + remain `legacy` or `dual-route`. Confirm route diagnostics show that every + runtime wrapper generation unit uses the new route. +- [ ] Run the complete `tests/wrapper` suite in CI, including restored BLAS and + LAPACK coverage, and require every test to pass before legacy deletion. - [ ] Track which lanes still use the old `semantic_ir_to_codegen_ast()` path. - [ ] Track route support at whole-generation-unit granularity and keep unsupported owner-path diagnostics stable until the corresponding lane is @@ -920,6 +1523,9 @@ paths. - [ ] Remove the temporary legacy route and its route diagnostics after every live generation unit is supported; do not replace it with compatibility shims or per-function fallback. +- [ ] Remove migration-only dual-route orchestration after the complete existing + wrapper suite proves the wrapper-plan route and legacy rollback is no longer + supported. Keep the existing behavioral fixtures and assertions. - [ ] Keep source printers only for the remaining generated source fragments they still own, or replace them with narrower emitters once the model layer is no longer needed. @@ -929,18 +1535,21 @@ paths. - [ ] Documentation-only changes run `python3 -m pytest -q tests/docs/test_examples.py tests/docs/test_structure.py` and `git diff --check`. -- [ ] Wrapper-plan code changes run focused plan validation tests, focused - wrapper generation tests, and the required static-analysis suite from - `AGENTS.md`. +- [ ] Wrapper-plan code changes run the affected existing `tests/wrapper` nodes, + the minimal intermediate contract tests required above, and the required + static-analysis suite from `AGENTS.md`. +- [ ] Wrapper-codegen implementation changes pass + `python3 tools/check_wrapper_codegen_complexity.py` with no handler waiver. - [ ] Runtime wrapper tests are required when generated behavior changes. -- [ ] Every migrated lane runs eligible fixtures through both routes. Compare - behavior and ABI-relevant call mapping; do not require byte-identical source - when mechanical organization differs. +- [ ] Every migrated lane runs eligible existing fixtures and assertions through + both routes. Compare behavior and ABI-relevant call mapping; do not require + byte-identical source when mechanical organization differs. - [ ] Structural dependency tests prove complete generator isolation: no imports from `x2py.wrapper_codegen` to `x2py.codegen` or in the reverse direction. -- [ ] LAPACK remains excluded from local verification unless separately - authorized. +- [ ] BLAS and LAPACK full-library wrapper tests remain excluded locally and in + GitHub Actions throughout Phases 0-11. Re-enable both only at the explicit + Phase 12 gate after every other migration row is complete. ## Completion Record @@ -948,6 +1557,8 @@ paths. and bridge handlers they dispatch to, and the handoff specs validated. - [ ] The final report lists old lowering/codegen paths still used by unsupported lanes. +- [ ] The final cutover report includes the completed `tests/wrapper` migration + matrix and confirms every wrapper-generating row uses the wrapper-plan route. - [ ] The final report includes focused verification commands and results. - [ ] The final report includes the changed-stage breakdown required by `AGENTS.md` and names every test file added or updated with the behavior it diff --git a/tests/README.md b/tests/README.md index 439df126e..1ad3252ee 100644 --- a/tests/README.md +++ b/tests/README.md @@ -73,8 +73,11 @@ know roadmap wording but not the feature module. Source-build, generated-`.pyi`, and modified-`.pyi` scenarios for one feature stay together; identical source/generated behavior uses one shared assertion body. -Do not run LAPACK wrapper runtime tests locally. Leave LAPACK coverage to GitHub -Actions unless the task explicitly requests it. +During the wrapper-plan migration, do not run the full BLAS or LAPACK +real-library wrapper tests locally or in GitHub Actions. Exclude +`wrapper/fortran/real_libraries/test_real_blas_lapack.py`; keep the general +native-bundle tests active. Re-enable both corpora only after every other +migration row is complete. ## Adding a test or helper From 22f5081cbd72ac66db5a2e133f9f88476d0a4c3b Mon Sep 17 00:00:00 2001 From: said Date: Sat, 11 Jul 2026 22:10:33 +0100 Subject: [PATCH 03/30] start implementing phase 0 --- .../wrapper-plan-migration-checklist.md | 511 ++++++++++++++++-- .../policy/test_scalar_wrapper_policy.py | 124 +++++ .../layout_rules/test_wrapper_guide_layout.py | 90 +++ .../wrapper_codegen/test_phase0b_contracts.py | 184 +++++++ .../wrapper_codegen/test_phase0d_plan_core.py | 203 +++++++ tests/wrapper_codegen/test_visitor.py | 56 ++ x2py/pipeline/wrapper_artifacts.py | 29 + x2py/semantics/models.py | 1 + x2py/semantics/policy_completion.py | 21 +- x2py/semantics/scalar_wrapper_policy.py | 403 ++++++++++++++ x2py/wrapper_codegen/__init__.py | 52 ++ x2py/wrapper_codegen/checks.py | 272 ++++++++++ x2py/wrapper_codegen/plan.py | 169 ++++++ x2py/wrapper_codegen/planner.py | 216 ++++++++ x2py/wrapper_codegen/renderer.py | 101 ++++ x2py/wrapper_codegen/support.py | 64 +++ x2py/wrapper_codegen/validator.py | 180 ++++++ x2py/wrapper_codegen/visitor.py | 44 ++ 18 files changed, 2677 insertions(+), 43 deletions(-) create mode 100644 tests/semantics/policy/test_scalar_wrapper_policy.py create mode 100644 tests/wrapper_codegen/test_phase0b_contracts.py create mode 100644 tests/wrapper_codegen/test_phase0d_plan_core.py create mode 100644 tests/wrapper_codegen/test_visitor.py create mode 100644 x2py/pipeline/wrapper_artifacts.py create mode 100644 x2py/semantics/scalar_wrapper_policy.py create mode 100644 x2py/wrapper_codegen/__init__.py create mode 100644 x2py/wrapper_codegen/checks.py create mode 100644 x2py/wrapper_codegen/plan.py create mode 100644 x2py/wrapper_codegen/planner.py create mode 100644 x2py/wrapper_codegen/renderer.py create mode 100644 x2py/wrapper_codegen/support.py create mode 100644 x2py/wrapper_codegen/validator.py create mode 100644 x2py/wrapper_codegen/visitor.py diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 2a78cbfdc..2c5437e01 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -856,6 +856,447 @@ route easier to exercise. `test_stage7_native_bundles.py` remain active because they test linker/build mechanics independently of the full BLAS/LAPACK corpora. +### Wrapper Test Migration Matrix + +Matrix rows use pytest selector patterns. A row ending in `::*` covers every +collected test node in that Python file when all nodes share the same +generation classification. A row ending in `[*]` covers the parametrized nodes +for that test function. The structural layout test expands these selectors +against live `python3 -m pytest --collect-only -q tests/wrapper` output, so a +new wrapper test node must either match an existing row intentionally or add a +new row here before later implementation starts. + +Statuses have the meanings defined above: `legacy` still uses the current +`semantic_ir_to_codegen_ast()` route, `not-applicable` does not generate a +runtime wrapper, and `deferred-real-library` is reserved for the full BLAS and +LAPACK corpus until Phase 12. + +| Pytest selector | Generation unit | Blocking lanes | Status | +| --- | --- | --- | --- | +| `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/arrays/test_array_results.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::*` | non-generating: legacy model/printer/policy unit coverage | legacy model/printer mechanics; ordinary arrays; native handles/descriptors | `not-applicable` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_cycles_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_missing_relative_contract_before_native_validation` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_rejects_colliding_wildcard_exports` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_matches_checked_in_fixture` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mixed_entry_exposes_externals_at_root_and_modules_as_children` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_leaf_can_be_the_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_variable_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mutable_module_variable_default_initializes_native_storage` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_one_entry_preserves_multiple_native_module_namespaces` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_requires_a_native_link_input` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_a_missing_native_artifact` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_address_contracts_before_codegen[*]` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_projection_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_python_suffix_as_semantic_contract` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_extension_beside_source` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_source_build_result_records_structured_native_plan` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_custom_wrapper_flags` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_full_direct_build_commands` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_empty_source_list` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_makefile_verbose_combination` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_missing_source` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/build_from_source/test_compiler_verbose.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_runtime_abi.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_source_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines | `legacy` | +| `tests/wrapper/fortran/callbacks/test_array_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/callbacks/test_callback_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/callbacks/test_derived_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_type_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native-call projections | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `not-applicable` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | direct wrapper/build route | external symbols/native linkage; naming/visibility/dispatch | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bridge_uses_explicit_interface_and_no_module_use` | direct wrapper/build route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_fixed_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_free_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_generated_external_contracts_are_non_empty_root_fragments` | direct wrapper/build route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | direct wrapper/build route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_namespace_imported_module_rejects_external_marker_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_package_entry_rejects_non_external_root_declaration_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | +| `tests/wrapper/fortran/function_calls/test_function_call_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | direct wrapper/build route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | +| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | +| `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_generated_contract_build_matches_source_runtime_and_link_order` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_modified_entry_preserves_modules_and_adds_documented_alias` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_pyi_out_writes_one_flat_combined_package` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | +| `tests/wrapper/fortran/naming/test_defined_operators.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; operators; generic dispatch | `legacy` | +| `tests/wrapper/fortran/naming/test_generic_interfaces.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; generic dispatch | `legacy` | +| `tests/wrapper/fortran/naming/test_naming_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::*` | full BLAS/LAPACK wrapper generation unit | external symbols/native linkage; build/compile/link orchestration; broad wrapper corpus | `deferred-real-library` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_duplicate_native_definitions_report_linker_error` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_imported_contracts_resolve_from_one_archive_or_shared_library[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_incompatible_native_artifact_reports_linker_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_module_directory_reports_compile_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_symbol_reports_native_link_or_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_mixed_module_external_bundle_resolves_all_native_input_kinds` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_required_transitive_named_library_resolves_runtime_symbol` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_dependency_order_resolves_transitive_library` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_groups_resolve_cyclic_archive_dependencies` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_unavailable_dependent_shared_library_reports_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | +| `tests/wrapper/fortran/runtime_behavior/test_openmp_runtime.py::*` | direct wrapper/build route | runtime policies/errors/GIL; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_behavior_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::*` | direct wrapper/build route | runtime policies/errors/GIL | `legacy` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | direct wrapper/build route | scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | direct wrapper/build route | strings | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::*` | source/generated-.pyi parity or parametrized route | strings; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/strings/test_string_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | + +### Phase 0A Current Scalar Baseline + +The first migration baseline is +`tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]`. +It is the smallest existing runtime generation unit that directly covers the +initial scalar lane without arrays, strings, module variables, classes, hidden +native outputs, native result projections, or build-bundle behavior. Its +callables do use the current scalar `@bind(...)`, `@external`, and +`@native_call([Addr(Arg(...))])` contract, so the first scalar lane must treat +bind names, external status, scalar address projection, native-call slot order, +and scalar result ownership as completed post-IR policy. + +That unit must remain atomic: + +- Source mode builds the current source fixture named by `SCALAR_LEGACY_SOURCE` + through the normal source wrapper route. +- Generated-`.pyi` mode first checks the generated semantic contract fixture at + `tests/wrapper/fortran/scalars/contracts/fmath`, compiles the native object, + then builds the same runtime wrapper surface from the checked contract. +- Both modes assert the same public runtime behavior through + `_assert_fmath_examples(...)`: lower-case scalar functions accept scalar + Python/NumPy values and return scalar Python/NumPy-compatible results for + real, integer, complex, and logical families. +- The expected generated artifact set is + `bind_c_fmath_wrapper.f90`, `fmath_wrapper.c`, and `fmath_wrapper.h`, plus + shared runtime support installed by the compilation pipeline. + +This fixture is not Phase 1-only. The Python arguments are scalar value inputs, +but every callable also has a scalar result. The whole generation unit cannot +move to `dual-route` until Phase 1 scalar inputs and Phase 2 scalar result +projection are both represented, validated, emitted, compiled, and compared +against the legacy route. + +#### Current scalar route inventory + +| Current behavior or path | Legacy source owner | Proposed wrapper-plan record/action | Required backend behavior | Baseline evidence | +| --- | --- | --- | --- | --- | +| Complete source or generated-`.pyi` semantic module before runtime generation | `x2py/pipeline/build.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/policy_completion.py`, `x2py/semantics/readiness.py` | Route selector receives one policy-completed generation unit | No route selection before policy completion and readiness-owned blockers | `test_fortran_wrapper_pipeline_builds_importable_extension[*]` | +| Lower policy-completed semantic functions to the current codegen AST | `x2py/semantics/ir2ast.py::semantic_ir_to_codegen_ast` | `WrapperPlanner` copies completed module/function/argument/result policy into `ModulePlan`, `FunctionPlan`, `ArgumentTransferPlan`, and later `ResultPlan` | Planner must not infer scalar behavior from datatype or intent | same scalar fixture plus Phase 0C policy tests | +| Python scalar argument conversion dispatches from completed policy | `x2py/codegen/bindings/c_to_python.py::CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER`; `_convert_python_scalar_value_argument` | `ArgumentTransferPlan.python_action = PythonBarrierAction.SCALAR_VALUE` | Isolated binding handler keeps the audited scalar-value conversion behavior and records the binding-to-bridge value handoff | same scalar fixture | +| Bridge scalar address-projected argument dispatches from completed policy | `x2py/codegen/bridges/fortran_to_c.py::FortranToCBridgeGenerator._NATIVE_BARRIER_DISPATCHER`; `_convert_native_call_local_address_argument` | `ArgumentTransferPlan.native_action = NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS` and bridge ABI call-local address slot | Isolated bridge handler passes the address of call-local scalar storage into the native call without storage/writeback policy inference | same scalar fixture | +| Scalar native call uses completed `@native_call([Addr(Arg(...))])` order | `FortranToCBridgeGenerator._visit_FunctionDef` after `_convert_function_arguments(...)` and native projection lowering | `NativeCallPlan` with deterministic native slots copied from completed scalar native-call slot policy | Validator rejects duplicate, missing, or reordered slots not justified by completed policy | same scalar fixture; hidden/native-only projections remain later lanes | +| Scalar result is returned to Python | legacy bridge result conversion and `CPythonBindingGenerator` result wrapping paths | Phase 2 `ResultPlan` and Python result projection action | Result creation consumes a produced native result; Phase 1 must not fake this as input support | same scalar fixture; Phase 2 expansion owns details | +| Generate bridge and binding source/header artifacts | `x2py/codegen/binding_pipeline.py::BindingPipeline.generate` and `BindingPipeline.write`; `FCodePrinter`; `CPythonCodePrinter` | Phase 0E/1A isolated module/header assembly and printers | New route emits complete Fortran bridge, C binding source, C header, additional imports, and runtime requirements before compilation | expected artifact names asserted by the scalar fixture | +| Compile and link the importable extension | `x2py/compiling/python_wrapper.py::create_shared_library` | Shared generated-wrapper artifact handoff reused by both routes | Compilation/link orchestration stays shared; no new route-specific build policy in emitters | existing wrapper build assertions | + +#### Decorator and feature audit reconciliation + +The live wrapper suite currently covers direct scalar calls, `@bind(...)`, +`@external`, `@hold_gil`, `@native_call` argument and result projections, +typed hidden literals, `@raises(...)`, optional and presence-token behavior, +strings, ordinary arrays, native array handles/descriptors, module variables, +derived types, snapshots, constructors, methods, properties, overloads, +generic dispatch, visibility/naming policy, callbacks, multiple-source builds, +semantic-`.pyi` replay, native bundles, and full BLAS/LAPACK real-library +corpora. The migration matrix above reconciles those features to the broad +lanes in the existing phase order. + +No accepted decorator or native projection is considered migrated by this +audit. Rows remain `legacy` unless their complete generation unit has passed +the required dual-route parity evidence. The full BLAS/LAPACK corpus remains +`deferred-real-library`; general native-bundle tests stay active and legacy +because they cover shared build mechanics independently of the full real +library corpus. + +#### Maintained first-lane wrapper-plan contract + +For the selected scalar baseline, the planned records must expose these phases +without backend policy inference: + +- Python surface: one `FunctionPlan` per public scalar function, lower-case + Python name, ordered Python scalar value arguments, and one scalar result. +- Binding handoff: each scalar argument has one `ArgumentTransferPlan` using + `PythonBarrierAction.SCALAR_VALUE`, the existing scalar value conversion + behavior, and one symbolic value handoff to the bridge. +- Bridge ABI: each transfer consumes that value handoff through the completed + scalar native action. The selected scalar fixture uses + `NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS` because every argument is + address-projected with `Addr(Arg(...))`. +- Native call: each scalar function has completed native-call slots copied from + `@native_call([Addr(Arg(...))])`, including native position, Python argument + position, Python/native names, value kind, and the native barrier action. + Implicit declared-order scalar calls may use the same policy shape, but they + are not required for the selected baseline evidence. +- Result projection: scalar native results are represented only in Phase 2 + records; Phase 1 planning may record that the selected baseline is blocked + by scalar result support but must not emit an incomplete runtime wrapper. +- Cleanup and writeback: scalar value inputs have no writeback, destructor, or + ownership transfer phase. Backend-local error cleanup remains private to the + binding/bridge implementation methods and is not a plan policy decision. +- Node construction and printing: Phase 0E may reuse or rewrite only the + dependency-closed legacy node, header/source assembly, printer, and helper + behavior required by the scalar baseline. It must not import, alias, + subclass, or adapt legacy codegen model classes in `x2py.wrapper_codegen`. +- Legacy entrypoint: the independent baseline entrypoint is the existing + `semantic_ir_to_codegen_ast()` route into `BindingPipeline` and + `create_shared_library()`. Phase 0A records it only; it does not modify + legacy lowering, nodes, generators, printers, or compilation. + +### Phase 0B Isolated Package Contract + +Phase 0B introduces infrastructure only. It does not add a route selector, +planner, backend nodes, emitters, printers, compilation, or production +wrapper-plan entrypoint. + +The implemented package boundary is: + +- `x2py.wrapper_codegen` is an isolated package. It may depend on stable Python + infrastructure and shared semantic/pipeline value objects when needed later, + but it must not import `x2py.codegen`. +- Legacy `x2py.codegen` must not import `x2py.wrapper_codegen`. +- A source module that imports both route families must live under + `x2py.pipeline`, because route selection and shared orchestration belong + there. No current production module imports both. +- The package currently exports only its independent `ClassVisitor` protocol + and unsupported-node error. Production wrapper builds do not import the + package. + +The independent visitor contract is intentionally smaller than the legacy +utility visitor: + +- dispatch is through `visit(node, ...)`; +- handler names are deterministic `_` lookups over the + node class MRO; +- the default prefix is `_visit`, with an instance-level override for renderer + or emitter protocols; +- missing support raises `UnsupportedWrapperCodegenNodeError` with the visitor + type, node type, and prefix. + +The blocking `x2py.wrapper_codegen.checks` package checker enforces Phase 0B +static contracts before any emitter handlers exist: + +- wrapper-codegen production modules must not import `x2py.codegen`; +- production module-level functions are rejected so generation behavior stays + on owning classes; +- production `Analyzer`, `Emitter`, `Planner`, `Renderer`, and `Validator` + classes must inherit `ClassVisitor`; +- each function/method must stay within the wrapper-codegen complexity, + statement-count, and nesting limits; +- class-level `*_REGISTRY`, `*_DISPATCHER`, and `*_HANDLERS` mappings that + name handler methods must point at methods on the same class, including + nested secondary dispatch dictionaries; +- registered handlers and handler-like methods may not call source printers + directly through `doprint(...)` or `write(...)`. + +The pipeline-owned generated-wrapper handoff is +`x2py.pipeline.wrapper_artifacts.GeneratedWrapperArtifacts`. It records only +generated wrapper source files, generated headers, the generated module name, +and runtime-support requirement keys. Native source objects, prebuilt native +artifacts, libraries, include directories, library directories, link order, and +compile/link execution remain owned by the existing build plan and compiler +orchestration. + +No new runtime helper API is introduced in Phase 0B. Later lanes may share an +existing runtime helper only when the behavior is unchanged. If the +wrapper-plan route needs different runtime behavior, that helper must have a +separately named generated caller and cleanup contract recorded in the relevant +lane before use. + +CPython reference-counting conventions, new/borrowed/stolen-reference rules, +`Py_INCREF`/`Py_DECREF`, and partial-failure cleanup remain binding-emitter +implementation mechanics. They are absent from plan models, rendered plans, +and cross-backend validation. + +### Phase 0C Scalar Policy Completion Contract + +Phase 0C is a semantic policy phase only. It adds no wrapper-plan records, +planner, backend nodes, emitters, printers, compilation path, or route +selection. + +The selected existing scalar semantic fixture is +`tests/wrapper/fortran/scalars/contracts/fmath/__init__.pyi`. Its current +contract surface is: + +- public module functions only; no methods, classes, overloads, module + variables, arrays, strings, hidden outputs, optional arguments, result + projections, build bundles, or runtime helper selection; +- every callable is marked `@external`; +- every callable has a `@bind(...)` native symbol and lower-case Python + function name; +- every Python argument is a primitive scalar value, completed by policy as + `PythonBarrierAction.SCALAR_VALUE`; +- every native argument slot is `Addr(Arg(i))`, completed by policy as an + address-projected scalar call-local value with + `NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`; +- every callable has one scalar value result, completed by policy as + `CodegenAction.DIRECT_VALUE`, `PythonBarrierAction.NONE`, and + `NativeBarrierAction.NONE`; +- scalar inputs have no writeback, release, or public setter phase; backend + cleanup of call-local temporaries remains local to the eventual selected + handler. + +Post-IR policy completion must attach one typed +`ScalarWrapperFunctionPolicy` to each semantic function. A supported scalar +record contains: + +- stable owner path, Python name, native symbol, external flag, and bind target; +- one `ScalarWrapperArgumentPolicy` per declared Python argument, including + owner path, Python position, native position, Python/native names, + semantic scalar type name, rank, optional flag, completed ownership decision, + completed codegen action, completed Python barrier action, completed native + barrier action, storage mode, boundary storage mode, projection flag, and + Python visibility; +- one `ScalarWrapperResultPolicy` for scalar results, including owner path, + semantic scalar type name, rank, completed ownership decision, completed + codegen action, completed Python barrier action, completed native barrier + action, storage mode, and boundary storage mode; +- one `ScalarWrapperNativeCallSlotPolicy` per native slot, including native + position, source kind (`implicit` or `projection`), Python position, + Python/native names, projection value kind, owner path, completed native + barrier action, and completed codegen action; +- empty writeback, cleanup, and release action tuples for the selected scalar + fixture. + +Unsupported functions still receive a typed scalar-wrapper policy record with +`supported=False` and stable blocker text. Missing policy or blocked policy +must fail through the semantic accessor before planning starts. Later planner +code may consume the typed record, but it must not inspect bridge, binding, or +backend-local implementation state to recover any field listed above. + +### Phase 0D Hierarchical Plan-Core Contract + +Phase 0D converts completed first-lane scalar policy into route-neutral wrapper +plans only. It adds no backend nodes, source emitters, source printers, +compilation path, route selector, fallback route, or production wrapper-plan +entrypoint. + +The plan-core package owns these records: + +- `ModulePlan`: one generation-unit plan with the module owner path, tuple of + function plans, and the completed action-handler registry used by validation + and rendering. +- `FunctionPlan`: one semantic function owner with Python/native names, + external/bind metadata, tuple of argument transfer plans, one optional result + plan, one bridge ABI plan, available symbolic roles, and explicit writeback, + cleanup, and release lifecycle tuples. +- `ArgumentTransferPlan`: the single cross-boundary transfer record for one + argument. It contains the completed `PythonBarrierAction`, one binding + handoff, one bridge ABI slot, the completed `NativeBarrierAction`, and one + native-call slot reference. Binding and bridge subplans must not be split into + separate argument policies. +- `BridgeAbiPlan` and `BridgeAbiSlotPlan`: deterministic bridge ABI positions + and symbolic roles copied from completed scalar policy. +- `NativeCallSlotPlan`: deterministic native-call slot references copied from + completed scalar native-call slot policy. No native slot may be invented from + datatype, `intent`, or backend-local state. +- `ResultPlan` and `LifecycleActionPlan`: symbolic result/writeback/cleanup + consumers. The selected scalar baseline has a scalar result and empty + lifecycle tuples. +- `HandlerRegistryPlan`: primary Python-barrier handlers, secondary + native-barrier handlers, and result handlers keyed by existing completed + `PythonBarrierAction`, `NativeBarrierAction`, and `CodegenAction` values. The + plan core must not define duplicate action enums. + +`WrapperPlanner(ClassVisitor)` owns deterministic structural wiring: + +- `_visit_SemanticModule` returns one `ModulePlan` and recursively visits only + module functions after the support analyzer confirms the whole module is in + the completed first scalar lane. +- `_visit_SemanticFunction` consumes + `completed_scalar_wrapper_policy(function)` and returns one `FunctionPlan`. + Missing or blocked scalar-wrapper policy is an error before planning. +- `_visit_ScalarWrapperArgumentPolicy` and + `_visit_ScalarWrapperResultPolicy` copy only completed policy values into + transfer/result records and attach handler names from the class-owned + registries. + +`WrapperPlanValidator(ClassVisitor)` owns pre-emission invariants: + +- unknown primary Python action handlers, unknown secondary native action + handlers, and unknown result handlers; +- missing bridge ABI slots or missing native-call slots on argument transfers; +- inconsistent handoff, bridge-slot, and native-slot symbolic roles; +- duplicate symbolic roles within one function; +- results, writebacks, cleanups, or releases consuming unavailable symbolic + values. + +`WrapperPlanSupportAnalyzer(ClassVisitor)` reports whole-generation-unit +eligibility with stable owner paths and reasons. It does not select routes. + +`WrapperPlanRenderer(ClassVisitor)` renders deterministic maintainer text with +owner paths, completed policy action values, handler names, symbolic handoffs, +bridge ABI slots, native slots, and lifecycle order. It must not render backend +nodes, C/Fortran source, CPython reference-counting mechanics, or printer +output. + ### Intermediate Test Contract Add intermediate tests only where they protect a stable boundary or a failure @@ -1044,118 +1485,118 @@ value review rather than continuing automatically. ### Phase 0A — Wrapper Test Inventory And Current Scalar Baseline -- [ ] Enumerate every Python test node under `tests/wrapper` and add the +- [x] Enumerate every Python test node under `tests/wrapper` and add the migration matrix required above. Do not begin implementation with untracked wrapper tests. -- [ ] Classify each test as non-generating or map it to its complete wrapper +- [x] Classify each test as non-generating or map it to its complete wrapper generation unit and all semantic lanes needed before that unit can use the wrapper-plan route. -- [ ] Select the first existing generation unit whose coverage best matches the +- [x] Select the first existing generation unit whose coverage best matches the initial scalar lane. Record every additional feature in that unit that delays atomic route eligibility; do not replace it with a new narrower fixture. -- [ ] Inventory the current end-to-end wrapper behavior and implementation paths +- [x] Inventory the current end-to-end wrapper behavior and implementation paths for the first scalar lane, including lowering branches, binding/bridge helper methods, CPython/NumPy API primitives, source printers, generated artifacts, build integration, and focused runtime fixtures. -- [ ] Create a maintained baseline matrix mapping each first-lane current code +- [x] Create a maintained baseline matrix mapping each first-lane current code path and observable behavior to its proposed plan action, handler, required backend behavior, and parity evidence. Do not define an action from a hypothetical implementation. -- [ ] Audit every decorator, native-call projection kind, implicit call +- [x] Audit every decorator, native-call projection kind, implicit call behavior, and generated module/class feature accepted by the live semantic model; reconcile the coverage matrix with that audit. -- [ ] Confirm the legacy generator's independent entrypoint and baseline tests; +- [x] Confirm the legacy generator's independent entrypoint and baseline tests; do not modify legacy lowering, nodes, generators, or printers in this phase. -- [ ] Update the maintained wrapper-plan contract with the audited Python +- [x] Update the maintained wrapper-plan contract with the audited Python surface, binding handoff, bridge ABI, native call, result, cleanup, writeback, node-construction, and printing phases. ### Phase 0B — Isolated Package Boundary -- [ ] Create the `x2py.wrapper_codegen` package skeleton without connecting it +- [x] Create the `x2py.wrapper_codegen` package skeleton without connecting it to production build selection. -- [ ] Define and enforce the package boundary: `x2py.wrapper_codegen` cannot +- [x] Define and enforce the package boundary: `x2py.wrapper_codegen` cannot import `x2py.codegen`, legacy `x2py.codegen` cannot import `x2py.wrapper_codegen`, and only pipeline orchestration may eventually import both route entrypoints. -- [ ] Add dependency tests for that boundary before isolated backend code is +- [x] Add dependency tests for that boundary before isolated backend code is introduced. -- [ ] Implement and test the minimal independent `ClassVisitor` used throughout +- [x] Implement and test the minimal independent `ClassVisitor` used throughout the package, including deterministic MRO lookup, configurable method prefixes, and explicit unsupported-node failure. -- [ ] Add structural checks requiring visitor-based traversal and rejecting +- [x] Add structural checks requiring visitor-based traversal and rejecting undeclared module-level production functions in `x2py.wrapper_codegen`. -- [ ] Add the blocking wrapper-codegen complexity/traceability checker before +- [x] Add the blocking wrapper-codegen complexity/traceability checker before emitter handlers are introduced. Cover Radon complexity, statement count, nesting depth, registry completeness, secondary-dispatch completeness, and forbidden printer calls from handlers. -- [ ] Add focused tests for the checker, including one failure fixture for each +- [x] Add focused tests for the checker, including one failure fixture for each enforced limit and registry/dependency rule. -- [ ] Define the pipeline-owned generated-wrapper artifact result shared by both +- [x] Define the pipeline-owned generated-wrapper artifact result shared by both routes without duplicating native object/library/link-plan ownership. -- [ ] Keep runtime helper APIs shared only when their behavior is unchanged. Add +- [x] Keep runtime helper APIs shared only when their behavior is unchanged. Add separately named new-route helpers when different behavior is required, and record their generated callers and cleanup contract. -- [ ] Document that CPython reference counting and API ownership conventions are +- [x] Document that CPython reference counting and API ownership conventions are binding-emitter-local mechanics and are absent from plan models, rendered plans, and cross-backend plan validation. ### Phase 0C — Complete Scalar Policy -- [ ] Audit the selected existing scalar generation unit and list every +- [x] Audit the selected existing scalar generation unit and list every module/function/argument/result/decorator/native-projection decision the planner would otherwise need to infer. -- [ ] Define or complete typed post-IR policy records for the owners needed by +- [x] Define or complete typed post-IR policy records for the owners needed by the first scalar lane. Do not use free-form metadata or planner defaults as a substitute for a completed policy field. -- [ ] Move any remaining scalar action, call/ABI order, ownership, lifecycle, +- [x] Move any remaining scalar action, call/ABI order, ownership, lifecycle, projection, writeback, or cleanup decisions into `complete_semantic_policies(...)` before implementing the planner. -- [ ] Verify the policy-completed module contains every datatype fact and +- [x] Verify the policy-completed module contains every datatype fact and completed policy value needed to reproduce the audited legacy behavior without reading bridge, binding, or backend-local state. -- [ ] Add only the intermediate policy tests required by the contract above, +- [x] Add only the intermediate policy tests required by the contract above, reusing the selected existing semantic fixture and covering missing/blocked policy failure before planning. -- [ ] Do not add plan models, backend nodes, emitters, printers, compilation, or +- [x] Do not add plan models, backend nodes, emitters, printers, compilation, or production route selection in this phase. ### Phase 0D — Hierarchical Wrapper Plan Core -- [ ] Define the first frozen plan data classes and tuple collections. -- [ ] Implement `WrapperPlanner(ClassVisitor)` with explicit `_visit_` +- [x] Define the first frozen plan data classes and tuple collections. +- [x] Implement `WrapperPlanner(ClassVisitor)` with explicit `_visit_` methods that each return one plan record, recursively visit only that owner's children, and perform only deterministic structural wiring. Keep each method within the strict planner complexity gate. -- [ ] Define one `ArgumentTransferPlan` containing the existing completed Python +- [x] Define one `ArgumentTransferPlan` containing the existing completed Python action, one binding-to-bridge handoff, bridge ABI slot, completed native action, and native-call slot. Do not create separate binding and bridge subplans for one argument. -- [ ] Define `BridgeAbiPlan`, native-call refs, handoff specs, primary/secondary +- [x] Define `BridgeAbiPlan`, native-call refs, handoff specs, primary/secondary handler registries, and validation errors around existing completed `PythonBarrierAction` and `NativeBarrierAction` values. Do not add duplicate plan action enums for behavior already represented by completed policy. -- [ ] Add a plan validator that catches inconsistent transfer handoffs, +- [x] Add a plan validator that catches inconsistent transfer handoffs, missing bridge/native-call slots, unknown primary or secondary handlers, duplicate symbolic roles, and writebacks/results that consume unavailable values. -- [ ] Define the whole-generation-unit support report, including stable +- [x] Define the whole-generation-unit support report, including stable owner-path reasons for unsupported elements, without changing production route selection yet. -- [ ] Implement class-owned `WrapperPlanSupportAnalyzer`, +- [x] Implement class-owned `WrapperPlanSupportAnalyzer`, `WrapperPlanValidator`, and `WrapperPlanRenderer` visitor APIs; do not add equivalent module-level functions. -- [ ] Add deterministic plan rendering that includes symbolic owner paths, +- [x] Add deterministic plan rendering that includes symbolic owner paths, completed policy values, dispatch handler names, handoffs, bridge ABI slots, native slots, and lifecycle order without backend nodes or CPython-specific mechanics. -- [ ] Verify the planner fails on missing/incomplete policy rather than deriving +- [x] Verify the planner fails on missing/incomplete policy rather than deriving defaults, and the validator produces owner-path diagnostics before node emission for policy/plan inconsistency. -- [ ] Add one policy-to-plan projection/rendering test for the selected existing +- [x] Add one policy-to-plan projection/rendering test for the selected existing scalar fixture and table-driven validator tests by invariant category. Do not add one test per planner method or plan field. -- [ ] Do not add backend nodes, emitters, printers, compilation, or production +- [x] Do not add backend nodes, emitters, printers, compilation, or production route selection in this phase. ### Phase 0E — Minimal Scalar Backend Foundation diff --git a/tests/semantics/policy/test_scalar_wrapper_policy.py b/tests/semantics/policy/test_scalar_wrapper_policy.py new file mode 100644 index 000000000..bec254f88 --- /dev/null +++ b/tests/semantics/policy/test_scalar_wrapper_policy.py @@ -0,0 +1,124 @@ +from pathlib import Path + +import pytest + +from x2py.pipeline.pyi import pyi_file_to_semantic_module +from x2py.semantics.models import ( + RESOLVED_SCALAR_WRAPPER_POLICY_METADATA, + SemanticFunction, + SemanticType, +) +from x2py.semantics.ownership import ( + CodegenAction, + NativeBarrierAction, + ObjectKind, + PythonBarrierAction, + StorageMode, +) +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.scalar_wrapper_policy import ( + ScalarWrapperFunctionPolicy, + completed_scalar_wrapper_policy, +) + +from tests._shared.ownership_policy_support import parse_pyi_text + + +FMATH_CONTRACT = Path("tests/wrapper/fortran/scalars/contracts/fmath/__init__.pyi") + + +def test_fmath_fixture_gets_completed_scalar_wrapper_policy(): + module = pyi_file_to_semantic_module(FMATH_CONTRACT, module_name="fmath") + + complete_semantic_policies(module) + + policies = [function.metadata[RESOLVED_SCALAR_WRAPPER_POLICY_METADATA] for function in module.functions] + assert policies + assert all(isinstance(policy, ScalarWrapperFunctionPolicy) for policy in policies) + assert all(policy.supported for policy in policies) + assert all(policy.blockers == () for policy in policies) + assert all(policy.writeback_actions == () for policy in policies) + assert all(policy.cleanup_actions == () for policy in policies) + assert all(policy.release_actions == () for policy in policies) + + +def test_fmath_scalar_policy_records_address_projected_call_slots(): + module = pyi_file_to_semantic_module(FMATH_CONTRACT, module_name="fmath") + complete_semantic_policies(module) + function = next(item for item in module.functions if item.name == "add_r8") + + policy = completed_scalar_wrapper_policy(function) + + assert policy.owner_path == "fmath.add_r8" + assert policy.python_name == "add_r8" + assert policy.native_name == "ADD_R8" + assert policy.external is True + assert policy.bind_target == "ADD_R8" + + assert [argument.name for argument in policy.arguments] == ["X", "Y"] + assert [argument.python_position for argument in policy.arguments] == [0, 1] + assert [argument.native_position for argument in policy.arguments] == [0, 1] + for argument in policy.arguments: + assert argument.semantic_type_name == "Float64" + assert argument.rank == 0 + assert argument.optional is False + assert argument.ownership.kind is ObjectKind.SCALAR + assert argument.codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert argument.python_barrier_action is PythonBarrierAction.SCALAR_VALUE + assert argument.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert argument.storage_mode is StorageMode.ALIAS + assert argument.boundary_storage_mode is StorageMode.ALIAS + assert argument.projects_result is False + assert argument.python_visible is True + + assert [(slot.native_position, slot.python_position) for slot in policy.native_call_slots] == [ + (0, 0), + (1, 1), + ] + assert [slot.source_kind for slot in policy.native_call_slots] == ["projection", "projection"] + assert [slot.value_kind for slot in policy.native_call_slots] == ["addr", "addr"] + assert [slot.native_name for slot in policy.native_call_slots] == ["X", "Y"] + assert all( + slot.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS for slot in policy.native_call_slots + ) + + assert policy.result is not None + assert policy.result.owner_path == "fmath.add_r8.return" + assert policy.result.semantic_type_name == "Float64" + assert policy.result.rank == 0 + assert policy.result.ownership.kind is ObjectKind.SCALAR + assert policy.result.codegen_action is CodegenAction.DIRECT_VALUE + assert policy.result.python_barrier_action is PythonBarrierAction.NONE + assert policy.result.native_barrier_action is NativeBarrierAction.NONE + assert policy.result.storage_mode is StorageMode.STACK + assert policy.result.boundary_storage_mode is StorageMode.STACK + + +def test_scalar_wrapper_policy_blocks_non_scalar_arguments_before_planning(): + module = parse_pyi_text( + """ +def sum_values(values: Float64[:]) -> Float64: ... +""", + module_name="array_argument", + ) + complete_semantic_policies(module) + function = module.functions[0] + policy = function.metadata[RESOLVED_SCALAR_WRAPPER_POLICY_METADATA] + + assert isinstance(policy, ScalarWrapperFunctionPolicy) + assert policy.supported is False + assert "argument 'values' is not a first-lane primitive scalar" in policy.blockers + + with pytest.raises(ValueError, match="blocked scalar wrapper policy"): + completed_scalar_wrapper_policy(function) + + +def test_missing_scalar_wrapper_policy_fails_before_planning(): + function = SemanticFunction( + name="add", + arguments=[], + return_type=SemanticType(name="Float64", dtype="Float64"), + ) + + with pytest.raises(ValueError, match="missing completed scalar wrapper policy"): + completed_scalar_wrapper_policy(function) diff --git a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py index 1e7940546..63c902244 100644 --- a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py +++ b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py @@ -6,6 +6,8 @@ from collections import Counter from pathlib import Path import re +import subprocess +import sys from tests.wrapper.fortran._support import REPO_ROOT, WRAPPER_FORTRAN_DATA, WRAPPER_TEST_ROOT @@ -13,6 +15,7 @@ WRAPPER_SUITE_ROOT = WRAPPER_ROOT.parent DOCS_ROOT = REPO_ROOT / "docs" CHECKLIST_COVERAGE = WRAPPER_SUITE_ROOT / "CHECKLIST_COVERAGE.md" +WRAPPER_PLAN_MIGRATION_CHECKLIST = DOCS_ROOT / "maintainer" / "roadmap" / "wrapper-plan-migration-checklist.md" FORTRAN_SUFFIXES = {".f", ".f90", ".f95", ".for"} ROOT_FILES = { "README.md", @@ -113,6 +116,17 @@ SUBJECT_TEST_PATHS = tuple( f"{subject}/{filename}" for subject, filenames in SUBJECT_TEST_MODULES.items() for filename in filenames ) +MIGRATION_MATRIX_STATUS_VALUES = { + "not-applicable", + "deferred-real-library", + "legacy", + "dual-route", + "wrapper-plan", +} +MIGRATION_MATRIX_ROW_RE = re.compile( + r"^\| `(?Ptests/wrapper/[^`]+)` \| (?P[^|]+) " + r"\| (?P[^|]+) \| `(?P[^`]+)` \|$" +) def _is_meaningful(path: Path) -> bool: @@ -146,6 +160,47 @@ def _docs_and_test_text_paths() -> list[Path]: ) +def _collected_wrapper_test_nodes() -> list[str]: + result = subprocess.run( + [ + sys.executable, + "-m", + "pytest", + "--collect-only", + "-q", + "tests/wrapper", + ], + cwd=REPO_ROOT, + check=True, + capture_output=True, + text=True, + ) + return sorted(line for line in result.stdout.splitlines() if line.startswith("tests/wrapper/") and "::" in line) + + +def _wrapper_plan_migration_matrix_rows() -> dict[str, dict[str, str]]: + rows = {} + for line in WRAPPER_PLAN_MIGRATION_CHECKLIST.read_text(encoding="utf-8").splitlines(): + match = MIGRATION_MATRIX_ROW_RE.match(line) + if match is None: + continue + selector = match.group("selector") + rows[selector] = { + "unit": match.group("unit").strip(), + "lanes": match.group("lanes").strip(), + "status": match.group("status"), + } + return rows + + +def _migration_selector_matches(selector: str, nodeid: str) -> bool: + if selector.endswith("::*"): + return nodeid.startswith(f"{selector[:-3]}::") + if selector.endswith("[*]"): + return nodeid.startswith(f"{selector[:-3]}[") + return nodeid == selector + + def test_fortran_wrapper_tree_uses_only_allowed_subjects(): missing_subjects = [subject for subject in ALLOWED_SUBJECTS if not (WRAPPER_ROOT / subject).is_dir()] assert missing_subjects == [] @@ -282,6 +337,41 @@ def test_wrapper_checklist_python_evidence_references_existing_test_nodes(): assert missing == [] +def test_wrapper_plan_migration_matrix_tracks_collected_wrapper_nodes(): + matrix_rows = _wrapper_plan_migration_matrix_rows() + assert matrix_rows + + invalid_status_rows = sorted( + selector for selector, row in matrix_rows.items() if row["status"] not in MIGRATION_MATRIX_STATUS_VALUES + ) + assert invalid_status_rows == [] + + incomplete_rows = sorted(selector for selector, row in matrix_rows.items() if not row["unit"] or not row["lanes"]) + assert incomplete_rows == [] + + collected_nodes = _collected_wrapper_test_nodes() + assert collected_nodes + + unmatched_nodes = [] + multiply_matched_nodes = [] + for nodeid in collected_nodes: + matches = [selector for selector in matrix_rows if _migration_selector_matches(selector, nodeid)] + if not matches: + unmatched_nodes.append(nodeid) + elif len(matches) > 1: + multiply_matched_nodes.append((nodeid, matches)) + + stale_selectors = sorted( + selector + for selector in matrix_rows + if not any(_migration_selector_matches(selector, nodeid) for nodeid in collected_nodes) + ) + + assert unmatched_nodes == [] + assert multiply_matched_nodes == [] + assert stale_selectors == [] + + def test_wrapper_language_suite_and_user_guide_link_current_subject_paths(): root_test_modules = sorted(path.name for path in WRAPPER_SUITE_ROOT.glob("test_*.py")) assert root_test_modules == [] diff --git a/tests/wrapper_codegen/test_phase0b_contracts.py b/tests/wrapper_codegen/test_phase0b_contracts.py new file mode 100644 index 000000000..b0de720d0 --- /dev/null +++ b/tests/wrapper_codegen/test_phase0b_contracts.py @@ -0,0 +1,184 @@ +"""Phase 0B boundary tests for the isolated wrapper-codegen package.""" + +from __future__ import annotations + +import ast +from dataclasses import fields +from pathlib import Path + +from tests.wrapper.fortran._support import REPO_ROOT +from x2py.pipeline.wrapper_artifacts import GeneratedWrapperArtifacts +from x2py.wrapper_codegen.checks import ( + WrapperCodegenCheckConfig, + check_wrapper_codegen_package, + check_wrapper_codegen_paths, +) + +SOURCE_ROOT = REPO_ROOT / "x2py" +WRAPPER_CODEGEN_ROOT = SOURCE_ROOT / "wrapper_codegen" +LEGACY_CODEGEN_ROOT = SOURCE_ROOT / "codegen" + + +def _python_modules(root: Path) -> list[Path]: + return sorted(path for path in root.rglob("*.py") if "__pycache__" not in path.parts) + + +def _imported_modules(path: Path) -> set[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + imported = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + if isinstance(node, ast.ImportFrom) and node.module: + imported.add(node.module) + return imported + + +def _imports_under(imports: set[str], package: str) -> bool: + return any(name == package or name.startswith(f"{package}.") for name in imports) + + +def _write_module(root: Path, relative_path: str, source: str) -> Path: + path = root / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return path + + +def _check_source(tmp_path: Path, source: str, *, filename: str = "bad.py") -> set[str]: + path = _write_module(tmp_path, filename, source) + violations = check_wrapper_codegen_paths( + [path], + package_root=tmp_path, + config=WrapperCodegenCheckConfig(max_complexity=3, max_statements=4, max_nesting=2), + ) + return {violation.code for violation in violations} + + +def test_wrapper_codegen_and_legacy_codegen_do_not_import_each_other(): + wrapper_codegen_imports = {path: _imported_modules(path) for path in _python_modules(WRAPPER_CODEGEN_ROOT)} + wrapper_codegen_violations = sorted( + path.relative_to(REPO_ROOT).as_posix() + for path, imports in wrapper_codegen_imports.items() + if _imports_under(imports, "x2py.codegen") + ) + assert wrapper_codegen_violations == [] + + legacy_codegen_imports = {path: _imported_modules(path) for path in _python_modules(LEGACY_CODEGEN_ROOT)} + legacy_codegen_violations = sorted( + path.relative_to(REPO_ROOT).as_posix() + for path, imports in legacy_codegen_imports.items() + if _imports_under(imports, "x2py.wrapper_codegen") + ) + assert legacy_codegen_violations == [] + + +def test_only_pipeline_modules_may_import_both_wrapper_routes(): + modules_importing_both = [] + for path in _python_modules(SOURCE_ROOT): + imports = _imported_modules(path) + if _imports_under(imports, "x2py.codegen") and _imports_under(imports, "x2py.wrapper_codegen"): + modules_importing_both.append(path.relative_to(SOURCE_ROOT).as_posix()) + + outside_pipeline = sorted(path for path in modules_importing_both if not path.startswith("pipeline/")) + assert outside_pipeline == [] + + +def test_wrapper_codegen_package_static_contracts_pass(): + assert check_wrapper_codegen_package(WRAPPER_CODEGEN_ROOT) == () + + +def test_checker_rejects_legacy_codegen_imports(tmp_path: Path): + codes = _check_source(tmp_path, "import x2py.codegen\n") + + assert "legacy-codegen-import" in codes + + +def test_checker_rejects_module_level_production_functions(tmp_path: Path): + codes = _check_source(tmp_path, "def build_plan():\n return None\n") + + assert "module-function" in codes + + +def test_checker_requires_visitor_based_production_classes(tmp_path: Path): + codes = _check_source(tmp_path, "class WrapperPlanner:\n pass\n") + + assert "visitor-class" in codes + + +def test_checker_enforces_complexity_statement_and_nesting_limits(tmp_path: Path): + codes = _check_source( + tmp_path, + """ +def oversized(value): + first = value + 1 + second = first + 1 + third = second + 1 + fourth = third + 1 + if value: + if first: + if second: + return third + if value == 1: + return first + if value == 2: + return second + if value == 3: + return third + return fourth +""", + ) + + assert {"complexity", "statement-count", "nesting-depth"} <= codes + + +def test_checker_rejects_missing_primary_and_secondary_registry_handlers(tmp_path: Path): + codes = _check_source( + tmp_path, + """ +from x2py.wrapper_codegen import ClassVisitor + +class DemoEmitter(ClassVisitor): + PRIMARY_REGISTRY = {"scalar": "_emit_scalar"} + SECONDARY_DISPATCHER = {"scalar": {"value": "_emit_scalar_value"}} +""", + ) + + assert "registry-missing-handler" in codes + + +def test_checker_rejects_printer_calls_from_handlers(tmp_path: Path): + codes = _check_source( + tmp_path, + """ +from x2py.wrapper_codegen import ClassVisitor + +class DemoEmitter(ClassVisitor): + HANDLER_REGISTRY = {"scalar": "_emit_scalar"} + + def _emit_scalar(self, node): + return self.printer.doprint(node) +""", + ) + + assert "handler-printer-call" in codes + + +def test_generated_wrapper_artifacts_keep_compile_and_link_ownership_out_of_the_handoff(): + artifacts = GeneratedWrapperArtifacts( + module_name="demo", + bridge_sources=(Path("bind_c_demo.f90"),), + binding_sources=(Path("demo.c"),), + header_files=(Path("demo.h"),), + runtime_support_keys=("python_runtime",), + ) + + assert artifacts.source_files == (Path("bind_c_demo.f90"), Path("demo.c")) + assert artifacts.generated_files == (Path("bind_c_demo.f90"), Path("demo.c"), Path("demo.h")) + assert {field.name for field in fields(GeneratedWrapperArtifacts)} == { + "module_name", + "bridge_sources", + "binding_sources", + "header_files", + "runtime_support_keys", + } diff --git a/tests/wrapper_codegen/test_phase0d_plan_core.py b/tests/wrapper_codegen/test_phase0d_plan_core.py new file mode 100644 index 000000000..39e2b5224 --- /dev/null +++ b/tests/wrapper_codegen/test_phase0d_plan_core.py @@ -0,0 +1,203 @@ +"""Phase 0D tests for route-neutral wrapper-plan core records.""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path + +import pytest + +from x2py.pipeline.pyi import pyi_file_to_semantic_module +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, PythonBarrierAction +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.wrapper_codegen import ( + ArgumentTransferPlan, + FunctionPlan, + LifecycleActionPlan, + ModulePlan, + WrapperPlanRenderer, + WrapperPlanSupportAnalyzer, + WrapperPlanValidator, + WrapperPlanner, +) + +from tests._shared.ownership_policy_support import parse_pyi_text + + +FMATH_CONTRACT = Path("tests/wrapper/fortran/scalars/contracts/fmath/__init__.pyi") + + +def _fmath_plan() -> ModulePlan: + module = pyi_file_to_semantic_module(FMATH_CONTRACT, module_name="fmath") + complete_semantic_policies(module) + return WrapperPlanner().visit(module) + + +def _function(plan: ModulePlan, owner_path: str = "fmath.add_r8") -> FunctionPlan: + return next(function for function in plan.functions if function.owner_path == owner_path) + + +def _replace_function(plan: ModulePlan, function: FunctionPlan) -> ModulePlan: + functions = tuple(function if item.owner_path == function.owner_path else item for item in plan.functions) + return replace(plan, functions=functions) + + +def _replace_argument(function: FunctionPlan, argument: ArgumentTransferPlan) -> FunctionPlan: + arguments = tuple(argument if item.owner_path == argument.owner_path else item for item in function.arguments) + slots = tuple(item.bridge_abi_slot for item in arguments if item.bridge_abi_slot is not None) + return replace(function, arguments=arguments, bridge_abi=replace(function.bridge_abi, slots=slots)) + + +def _with_first_argument(plan: ModulePlan, argument: ArgumentTransferPlan) -> ModulePlan: + return _replace_function(plan, _replace_argument(_function(plan), argument)) + + +def _drop_primary_handlers(plan: ModulePlan) -> ModulePlan: + registry = replace(plan.handler_registry, python_action_handlers=()) + return replace(plan, handler_registry=registry) + + +def _drop_secondary_handlers(plan: ModulePlan) -> ModulePlan: + registry = replace(plan.handler_registry, native_action_handlers=()) + return replace(plan, handler_registry=registry) + + +def _drop_bridge_slot(plan: ModulePlan) -> ModulePlan: + argument = replace(_function(plan).arguments[0], bridge_abi_slot=None) + return _with_first_argument(plan, argument) + + +def _drop_native_slot(plan: ModulePlan) -> ModulePlan: + argument = replace(_function(plan).arguments[0], native_call_slot=None) + return _with_first_argument(plan, argument) + + +def _break_handoff_role(plan: ModulePlan) -> ModulePlan: + argument = _function(plan).arguments[0] + handoff = replace(argument.binding_handoff, consumed_role="fmath.add_r8.X:other") + return _with_first_argument(plan, replace(argument, binding_handoff=handoff)) + + +def _duplicate_symbolic_role(plan: ModulePlan) -> ModulePlan: + function = _function(plan) + first, second = function.arguments + role = first.binding_handoff.produced_role + handoff = replace(second.binding_handoff, produced_role=role, consumed_role=role) + bridge = replace(second.bridge_abi_slot, symbolic_role=role) + native = replace(second.native_call_slot, symbolic_role=role) + updated = replace(second, binding_handoff=handoff, bridge_abi_slot=bridge, native_call_slot=native) + return _replace_function(plan, _replace_argument(function, updated)) + + +def _remove_result_role(plan: ModulePlan) -> ModulePlan: + function = _function(plan) + roles = tuple(role for role in function.available_roles if role != function.result.native_result_role) + return _replace_function(plan, replace(function, available_roles=roles)) + + +def _add_unavailable_writeback(plan: ModulePlan) -> ModulePlan: + function = _function(plan) + action = LifecycleActionPlan( + owner_path="fmath.add_r8.X", + phase="writeback", + source_role="missing:role", + handler_name="_handle_writeback", + ) + return _replace_function(plan, replace(function, writeback_actions=(action,))) + + +def test_fmath_policy_projects_to_plan_and_deterministic_rendering(): + plan = _fmath_plan() + add_r8 = _function(plan) + + assert WrapperPlanValidator().visit(plan) == () + assert len(plan.functions) == 85 + assert add_r8.python_name == "add_r8" + assert add_r8.native_name == "ADD_R8" + assert add_r8.external is True + assert add_r8.bind_target == "ADD_R8" + assert [argument.owner_path for argument in add_r8.arguments] == ["fmath.add_r8.X", "fmath.add_r8.Y"] + assert [argument.python_action for argument in add_r8.arguments] == [ + PythonBarrierAction.SCALAR_VALUE, + PythonBarrierAction.SCALAR_VALUE, + ] + assert [argument.native_action for argument in add_r8.arguments] == [ + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + ] + assert [slot.index for slot in add_r8.bridge_abi.slots] == [0, 1] + assert [ + (argument.native_call_slot.native_position, argument.native_call_slot.value_kind) + for argument in add_r8.arguments + ] == [ + (0, "addr"), + (1, "addr"), + ] + assert add_r8.result is not None + assert add_r8.result.codegen_action is CodegenAction.DIRECT_VALUE + + first = add_r8.arguments[0] + assert first.binding_handoff.produced_role == "fmath.add_r8.X:value" + assert first.binding_handoff.consumed_role == "fmath.add_r8.X:value" + assert first.bridge_abi_slot.symbolic_role == "fmath.add_r8.X:value" + assert first.native_call_slot.symbolic_role == "fmath.add_r8.X:value" + assert first.native_call_slot.source_kind == "projection" + assert first.native_call_slot.value_kind == "addr" + + rendered = WrapperPlanRenderer().visit(plan) + assert "function fmath.add_r8 python=add_r8 native=ADD_R8" in rendered + assert "python:scalar_value->_handle_python_scalar_value" in rendered + assert "native:pass_call_local_address->_handle_native_call_local_address" in rendered + assert "native_slot=0:addr:fmath.add_r8.X:value" in rendered + assert "lifecycle=none" in rendered + + +def test_support_analyzer_reports_unsupported_generation_units_without_route_selection(): + module = parse_pyi_text( + """ +def sum_values(values: Float64[:]) -> Float64: ... +""", + module_name="array_argument", + ) + complete_semantic_policies(module) + + report = WrapperPlanSupportAnalyzer().visit(module) + + assert report.supported is False + assert report.blockers[0].owner_path == "array_argument.sum_values" + assert "not a first-lane primitive scalar" in report.blockers[0].reason + with pytest.raises(ValueError, match="Unsupported wrapper-plan generation unit"): + WrapperPlanner().visit(module) + + +def test_planner_fails_on_missing_policy_before_deriving_defaults(): + module = parse_pyi_text( + """ +def add(x: Float64, y: Float64) -> Float64: ... +""", + module_name="missing_policy", + ) + + with pytest.raises(ValueError, match="missing completed scalar wrapper policy"): + WrapperPlanner().visit(module) + + +@pytest.mark.parametrize( + ("mutate", "expected_code"), + [ + (_drop_primary_handlers, "unknown-primary-handler"), + (_drop_secondary_handlers, "unknown-secondary-handler"), + (_drop_bridge_slot, "missing-bridge-abi-slot"), + (_drop_native_slot, "missing-native-call-slot"), + (_break_handoff_role, "inconsistent-binding-handoff"), + (_duplicate_symbolic_role, "duplicate-symbolic-role"), + (_remove_result_role, "unavailable-result-role"), + (_add_unavailable_writeback, "unavailable-writeback-role"), + ], +) +def test_validator_reports_invariant_categories_before_backend_emission(mutate, expected_code): + invalid = mutate(_fmath_plan()) + + diagnostics = WrapperPlanValidator().visit(invalid) + + assert expected_code in {diagnostic.code for diagnostic in diagnostics} diff --git a/tests/wrapper_codegen/test_visitor.py b/tests/wrapper_codegen/test_visitor.py new file mode 100644 index 000000000..19aa4a87a --- /dev/null +++ b/tests/wrapper_codegen/test_visitor.py @@ -0,0 +1,56 @@ +"""Tests for the isolated wrapper-codegen visitor protocol.""" + +from __future__ import annotations + +import pytest + +from x2py.wrapper_codegen import ClassVisitor, UnsupportedWrapperCodegenNodeError + + +class BaseNode: + """Base node used to prove MRO dispatch.""" + + +class ChildNode(BaseNode): + """Child node that should use the most specific available handler.""" + + +class UnsupportedNode: + """Node with no matching handler.""" + + +def test_class_visitor_uses_mro_specific_handler(): + class Visitor(ClassVisitor): + def _visit_BaseNode(self, node): + return ("base", type(node).__name__) + + def _visit_ChildNode(self, node): + return ("child", type(node).__name__) + + assert Visitor().visit(ChildNode()) == ("child", "ChildNode") + + +def test_class_visitor_falls_back_to_base_handler(): + class Visitor(ClassVisitor): + def _visit_BaseNode(self, node): + return ("base", type(node).__name__) + + assert Visitor().visit(ChildNode()) == ("base", "ChildNode") + + +def test_class_visitor_supports_configurable_prefix(): + class Visitor(ClassVisitor): + def _render_BaseNode(self, node): + return ("rendered", type(node).__name__) + + assert Visitor(method_prefix="_render").visit(BaseNode()) == ("rendered", "BaseNode") + + +def test_class_visitor_reports_unsupported_nodes(): + visitor = ClassVisitor() + + with pytest.raises(UnsupportedWrapperCodegenNodeError) as exc_info: + visitor.visit(UnsupportedNode()) + + assert "UnsupportedNode" in str(exc_info.value) + assert "_visit" in str(exc_info.value) diff --git a/x2py/pipeline/wrapper_artifacts.py b/x2py/pipeline/wrapper_artifacts.py new file mode 100644 index 000000000..bf481ead5 --- /dev/null +++ b/x2py/pipeline/wrapper_artifacts.py @@ -0,0 +1,29 @@ +"""Generated-wrapper artifact handoff shared by wrapper build routes.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +__all__ = ("GeneratedWrapperArtifacts",) + + +@dataclass(frozen=True) +class GeneratedWrapperArtifacts: + """Generated wrapper files produced before compile/link orchestration.""" + + module_name: str + bridge_sources: tuple[Path, ...] = () + binding_sources: tuple[Path, ...] = () + header_files: tuple[Path, ...] = () + runtime_support_keys: tuple[str, ...] = () + + @property + def source_files(self) -> tuple[Path, ...]: + """Return all generated wrapper sources in compile order.""" + return (*self.bridge_sources, *self.binding_sources) + + @property + def generated_files(self) -> tuple[Path, ...]: + """Return all generated wrapper files, including headers.""" + return (*self.source_files, *self.header_files) diff --git a/x2py/semantics/models.py b/x2py/semantics/models.py index cbeaea692..09a405de4 100644 --- a/x2py/semantics/models.py +++ b/x2py/semantics/models.py @@ -324,6 +324,7 @@ class ProcedureOverloadSet: RESOLVED_GETTER_OWNERSHIP_POLICY_METADATA = "resolved_getter_ownership_policy" RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA = "resolved_setter_ownership_policy" RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA = "resolved_native_array_handle_policy" +RESOLVED_SCALAR_WRAPPER_POLICY_METADATA = "resolved_scalar_wrapper_policy" RESOLVED_MODULE_VARIABLE_INITIALIZER_METADATA = "resolved_module_variable_initializer" MODULE_VARIABLE_INITIALIZER_UNSUPPORTED_BLOCKER = "module_variable_initializer_unsupported" PYTHON_STATIC_METADATA = "python_static" diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index 9b5e6d63d..31deb5f6d 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -30,6 +30,7 @@ ) from x2py.semantics import models from x2py.semantics.native_array_handles import NativeArrayHandlePolicy, native_array_descriptor_kind +from x2py.semantics.scalar_wrapper_policy import build_scalar_wrapper_function_policy __all__ = ("complete_semantic_policies",) @@ -125,12 +126,12 @@ def _complete_ownership_policies(module: models.SemanticModule) -> models.Semant _complete_module_variable_initializer(variable) _block_unsupported_snapshot_contract(variable) for semantic_class in module.classes: - _complete_class(semantic_class) + _complete_class(semantic_class, f"{module.name}.{semantic_class.name}") for function in module.functions: - _complete_function(function) + _complete_function(function, f"{module.name}.{function.name}") for overload_set in module.overload_sets: for procedure in overload_set.procedures: - _complete_function(procedure) + _complete_function(procedure, f"{module.name}.{overload_set.name}.{procedure.name}") module.metadata[models.POLICY_COMPLETION_PREPARED_METADATA] = True return module @@ -162,7 +163,7 @@ def _blocked_snapshot_decision(decision: OwnershipDecision) -> OwnershipDecision ) -def _complete_class(semantic_class: models.SemanticClass) -> None: +def _complete_class(semantic_class: models.SemanticClass, owner_path: str) -> None: class_type = models.SemanticType(name=semantic_class.name, dtype=semantic_class.name) semantic_class.metadata[models.RESOLVED_CLASS_INSTANCE_POLICY_METADATA] = ( default_ownership_policy.decide_semantic_type(class_type, OwnershipContext.result()) @@ -175,15 +176,15 @@ def _complete_class(semantic_class: models.SemanticClass) -> None: _complete_variable(field, OwnershipContext.field()) _complete_accessor_policies(field, OwnershipContext.field()) for nested in semantic_class.classes: - _complete_class(nested) + _complete_class(nested, f"{owner_path}.{nested.name}") for method in semantic_class.methods: - _complete_function(method) + _complete_function(method, f"{owner_path}.{method.name}") for overload_set in semantic_class.overload_sets: for procedure in overload_set.procedures: - _complete_function(procedure) + _complete_function(procedure, f"{owner_path}.{overload_set.name}.{procedure.name}") -def _complete_function(function: models.SemanticFunction) -> None: +def _complete_function(function: models.SemanticFunction, owner_path: str) -> None: _complete_callable_address_policy(function) for argument in function.arguments: _complete_variable(argument, ownership_context_for_argument(function, argument)) @@ -194,6 +195,10 @@ def _complete_function(function: models.SemanticFunction) -> None: else: function.metadata.pop(models.RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA, None) function.metadata.pop(models.RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA, None) + function.metadata[models.RESOLVED_SCALAR_WRAPPER_POLICY_METADATA] = build_scalar_wrapper_function_policy( + function, + owner_path=owner_path, + ) def _complete_native_array_handle_result_policy( diff --git a/x2py/semantics/scalar_wrapper_policy.py b/x2py/semantics/scalar_wrapper_policy.py new file mode 100644 index 000000000..0e7558bfa --- /dev/null +++ b/x2py/semantics/scalar_wrapper_policy.py @@ -0,0 +1,403 @@ +"""Completed first-lane scalar wrapper policy records.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from x2py.types.numpy import SEMANTIC_SCALAR_TYPE_NAMES +from x2py.semantics import models +from x2py.semantics.metadata import BIND_TARGET_METADATA +from x2py.semantics.ownership import ( + CodegenAction, + NativeBarrierAction, + ObjectKind, + OwnershipDecision, + PythonBarrierAction, + StorageMode, +) + + +@dataclass(frozen=True) +class ScalarWrapperArgumentPolicy: + """Completed scalar wrapper policy for one Python-visible argument.""" + + owner_path: str + name: str + python_name: str + native_name: str + python_position: int + native_position: int + semantic_type_name: str + rank: int + optional: bool + ownership: OwnershipDecision + codegen_action: CodegenAction + python_barrier_action: PythonBarrierAction + native_barrier_action: NativeBarrierAction + storage_mode: StorageMode + boundary_storage_mode: StorageMode + projects_result: bool + python_visible: bool + + +@dataclass(frozen=True) +class ScalarWrapperResultPolicy: + """Completed scalar wrapper policy for one native scalar result.""" + + owner_path: str + semantic_type_name: str + rank: int + ownership: OwnershipDecision + codegen_action: CodegenAction + python_barrier_action: PythonBarrierAction + native_barrier_action: NativeBarrierAction + storage_mode: StorageMode + boundary_storage_mode: StorageMode + + +@dataclass(frozen=True) +class ScalarWrapperNativeCallSlotPolicy: + """Completed native-call slot for first-lane scalar wrapper planning.""" + + owner_path: str + native_position: int + source_kind: str + python_position: int | None + python_name: str | None + native_name: str + value_kind: str + native_barrier_action: NativeBarrierAction + codegen_action: CodegenAction + + +@dataclass(frozen=True) +class ScalarWrapperFunctionPolicy: + """Completed first-lane scalar wrapper policy for one semantic function.""" + + owner_path: str + python_name: str + native_name: str + external: bool + bind_target: str | None + supported: bool + arguments: tuple[ScalarWrapperArgumentPolicy, ...] = () + result: ScalarWrapperResultPolicy | None = None + native_call_slots: tuple[ScalarWrapperNativeCallSlotPolicy, ...] = () + blockers: tuple[str, ...] = () + writeback_actions: tuple[str, ...] = () + cleanup_actions: tuple[str, ...] = () + release_actions: tuple[str, ...] = () + + +def completed_scalar_wrapper_policy(function: models.SemanticFunction) -> ScalarWrapperFunctionPolicy: + """Return a completed scalar wrapper policy or fail before planning.""" + + policy = function.metadata.get(models.RESOLVED_SCALAR_WRAPPER_POLICY_METADATA) + if not isinstance(policy, ScalarWrapperFunctionPolicy): + raise ValueError( + f"Semantic function {function.name!r} is missing completed scalar wrapper policy; " + "run complete_semantic_policies before wrapper planning" + ) + if not policy.supported: + details = "; ".join(policy.blockers) or "unsupported first-lane scalar wrapper policy" + raise ValueError(f"Semantic function {policy.owner_path!r} has blocked scalar wrapper policy: {details}") + return policy + + +def build_scalar_wrapper_function_policy( + function: models.SemanticFunction, + *, + owner_path: str, +) -> ScalarWrapperFunctionPolicy: + """Build a typed scalar-wrapper policy from completed post-IR decisions.""" + + argument_native_positions, native_call_slots, slot_blockers = _native_call_slot_policies(function, owner_path) + arguments, argument_blockers = _argument_policies(function, owner_path, argument_native_positions) + result, result_blockers = _result_policy(function, owner_path) + blockers = ( + _function_shape_blockers(function) + + argument_blockers + + result_blockers + + slot_blockers + + _lifecycle_blockers(arguments) + ) + return ScalarWrapperFunctionPolicy( + owner_path=owner_path, + python_name=function.name, + native_name=_native_name(function), + external=_is_external(function), + bind_target=_bind_target(function), + supported=not blockers, + arguments=tuple(arguments), + result=result, + native_call_slots=tuple(native_call_slots), + blockers=tuple(blockers), + ) + + +def _argument_policies( + function: models.SemanticFunction, + owner_path: str, + argument_native_positions: dict[int, int], +) -> tuple[list[ScalarWrapperArgumentPolicy], tuple[str, ...]]: + policies: list[ScalarWrapperArgumentPolicy] = [] + blockers: list[str] = [] + for python_position, argument in enumerate(function.arguments): + decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) + if decision is None: + blockers.append(f"argument {argument.name!r} is missing completed ownership policy") + continue + blockers.extend(_argument_blockers(argument, decision)) + native_position = argument_native_positions.get(python_position) + if native_position is None: + blockers.append(f"argument {argument.name!r} has no completed native-call slot") + native_position = -1 + policies.append( + ScalarWrapperArgumentPolicy( + owner_path=f"{owner_path}.{argument.name}", + name=argument.name, + python_name=argument.name, + native_name=_argument_native_name(function, python_position, argument), + python_position=python_position, + native_position=native_position, + semantic_type_name=argument.semantic_type.name, + rank=int(argument.semantic_type.rank or 0), + optional=argument.optional, + ownership=decision, + codegen_action=decision.codegen_action, + python_barrier_action=decision.python_barrier_action, + native_barrier_action=decision.native_barrier_action, + storage_mode=decision.storage_mode, + boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, + projects_result=decision.projects_result, + python_visible=decision.python_visible, + ) + ) + return policies, tuple(blockers) + + +def _result_policy( + function: models.SemanticFunction, + owner_path: str, +) -> tuple[ScalarWrapperResultPolicy | None, tuple[str, ...]]: + if function.return_type is None: + return None, ("first scalar lane requires one scalar result",) + decision = function.metadata.get(models.RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA) + if not isinstance(decision, OwnershipDecision): + return None, ("function result is missing completed ownership policy",) + blockers = _result_blockers(function.return_type, decision) + return ( + ScalarWrapperResultPolicy( + owner_path=f"{owner_path}.return", + semantic_type_name=function.return_type.name, + rank=int(function.return_type.rank or 0), + ownership=decision, + codegen_action=decision.codegen_action, + python_barrier_action=decision.python_barrier_action, + native_barrier_action=decision.native_barrier_action, + storage_mode=decision.storage_mode, + boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, + ), + tuple(blockers), + ) + + +def _native_call_slot_policies( + function: models.SemanticFunction, + owner_path: str, +) -> tuple[dict[int, int], tuple[ScalarWrapperNativeCallSlotPolicy, ...], tuple[str, ...]]: + if function.projection: + return _projected_native_call_slot_policies(function, owner_path) + return _implicit_native_call_slot_policies(function, owner_path) + + +def _projected_native_call_slot_policies( + function: models.SemanticFunction, + owner_path: str, +) -> tuple[dict[int, int], tuple[ScalarWrapperNativeCallSlotPolicy, ...], tuple[str, ...]]: + slots: list[ScalarWrapperNativeCallSlotPolicy] = [] + blockers: list[str] = [] + positions: dict[int, int] = {} + for mapping in sorted( + function.projection, key=lambda item: item.native_position if item.native_position is not None else -1 + ): + native_position = mapping.native_position + python_position = mapping.python_position + if not isinstance(native_position, int): + blockers.append("native-call projection is missing a native position") + continue + if mapping.result_position is not None or python_position is None: + blockers.append(f"native-call slot {native_position} is not a first-lane Python argument projection") + continue + if not 0 <= python_position < len(function.arguments): + blockers.append(f"native-call slot {native_position} references argument position {python_position}") + continue + argument = function.arguments[python_position] + decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) + if decision is None: + blockers.append(f"native-call slot {native_position} references argument without completed policy") + continue + value_kind = mapping.value_kind or "arg" + if value_kind not in {"addr", "arg"}: + blockers.append(f"native-call slot {native_position} uses unsupported scalar value kind {value_kind!r}") + if python_position in positions: + blockers.append(f"argument {argument.name!r} appears in more than one native-call slot") + positions[python_position] = native_position + slots.append( + ScalarWrapperNativeCallSlotPolicy( + owner_path=f"{owner_path}.{argument.name}", + native_position=native_position, + source_kind="projection", + python_position=python_position, + python_name=mapping.python_name or argument.name, + native_name=mapping.native_name or argument.name, + value_kind=value_kind, + native_barrier_action=decision.native_barrier_action, + codegen_action=decision.codegen_action, + ) + ) + blockers.extend(_native_position_blockers(slot.native_position for slot in slots)) + return positions, tuple(slots), tuple(blockers) + + +def _implicit_native_call_slot_policies( + function: models.SemanticFunction, + owner_path: str, +) -> tuple[dict[int, int], tuple[ScalarWrapperNativeCallSlotPolicy, ...], tuple[str, ...]]: + slots: list[ScalarWrapperNativeCallSlotPolicy] = [] + positions: dict[int, int] = {} + blockers: list[str] = [] + for position, argument in enumerate(function.arguments): + decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) + if decision is None: + blockers.append(f"implicit native-call slot {position} references argument without completed policy") + continue + positions[position] = position + slots.append( + ScalarWrapperNativeCallSlotPolicy( + owner_path=f"{owner_path}.{argument.name}", + native_position=position, + source_kind="implicit", + python_position=position, + python_name=argument.name, + native_name=argument.name, + value_kind="arg", + native_barrier_action=decision.native_barrier_action, + codegen_action=decision.codegen_action, + ) + ) + return positions, tuple(slots), tuple(blockers) + + +def _argument_blockers(argument: models.SemanticArgument, decision: OwnershipDecision) -> tuple[str, ...]: + blockers: list[str] = [] + if decision.is_blocked: + blockers.append( + f"argument {argument.name!r} has blocked ownership policy: {decision.blocker or decision.reason}" + ) + if not _is_first_lane_scalar_type(argument.semantic_type): + blockers.append(f"argument {argument.name!r} is not a first-lane primitive scalar") + if argument.optional: + blockers.append(f"argument {argument.name!r} is optional") + if not decision.python_visible: + blockers.append(f"argument {argument.name!r} is not Python-visible") + if decision.kind is not ObjectKind.SCALAR: + blockers.append(f"argument {argument.name!r} policy kind is {decision.kind.value}, not scalar") + if decision.python_barrier_action is not PythonBarrierAction.SCALAR_VALUE: + blockers.append( + f"argument {argument.name!r} Python action is {decision.python_barrier_action.value}, not scalar_value" + ) + if decision.native_barrier_action not in { + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + NativeBarrierAction.PASS_VALUE, + }: + blockers.append( + f"argument {argument.name!r} native action is {decision.native_barrier_action.value}, " + "not pass_call_local_address or pass_value" + ) + if decision.projects_result: + blockers.append(f"argument {argument.name!r} projects a result") + return tuple(blockers) + + +def _result_blockers(semantic_type: models.SemanticType, decision: OwnershipDecision) -> tuple[str, ...]: + blockers: list[str] = [] + if decision.is_blocked: + blockers.append(f"result has blocked ownership policy: {decision.blocker or decision.reason}") + if not _is_first_lane_scalar_type(semantic_type): + blockers.append("result is not a first-lane primitive scalar") + if decision.kind is not ObjectKind.SCALAR: + blockers.append(f"result policy kind is {decision.kind.value}, not scalar") + if decision.codegen_action is not CodegenAction.DIRECT_VALUE: + blockers.append(f"result codegen action is {decision.codegen_action.value}, not direct_value") + if decision.python_barrier_action is not PythonBarrierAction.NONE: + blockers.append(f"result Python action is {decision.python_barrier_action.value}, not none") + if decision.native_barrier_action is not NativeBarrierAction.NONE: + blockers.append(f"result native action is {decision.native_barrier_action.value}, not none") + return tuple(blockers) + + +def _function_shape_blockers(function: models.SemanticFunction) -> tuple[str, ...]: + blockers: list[str] = [] + if function.visibility != "public": + blockers.append("function is not public") + if isinstance(function, models.SemanticMethod): + blockers.append("methods are outside the first scalar lane") + if function.locals: + blockers.append("function locals are outside the first scalar lane") + if function.contracts: + blockers.append("function contracts are outside the first scalar lane") + return tuple(blockers) + + +def _lifecycle_blockers(arguments: list[ScalarWrapperArgumentPolicy]) -> tuple[str, ...]: + blockers: list[str] = [] + for argument in arguments: + if argument.projects_result: + blockers.append(f"argument {argument.name!r} requires writeback/result lifecycle policy") + return tuple(blockers) + + +def _native_position_blockers(native_positions: object) -> tuple[str, ...]: + positions = tuple(native_positions) + if sorted(positions) != list(range(len(positions))): + return ("native-call slots must cover each native position exactly once in order",) + return () + + +def _ownership_decision(owner: object, metadata_key: str) -> OwnershipDecision | None: + decision = getattr(owner, "metadata", {}).get(metadata_key) + return decision if isinstance(decision, OwnershipDecision) else None + + +def _is_first_lane_scalar_type(semantic_type: models.SemanticType) -> bool: + scalar_name = semantic_type.dtype or semantic_type.name + return bool( + int(semantic_type.rank or 0) == 0 + and semantic_type.name != "String" + and scalar_name in SEMANTIC_SCALAR_TYPE_NAMES + ) + + +def _native_name(function: models.SemanticFunction) -> str: + return str(function.native_name or function.origin.native_name or function.name) + + +def _bind_target(function: models.SemanticFunction) -> str | None: + target = function.metadata.get(BIND_TARGET_METADATA) + return str(target) if target is not None else None + + +def _is_external(function: models.SemanticFunction) -> bool: + return bool(function.origin.source_language == "fortran" and function.origin.native_scope is None) + + +def _argument_native_name( + function: models.SemanticFunction, + python_position: int, + argument: models.SemanticArgument, +) -> str: + for mapping in function.projection: + if mapping.python_position == python_position: + return mapping.native_name or argument.name + return argument.name diff --git a/x2py/wrapper_codegen/__init__.py b/x2py/wrapper_codegen/__init__.py new file mode 100644 index 000000000..fcaee0903 --- /dev/null +++ b/x2py/wrapper_codegen/__init__.py @@ -0,0 +1,52 @@ +"""Isolated wrapper-plan generator infrastructure. + +This package is intentionally disconnected from production wrapper selection +until the migration checklist reaches the route-integration phase. +""" + +from __future__ import annotations + +from .plan import ( + ActionHandlerPlan, + ArgumentTransferPlan, + BindingHandoffPlan, + BridgeAbiPlan, + BridgeAbiSlotPlan, + FunctionPlan, + HandlerRegistryPlan, + LifecycleActionPlan, + ModulePlan, + NativeCallSlotPlan, + ResultPlan, + WrapperPlanDiagnostic, + WrapperPlanSupportBlocker, + WrapperPlanSupportReport, +) +from .planner import WrapperPlanner +from .renderer import WrapperPlanRenderer +from .support import WrapperPlanSupportAnalyzer +from .validator import WrapperPlanValidator +from .visitor import ClassVisitor, UnsupportedWrapperCodegenNodeError + +__all__ = ( + "ActionHandlerPlan", + "ArgumentTransferPlan", + "BindingHandoffPlan", + "BridgeAbiPlan", + "BridgeAbiSlotPlan", + "ClassVisitor", + "FunctionPlan", + "HandlerRegistryPlan", + "LifecycleActionPlan", + "ModulePlan", + "NativeCallSlotPlan", + "ResultPlan", + "UnsupportedWrapperCodegenNodeError", + "WrapperPlanDiagnostic", + "WrapperPlanRenderer", + "WrapperPlanSupportAnalyzer", + "WrapperPlanSupportBlocker", + "WrapperPlanSupportReport", + "WrapperPlanValidator", + "WrapperPlanner", +) diff --git a/x2py/wrapper_codegen/checks.py b/x2py/wrapper_codegen/checks.py new file mode 100644 index 000000000..178eda9df --- /dev/null +++ b/x2py/wrapper_codegen/checks.py @@ -0,0 +1,272 @@ +"""Static contracts for the isolated wrapper-plan generator package.""" + +from __future__ import annotations + +import ast +from dataclasses import dataclass +from pathlib import Path + +from radon.complexity import cc_visit + +__all__ = ( + "WrapperCodegenCheckConfig", + "WrapperCodegenViolation", + "check_wrapper_codegen_package", + "check_wrapper_codegen_paths", +) + + +INFRASTRUCTURE_MODULES = frozenset({"__init__.py", "checks.py", "visitor.py"}) +VISITOR_CLASS_SUFFIXES = ("Analyzer", "Emitter", "Planner", "Renderer", "Validator") +REGISTRY_SUFFIXES = ("_DISPATCHER", "_HANDLERS", "_REGISTRY") +HANDLER_PREFIXES = ("_convert_", "_emit_", "_handle_", "_visit_") +CONTROL_NODES = (ast.For, ast.AsyncFor, ast.If, ast.Match, ast.Try, ast.While, ast.With, ast.AsyncWith) + + +@dataclass(frozen=True) +class WrapperCodegenCheckConfig: + """Limits enforced for new wrapper-codegen implementation code.""" + + max_complexity: int = 10 + max_statements: int = 30 + max_nesting: int = 4 + + +@dataclass(frozen=True) +class WrapperCodegenViolation: + """One static-contract violation in ``x2py.wrapper_codegen``.""" + + path: Path + lineno: int + code: str + message: str + + @property + def label(self) -> str: + return f"{self.path}:{self.lineno}: {self.code}: {self.message}" + + +DEFAULT_CHECK_CONFIG = WrapperCodegenCheckConfig() + + +def check_wrapper_codegen_package( + package_root: Path | None = None, + *, + config: WrapperCodegenCheckConfig | None = None, +) -> tuple[WrapperCodegenViolation, ...]: + """Check every Python module in the isolated wrapper-codegen package.""" + root = package_root or Path(__file__).resolve().parent + return check_wrapper_codegen_paths( + sorted(root.rglob("*.py")), package_root=root, config=config or DEFAULT_CHECK_CONFIG + ) + + +def check_wrapper_codegen_paths( + paths: list[Path], + *, + package_root: Path, + config: WrapperCodegenCheckConfig | None = None, +) -> tuple[WrapperCodegenViolation, ...]: + """Check selected wrapper-codegen modules.""" + violations = [] + resolved_config = config or DEFAULT_CHECK_CONFIG + for path in paths: + source = path.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(path)) + violations.extend(_module_violations(path, tree, package_root)) + violations.extend(_function_size_violations(path, tree, source, resolved_config)) + violations.extend(_registry_violations(path, tree)) + return tuple(violations) + + +def _module_violations(path: Path, tree: ast.Module, package_root: Path) -> list[WrapperCodegenViolation]: + if path.name in INFRASTRUCTURE_MODULES: + return [] + return [ + *_module_function_violations(path, tree), + *_visitor_class_violations(path, tree), + *_dependency_violations(path, tree, package_root), + ] + + +def _module_function_violations(path: Path, tree: ast.Module) -> list[WrapperCodegenViolation]: + return [ + _violation(path, node, "module-function", f"move production function {node.name!r} onto a class") + for node in tree.body + if isinstance(node, ast.FunctionDef) + ] + + +def _visitor_class_violations(path: Path, tree: ast.Module) -> list[WrapperCodegenViolation]: + return [ + _violation(path, node, "visitor-class", f"{node.name} must inherit ClassVisitor") + for node in tree.body + if isinstance(node, ast.ClassDef) + and node.name.endswith(VISITOR_CLASS_SUFFIXES) + and not _inherits_class_visitor(node) + ] + + +def _dependency_violations(path: Path, tree: ast.Module, package_root: Path) -> list[WrapperCodegenViolation]: + if not _is_under(path, package_root): + return [] + return [ + _violation(path, node, "legacy-codegen-import", "wrapper_codegen must not import x2py.codegen") + for node in ast.walk(tree) + if _imports_legacy_codegen(node) + ] + + +def _function_size_violations( + path: Path, + tree: ast.Module, + source: str, + config: WrapperCodegenCheckConfig, +) -> list[WrapperCodegenViolation]: + return [ + *_complexity_violations(path, source, config.max_complexity), + *_statement_count_violations(path, tree, config.max_statements), + *_nesting_violations(path, tree, config.max_nesting), + ] + + +def _complexity_violations(path: Path, source: str, max_complexity: int) -> list[WrapperCodegenViolation]: + return [ + WrapperCodegenViolation(path, block.lineno, "complexity", f"{block.name} has complexity {block.complexity}") + for block in cc_visit(source) + if block.complexity > max_complexity + ] + + +def _statement_count_violations(path: Path, tree: ast.Module, max_statements: int) -> list[WrapperCodegenViolation]: + return [ + _violation(path, node, "statement-count", f"{node.name} has {statement_count} statements") + for node in _functions(tree) + for statement_count in (_statement_count(node),) + if statement_count > max_statements + ] + + +def _nesting_violations(path: Path, tree: ast.Module, max_nesting: int) -> list[WrapperCodegenViolation]: + return [ + _violation(path, node, "nesting-depth", f"{node.name} has nesting depth {nesting_depth}") + for node in _functions(tree) + for nesting_depth in (_nesting_depth(node),) + if nesting_depth > max_nesting + ] + + +def _registry_violations(path: Path, tree: ast.Module) -> list[WrapperCodegenViolation]: + violations = [] + for class_node in (node for node in tree.body if isinstance(node, ast.ClassDef)): + method_names = {node.name for node in class_node.body if isinstance(node, ast.FunctionDef)} + registry_methods = _registry_methods(class_node) + violations.extend(_missing_registry_methods(path, class_node, method_names, registry_methods)) + violations.extend(_forbidden_printer_calls(path, class_node, registry_methods)) + return violations + + +def _missing_registry_methods( + path: Path, + class_node: ast.ClassDef, + method_names: set[str], + registry_methods: dict[str, ast.AST], +) -> list[WrapperCodegenViolation]: + return [ + _violation(path, owner, "registry-missing-handler", f"{class_node.name}.{method_name} is not defined") + for method_name, owner in registry_methods.items() + if method_name not in method_names + ] + + +def _forbidden_printer_calls( + path: Path, + class_node: ast.ClassDef, + registry_methods: dict[str, ast.AST], +) -> list[WrapperCodegenViolation]: + handler_names = set(registry_methods) | { + node.name + for node in class_node.body + if isinstance(node, ast.FunctionDef) and node.name.startswith(HANDLER_PREFIXES) + } + return [ + _violation(path, call, "handler-printer-call", f"{class_node.name}.{method.name} calls a printer directly") + for method in class_node.body + if isinstance(method, ast.FunctionDef) and method.name in handler_names + for call in ast.walk(method) + if _is_forbidden_printer_call(call) + ] + + +def _functions(tree: ast.Module) -> list[ast.FunctionDef]: + return [node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)] + + +def _statement_count(node: ast.FunctionDef) -> int: + return sum(isinstance(child, ast.stmt) for child in ast.walk(node)) - 1 + + +def _nesting_depth(node: ast.FunctionDef) -> int: + return max((_node_nesting_depth(statement, 0) for statement in node.body), default=0) + + +def _node_nesting_depth(node: ast.AST, depth: int) -> int: + next_depth = depth + 1 if isinstance(node, CONTROL_NODES) else depth + child_depths = [_node_nesting_depth(child, next_depth) for child in ast.iter_child_nodes(node)] + return max([next_depth, *child_depths]) + + +def _registry_methods(class_node: ast.ClassDef) -> dict[str, ast.AST]: + methods = {} + for assignment in (node for node in class_node.body if isinstance(node, ast.Assign)): + if _is_registry_assignment(assignment): + _collect_registry_methods(assignment.value, assignment, methods) + return methods + + +def _is_registry_assignment(node: ast.Assign) -> bool: + return any(isinstance(target, ast.Name) and target.id.endswith(REGISTRY_SUFFIXES) for target in node.targets) + + +def _collect_registry_methods(node: ast.AST, owner: ast.AST, methods: dict[str, ast.AST]) -> None: + if isinstance(node, ast.Constant) and isinstance(node.value, str): + methods.setdefault(node.value, owner) + if isinstance(node, ast.Dict): + for value in node.values: + _collect_registry_methods(value, owner, methods) + + +def _is_forbidden_printer_call(node: ast.AST) -> bool: + return ( + isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) and node.func.attr in {"doprint", "write"} + ) + + +def _inherits_class_visitor(node: ast.ClassDef) -> bool: + return any(_base_name(base) == "ClassVisitor" for base in node.bases) + + +def _base_name(node: ast.AST) -> str | None: + if isinstance(node, ast.Name): + return node.id + if isinstance(node, ast.Attribute): + return node.attr + return None + + +def _imports_legacy_codegen(node: ast.AST) -> bool: + if isinstance(node, ast.Import): + return any(alias.name == "x2py.codegen" or alias.name.startswith("x2py.codegen.") for alias in node.names) + return isinstance(node, ast.ImportFrom) and bool(node.module) and _is_codegen_module(node.module) + + +def _is_codegen_module(module_name: str) -> bool: + return module_name == "x2py.codegen" or module_name.startswith("x2py.codegen.") + + +def _is_under(path: Path, directory: Path) -> bool: + return path.resolve().is_relative_to(directory.resolve()) + + +def _violation(path: Path, node: ast.AST, code: str, message: str) -> WrapperCodegenViolation: + return WrapperCodegenViolation(path, getattr(node, "lineno", 1), code, message) diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py new file mode 100644 index 000000000..05c805e09 --- /dev/null +++ b/x2py/wrapper_codegen/plan.py @@ -0,0 +1,169 @@ +"""Frozen wrapper-plan records for the isolated wrapper-plan route.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, PythonBarrierAction + + +@dataclass(frozen=True) +class ActionHandlerPlan: + """One completed semantic action mapped to one implementation handler.""" + + action: CodegenAction | NativeBarrierAction | PythonBarrierAction + handler_name: str + + +@dataclass(frozen=True) +class HandlerRegistryPlan: + """Handler names keyed by existing completed semantic action values.""" + + python_action_handlers: tuple[ActionHandlerPlan, ...] + native_action_handlers: tuple[ActionHandlerPlan, ...] + result_action_handlers: tuple[ActionHandlerPlan, ...] + + +@dataclass(frozen=True) +class BindingHandoffPlan: + """Symbolic handoff produced by the binding layer for bridge consumption.""" + + owner_path: str + produced_role: str + consumed_role: str + python_action: PythonBarrierAction + handler_name: str + + +@dataclass(frozen=True) +class BridgeAbiSlotPlan: + """One deterministic bridge ABI slot for an argument transfer.""" + + owner_path: str + index: int + symbolic_role: str + native_action: NativeBarrierAction + handler_name: str + + +@dataclass(frozen=True) +class BridgeAbiPlan: + """Bridge ABI slot collection for one function.""" + + owner_path: str + slots: tuple[BridgeAbiSlotPlan, ...] + + +@dataclass(frozen=True) +class NativeCallSlotPlan: + """Native-call slot copied from completed scalar policy.""" + + owner_path: str + native_position: int + source_kind: str + python_position: int | None + python_name: str | None + native_name: str + value_kind: str + symbolic_role: str + native_action: NativeBarrierAction + codegen_action: CodegenAction + + +@dataclass(frozen=True) +class ArgumentTransferPlan: + """Single Python-to-native transfer plan for one argument.""" + + owner_path: str + python_name: str + native_name: str + python_position: int + native_position: int + semantic_type_name: str + python_action: PythonBarrierAction + native_action: NativeBarrierAction + codegen_action: CodegenAction + binding_handoff: BindingHandoffPlan + bridge_abi_slot: BridgeAbiSlotPlan | None + native_call_slot: NativeCallSlotPlan | None + + +@dataclass(frozen=True) +class ResultPlan: + """Symbolic result conversion plan for one function result.""" + + owner_path: str + semantic_type_name: str + codegen_action: CodegenAction + python_action: PythonBarrierAction + native_action: NativeBarrierAction + native_result_role: str + python_result_role: str + handler_name: str + + +@dataclass(frozen=True) +class LifecycleActionPlan: + """Symbolic lifecycle action that consumes a previously available role.""" + + owner_path: str + phase: str + source_role: str + handler_name: str + + +@dataclass(frozen=True) +class FunctionPlan: + """Wrapper plan for one semantic function owner.""" + + owner_path: str + python_name: str + native_name: str + external: bool + bind_target: str | None + arguments: tuple[ArgumentTransferPlan, ...] + result: ResultPlan | None + bridge_abi: BridgeAbiPlan + available_roles: tuple[str, ...] + writeback_actions: tuple[LifecycleActionPlan, ...] = () + cleanup_actions: tuple[LifecycleActionPlan, ...] = () + release_actions: tuple[LifecycleActionPlan, ...] = () + + +@dataclass(frozen=True) +class ModulePlan: + """Wrapper plan for one generation unit.""" + + owner_path: str + functions: tuple[FunctionPlan, ...] + handler_registry: HandlerRegistryPlan + + +@dataclass(frozen=True) +class WrapperPlanDiagnostic: + """One owner-path diagnostic produced before backend emission.""" + + owner_path: str + code: str + message: str + + +@dataclass(frozen=True) +class WrapperPlanSupportBlocker: + """One stable unsupported-owner reason for the wrapper-plan route.""" + + owner_path: str + reason: str + + +@dataclass(frozen=True) +class WrapperPlanSupportReport: + """Whole-generation-unit support report for route selection callers.""" + + owner_path: str + blockers: tuple[WrapperPlanSupportBlocker, ...] = () + + @property + def supported(self) -> bool: + """Return whether the whole generation unit is supported.""" + return not self.blockers diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py new file mode 100644 index 000000000..c81a76f78 --- /dev/null +++ b/x2py/wrapper_codegen/planner.py @@ -0,0 +1,216 @@ +"""Hierarchical wrapper-plan construction from completed semantic policy.""" + +from __future__ import annotations + +from typing import ClassVar + +from x2py.semantics import models +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, PythonBarrierAction +from x2py.semantics.scalar_wrapper_policy import ( + ScalarWrapperArgumentPolicy, + ScalarWrapperFunctionPolicy, + ScalarWrapperNativeCallSlotPolicy, + ScalarWrapperResultPolicy, + completed_scalar_wrapper_policy, +) +from x2py.wrapper_codegen.plan import ( + ActionHandlerPlan, + ArgumentTransferPlan, + BindingHandoffPlan, + BridgeAbiPlan, + BridgeAbiSlotPlan, + FunctionPlan, + HandlerRegistryPlan, + ModulePlan, + NativeCallSlotPlan, + ResultPlan, +) +from x2py.wrapper_codegen.support import WrapperPlanSupportAnalyzer +from x2py.wrapper_codegen.visitor import ClassVisitor + + +class WrapperPlanner(ClassVisitor): + """Build route-neutral wrapper plans from completed semantic policies.""" + + PYTHON_ACTION_REGISTRY: ClassVar[dict[PythonBarrierAction, str]] = { + PythonBarrierAction.SCALAR_VALUE: "_handle_python_scalar_value", + } + NATIVE_ACTION_REGISTRY: ClassVar[dict[NativeBarrierAction, str]] = { + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: "_handle_native_call_local_address", + NativeBarrierAction.PASS_VALUE: "_handle_native_value", + } + RESULT_ACTION_REGISTRY: ClassVar[dict[CodegenAction, str]] = { + CodegenAction.DIRECT_VALUE: "_handle_direct_scalar_result", + } + + def __init__(self, *, support_analyzer: WrapperPlanSupportAnalyzer | None = None): + """Create a planner with an optional support analyzer.""" + super().__init__() + self.support_analyzer = support_analyzer or WrapperPlanSupportAnalyzer() + + def _visit_SemanticModule(self, module: models.SemanticModule) -> ModulePlan: + """Return one module plan after whole-unit support analysis.""" + report = self.support_analyzer.visit(module) + if not report.supported: + raise ValueError(self._support_error(module.name, report.blockers)) + return ModulePlan( + owner_path=module.name, + functions=tuple(self.visit(function) for function in module.functions), + handler_registry=self._handler_registry(), + ) + + def _visit_SemanticFunction(self, function: models.SemanticFunction) -> FunctionPlan: + """Return one function plan from completed scalar wrapper policy.""" + policy = completed_scalar_wrapper_policy(function) + arguments = tuple( + self.visit(argument, native_slot=self._native_slot(policy, argument)) for argument in policy.arguments + ) + result = self.visit(policy.result) if policy.result is not None else None + return FunctionPlan( + owner_path=policy.owner_path, + python_name=policy.python_name, + native_name=policy.native_name, + external=policy.external, + bind_target=policy.bind_target, + arguments=arguments, + result=result, + bridge_abi=BridgeAbiPlan(policy.owner_path, tuple(argument.bridge_abi_slot for argument in arguments)), + available_roles=self._available_roles(arguments, result), + ) + + def _visit_ScalarWrapperArgumentPolicy( + self, + policy: ScalarWrapperArgumentPolicy, + *, + native_slot: ScalarWrapperNativeCallSlotPolicy, + ) -> ArgumentTransferPlan: + """Return one argument transfer plan from completed scalar policy.""" + role = self._value_role(policy.owner_path) + binding_handoff = self._binding_handoff(policy, role) + bridge_slot = self._bridge_slot(policy, native_slot, role) + return ArgumentTransferPlan( + owner_path=policy.owner_path, + python_name=policy.python_name, + native_name=policy.native_name, + python_position=policy.python_position, + native_position=policy.native_position, + semantic_type_name=policy.semantic_type_name, + python_action=policy.python_barrier_action, + native_action=policy.native_barrier_action, + codegen_action=policy.codegen_action, + binding_handoff=binding_handoff, + bridge_abi_slot=bridge_slot, + native_call_slot=self._native_slot_plan(native_slot, role), + ) + + def _visit_ScalarWrapperResultPolicy(self, policy: ScalarWrapperResultPolicy) -> ResultPlan: + """Return one result plan from completed scalar result policy.""" + return ResultPlan( + owner_path=policy.owner_path, + semantic_type_name=policy.semantic_type_name, + codegen_action=policy.codegen_action, + python_action=policy.python_barrier_action, + native_action=policy.native_barrier_action, + native_result_role=f"{policy.owner_path}:native-result", + python_result_role=f"{policy.owner_path}:python-result", + handler_name=self.RESULT_ACTION_REGISTRY[policy.codegen_action], + ) + + def _binding_handoff(self, policy: ScalarWrapperArgumentPolicy, role: str) -> BindingHandoffPlan: + """Return the binding-to-bridge handoff for one transfer.""" + return BindingHandoffPlan( + owner_path=policy.owner_path, + produced_role=role, + consumed_role=role, + python_action=policy.python_barrier_action, + handler_name=self.PYTHON_ACTION_REGISTRY[policy.python_barrier_action], + ) + + def _bridge_slot( + self, + policy: ScalarWrapperArgumentPolicy, + native_slot: ScalarWrapperNativeCallSlotPolicy, + role: str, + ) -> BridgeAbiSlotPlan: + """Return the bridge ABI slot for one transfer.""" + return BridgeAbiSlotPlan( + owner_path=policy.owner_path, + index=native_slot.native_position, + symbolic_role=role, + native_action=policy.native_barrier_action, + handler_name=self.NATIVE_ACTION_REGISTRY[policy.native_barrier_action], + ) + + def _native_slot_plan(self, native_slot: ScalarWrapperNativeCallSlotPolicy, role: str) -> NativeCallSlotPlan: + """Return the native-call slot for one transfer.""" + return NativeCallSlotPlan( + owner_path=native_slot.owner_path, + native_position=native_slot.native_position, + source_kind=native_slot.source_kind, + python_position=native_slot.python_position, + python_name=native_slot.python_name, + native_name=native_slot.native_name, + value_kind=native_slot.value_kind, + symbolic_role=role, + native_action=native_slot.native_barrier_action, + codegen_action=native_slot.codegen_action, + ) + + def _handler_registry(self) -> HandlerRegistryPlan: + """Return the class-owned completed action registry as plan data.""" + return HandlerRegistryPlan( + python_action_handlers=self._handler_refs(self.PYTHON_ACTION_REGISTRY), + native_action_handlers=self._handler_refs(self.NATIVE_ACTION_REGISTRY), + result_action_handlers=self._handler_refs(self.RESULT_ACTION_REGISTRY), + ) + + def _native_slot( + self, + function_policy: ScalarWrapperFunctionPolicy, + argument_policy: ScalarWrapperArgumentPolicy, + ) -> ScalarWrapperNativeCallSlotPolicy: + """Return the completed native-call slot for one argument policy.""" + for slot in function_policy.native_call_slots: + if slot.owner_path == argument_policy.owner_path: + return slot + raise ValueError(f"{argument_policy.owner_path!r} is missing a completed native-call slot") + + def _available_roles( + self, + arguments: tuple[ArgumentTransferPlan, ...], + result: ResultPlan | None, + ) -> tuple[str, ...]: + """Return symbolic values available after the native call.""" + roles = [argument.binding_handoff.produced_role for argument in arguments] + if result is not None: + roles.append(result.native_result_role) + return tuple(roles) + + def _handler_refs(self, registry: dict[object, str]) -> tuple[ActionHandlerPlan, ...]: + """Return handler registry entries as deterministic plan records.""" + return tuple(ActionHandlerPlan(action=action, handler_name=handler) for action, handler in registry.items()) + + def _support_error(self, owner_path: str, blockers: object) -> str: + """Return a compact unsupported-generation-unit error.""" + details = "; ".join(f"{item.owner_path}: {item.reason}" for item in blockers) + return f"Unsupported wrapper-plan generation unit {owner_path!r}: {details}" + + def _value_role(self, owner_path: str) -> str: + """Return the symbolic value role for one transfer owner.""" + return f"{owner_path}:value" + + def _handle_python_scalar_value(self, plan: object) -> object: + """Registry target for scalar Python argument values.""" + return plan + + def _handle_native_call_local_address(self, plan: object) -> object: + """Registry target for scalar call-local address native slots.""" + return plan + + def _handle_native_value(self, plan: object) -> object: + """Registry target for scalar by-value native slots.""" + return plan + + def _handle_direct_scalar_result(self, plan: object) -> object: + """Registry target for scalar direct-value results.""" + return plan diff --git a/x2py/wrapper_codegen/renderer.py b/x2py/wrapper_codegen/renderer.py new file mode 100644 index 000000000..2a9ed7992 --- /dev/null +++ b/x2py/wrapper_codegen/renderer.py @@ -0,0 +1,101 @@ +"""Deterministic text rendering for wrapper plans.""" + +from __future__ import annotations + +from x2py.wrapper_codegen.plan import ( + ActionHandlerPlan, + ArgumentTransferPlan, + FunctionPlan, + HandlerRegistryPlan, + LifecycleActionPlan, + ModulePlan, + ResultPlan, +) +from x2py.wrapper_codegen.visitor import ClassVisitor + + +class WrapperPlanRenderer(ClassVisitor): + """Render wrapper plans for maintainer diagnostics.""" + + def _visit_ModulePlan(self, plan: ModulePlan) -> str: + """Render one module plan.""" + lines = [f"module {plan.owner_path}", self.visit(plan.handler_registry)] + lines.extend(self.visit(function) for function in plan.functions) + return "\n".join(lines) + + def _visit_HandlerRegistryPlan(self, plan: HandlerRegistryPlan) -> str: + """Render action handler registries.""" + lines = ["handlers:"] + lines.extend(self._handler_lines("python", plan.python_action_handlers)) + lines.extend(self._handler_lines("native", plan.native_action_handlers)) + lines.extend(self._handler_lines("result", plan.result_action_handlers)) + return "\n".join(lines) + + def _visit_FunctionPlan(self, plan: FunctionPlan) -> str: + """Render one function plan.""" + lines = [ + f"function {plan.owner_path} python={plan.python_name} native={plan.native_name}", + f" external={plan.external} bind={plan.bind_target}", + f" available_roles={','.join(plan.available_roles)}", + ] + lines.extend(f" {self.visit(argument)}" for argument in plan.arguments) + if plan.result is not None: + lines.append(f" {self.visit(plan.result)}") + lines.extend(self._lifecycle_lines(plan)) + return "\n".join(lines) + + def _visit_ArgumentTransferPlan(self, plan: ArgumentTransferPlan) -> str: + """Render one argument transfer.""" + return ( + f"arg {plan.owner_path} py={plan.python_action.value} " + f"py_handler={plan.binding_handoff.handler_name} " + f"handoff={plan.binding_handoff.produced_role}->{plan.binding_handoff.consumed_role} " + f"bridge_slot={self._bridge_slot_label(plan)} native={plan.native_action.value} " + f"native_handler={self._bridge_handler_label(plan)} native_slot={self._native_slot_label(plan)}" + ) + + def _visit_ResultPlan(self, plan: ResultPlan) -> str: + """Render one result plan.""" + return ( + f"result {plan.owner_path} codegen={plan.codegen_action.value} " + f"python={plan.python_action.value} native={plan.native_action.value} " + f"handler={plan.handler_name} role={plan.native_result_role}->{plan.python_result_role}" + ) + + def _visit_LifecycleActionPlan(self, plan: LifecycleActionPlan) -> str: + """Render one lifecycle action.""" + return f"{plan.phase} {plan.owner_path} source={plan.source_role} handler={plan.handler_name}" + + def _handler_lines( + self, + label: str, + handlers: tuple[ActionHandlerPlan, ...], + ) -> tuple[str, ...]: + """Render one handler registry group.""" + return tuple(f" {label}:{handler.action.value}->{handler.handler_name}" for handler in handlers) + + def _lifecycle_lines(self, plan: FunctionPlan) -> tuple[str, ...]: + """Render lifecycle actions in execution order.""" + actions = plan.writeback_actions + plan.cleanup_actions + plan.release_actions + if not actions: + return (" lifecycle=none",) + return tuple(f" {self.visit(action)}" for action in actions) + + def _bridge_slot_label(self, plan: ArgumentTransferPlan) -> str: + """Return a printable bridge ABI slot label.""" + if plan.bridge_abi_slot is None: + return "" + return f"{plan.bridge_abi_slot.index}:{plan.bridge_abi_slot.symbolic_role}" + + def _bridge_handler_label(self, plan: ArgumentTransferPlan) -> str: + """Return a printable native handler label.""" + if plan.bridge_abi_slot is None: + return "" + return plan.bridge_abi_slot.handler_name + + def _native_slot_label(self, plan: ArgumentTransferPlan) -> str: + """Return a printable native-call slot label.""" + if plan.native_call_slot is None: + return "" + slot = plan.native_call_slot + return f"{slot.native_position}:{slot.value_kind}:{slot.symbolic_role}" diff --git a/x2py/wrapper_codegen/support.py b/x2py/wrapper_codegen/support.py new file mode 100644 index 000000000..430a8d81d --- /dev/null +++ b/x2py/wrapper_codegen/support.py @@ -0,0 +1,64 @@ +"""Support analysis for wrapper-plan generation units.""" + +from __future__ import annotations + +from x2py.semantics import models +from x2py.semantics.scalar_wrapper_policy import ScalarWrapperFunctionPolicy +from x2py.wrapper_codegen.plan import WrapperPlanSupportBlocker, WrapperPlanSupportReport +from x2py.wrapper_codegen.visitor import ClassVisitor + + +class WrapperPlanSupportAnalyzer(ClassVisitor): + """Report whole-generation-unit eligibility without selecting a route.""" + + def _visit_SemanticModule(self, module: models.SemanticModule) -> WrapperPlanSupportReport: + """Return a stable support report for a semantic module.""" + blockers = [ + *self._module_blockers(module), + *self._child_blockers(module), + ] + return WrapperPlanSupportReport(owner_path=module.name, blockers=tuple(blockers)) + + def _visit_SemanticFunction(self, function: models.SemanticFunction) -> WrapperPlanSupportReport: + """Return a stable support report for one semantic function.""" + policy = function.metadata.get(models.RESOLVED_SCALAR_WRAPPER_POLICY_METADATA) + if not isinstance(policy, ScalarWrapperFunctionPolicy): + blocker = WrapperPlanSupportBlocker( + owner_path=function.name, + reason="missing completed scalar wrapper policy", + ) + return WrapperPlanSupportReport(owner_path=function.name, blockers=(blocker,)) + return WrapperPlanSupportReport( + owner_path=policy.owner_path, + blockers=tuple(WrapperPlanSupportBlocker(policy.owner_path, reason) for reason in policy.blockers), + ) + + def _module_blockers(self, module: models.SemanticModule) -> tuple[WrapperPlanSupportBlocker, ...]: + """Return blockers for module-level owners outside the first scalar lane.""" + blockers = [] + blockers.extend(self._owner_blockers(module.name, "variables", module.variables)) + blockers.extend(self._owner_blockers(module.name, "classes", module.classes)) + blockers.extend(self._owner_blockers(module.name, "overload sets", module.overload_sets)) + return tuple(blockers) + + def _child_blockers(self, module: models.SemanticModule) -> tuple[WrapperPlanSupportBlocker, ...]: + """Return blockers reported by supported child visitor methods.""" + blockers = [] + for function in module.functions: + blockers.extend(self.visit(function).blockers) + return tuple(blockers) + + def _owner_blockers( + self, + module_name: str, + owner_kind: str, + owners: list[object], + ) -> tuple[WrapperPlanSupportBlocker, ...]: + """Return one blocker for each unsupported module child owner.""" + return tuple( + WrapperPlanSupportBlocker( + owner_path=f"{module_name}.{getattr(owner, 'name', owner_kind)}", + reason=f"{owner_kind} are outside the first scalar wrapper-plan lane", + ) + for owner in owners + ) diff --git a/x2py/wrapper_codegen/validator.py b/x2py/wrapper_codegen/validator.py new file mode 100644 index 000000000..56cfc912f --- /dev/null +++ b/x2py/wrapper_codegen/validator.py @@ -0,0 +1,180 @@ +"""Validation for route-neutral wrapper plans before backend emission.""" + +from __future__ import annotations + +from collections import Counter + +from x2py.wrapper_codegen.plan import ( + ActionHandlerPlan, + ArgumentTransferPlan, + FunctionPlan, + HandlerRegistryPlan, + LifecycleActionPlan, + ModulePlan, + ResultPlan, + WrapperPlanDiagnostic, +) +from x2py.wrapper_codegen.visitor import ClassVisitor + + +class WrapperPlanValidator(ClassVisitor): + """Validate wrapper-plan invariants before node emission.""" + + def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for every function in a module plan.""" + diagnostics = [] + for function in plan.functions: + diagnostics.extend(self.visit(function, handler_registry=plan.handler_registry)) + return tuple(diagnostics) + + def _visit_FunctionPlan( + self, + plan: FunctionPlan, + *, + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one function plan.""" + diagnostics = list(self._duplicate_role_diagnostics(plan)) + for argument in plan.arguments: + diagnostics.extend(self.visit(argument, handler_registry=handler_registry)) + if plan.result is not None: + diagnostics.extend( + self.visit(plan.result, available_roles=plan.available_roles, handler_registry=handler_registry) + ) + for action in self._lifecycle_actions(plan): + diagnostics.extend(self.visit(action, available_roles=plan.available_roles)) + return tuple(diagnostics) + + def _visit_ArgumentTransferPlan( + self, + plan: ArgumentTransferPlan, + *, + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one argument transfer.""" + diagnostics = [] + diagnostics.extend(self._primary_handler_diagnostics(plan, handler_registry)) + diagnostics.extend(self._secondary_handler_diagnostics(plan, handler_registry)) + diagnostics.extend(self._missing_slot_diagnostics(plan)) + diagnostics.extend(self._handoff_diagnostics(plan)) + return tuple(diagnostics) + + def _visit_ResultPlan( + self, + plan: ResultPlan, + *, + available_roles: tuple[str, ...], + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one result plan.""" + diagnostics = list(self._result_handler_diagnostics(plan, handler_registry)) + if plan.native_result_role not in available_roles: + diagnostics.append(self._diagnostic(plan.owner_path, "unavailable-result-role", plan.native_result_role)) + return tuple(diagnostics) + + def _visit_LifecycleActionPlan( + self, + plan: LifecycleActionPlan, + *, + available_roles: tuple[str, ...], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one lifecycle action.""" + if plan.source_role in available_roles: + return () + return (self._diagnostic(plan.owner_path, f"unavailable-{plan.phase}-role", plan.source_role),) + + def _primary_handler_diagnostics( + self, + plan: ArgumentTransferPlan, + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for missing Python action handlers.""" + handlers = self._handler_map(handler_registry.python_action_handlers) + if plan.python_action in handlers: + return () + return (self._diagnostic(plan.owner_path, "unknown-primary-handler", plan.python_action.value),) + + def _secondary_handler_diagnostics( + self, + plan: ArgumentTransferPlan, + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for missing native action handlers.""" + handlers = self._handler_map(handler_registry.native_action_handlers) + if plan.native_action in handlers: + return () + return (self._diagnostic(plan.owner_path, "unknown-secondary-handler", plan.native_action.value),) + + def _result_handler_diagnostics( + self, + plan: ResultPlan, + handler_registry: HandlerRegistryPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for missing result handlers.""" + handlers = self._handler_map(handler_registry.result_action_handlers) + if plan.codegen_action in handlers: + return () + return (self._diagnostic(plan.owner_path, "unknown-result-handler", plan.codegen_action.value),) + + def _missing_slot_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for absent bridge or native slots.""" + diagnostics = [] + if plan.bridge_abi_slot is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-bridge-abi-slot", plan.native_name)) + if plan.native_call_slot is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-native-call-slot", plan.native_name)) + return tuple(diagnostics) + + def _handoff_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for inconsistent transfer roles and actions.""" + if plan.bridge_abi_slot is None or plan.native_call_slot is None: + return () + diagnostics = [] + diagnostics.extend(self._handoff_role_diagnostics(plan)) + diagnostics.extend(self._slot_action_diagnostics(plan)) + return tuple(diagnostics) + + def _handoff_role_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for inconsistent symbolic handoff roles.""" + role = plan.binding_handoff.consumed_role + diagnostics = [] + if plan.binding_handoff.produced_role != role: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-binding-handoff", role)) + if plan.bridge_abi_slot is not None and plan.bridge_abi_slot.symbolic_role != role: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-bridge-handoff", role)) + if plan.native_call_slot is not None and plan.native_call_slot.symbolic_role != role: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-native-handoff", role)) + return tuple(diagnostics) + + def _slot_action_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for inconsistent slot actions or positions.""" + diagnostics = [] + if plan.bridge_abi_slot.native_action is not plan.native_action: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-bridge-action", plan.native_action.value) + ) + if plan.native_call_slot.native_action is not plan.native_action: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-native-action", plan.native_action.value) + ) + if plan.bridge_abi_slot.index != plan.native_call_slot.native_position: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-native-position", plan.native_name)) + return tuple(diagnostics) + + def _duplicate_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for duplicate argument symbolic roles.""" + roles = [argument.binding_handoff.produced_role for argument in plan.arguments] + duplicates = sorted(role for role, count in Counter(roles).items() if count > 1) + return tuple(self._diagnostic(plan.owner_path, "duplicate-symbolic-role", role) for role in duplicates) + + def _lifecycle_actions(self, plan: FunctionPlan) -> tuple[LifecycleActionPlan, ...]: + """Return lifecycle actions in execution order.""" + return plan.writeback_actions + plan.cleanup_actions + plan.release_actions + + def _handler_map(self, handlers: tuple[ActionHandlerPlan, ...]) -> dict[object, str]: + """Return a lookup for configured handler names.""" + return {handler.action: handler.handler_name for handler in handlers if handler.handler_name} + + def _diagnostic(self, owner_path: str, code: str, detail: object) -> WrapperPlanDiagnostic: + """Return one stable owner-path diagnostic.""" + return WrapperPlanDiagnostic(owner_path=owner_path, code=code, message=str(detail)) diff --git a/x2py/wrapper_codegen/visitor.py b/x2py/wrapper_codegen/visitor.py new file mode 100644 index 000000000..e790b1554 --- /dev/null +++ b/x2py/wrapper_codegen/visitor.py @@ -0,0 +1,44 @@ +"""Independent class-based visitor protocol for wrapper-plan generation.""" + +from __future__ import annotations + +from dataclasses import dataclass + +__all__ = ("ClassVisitor", "UnsupportedWrapperCodegenNodeError") + + +@dataclass(frozen=True) +class UnsupportedWrapperCodegenNodeError(TypeError): + """Raised when a wrapper-codegen visitor has no handler for a node.""" + + visitor_type: type + node_type: type + method_prefix: str + + def __str__(self) -> str: + return ( + f"{self.visitor_type.__name__} does not support {self.node_type.__name__} " + f"with prefix {self.method_prefix!r}" + ) + + +class ClassVisitor: + """Dispatch nodes through a deterministic ``_`` protocol.""" + + visitor_method_prefix = "_visit" + + def __init__(self, *, method_prefix: str | None = None): + if method_prefix is not None: + self.visitor_method_prefix = method_prefix + + def visit(self, node, *args, **kwargs): + """Call the most specific handler for ``node``.""" + for node_type in type(node).__mro__: + method = getattr(self, self._visitor_method_name(node_type), None) + if method is not None: + return method(node, *args, **kwargs) + raise UnsupportedWrapperCodegenNodeError(type(self), type(node), self.visitor_method_prefix) + + def _visitor_method_name(self, node_type: type) -> str: + """Return the handler method name for ``node_type``.""" + return f"{self.visitor_method_prefix}_{node_type.__name__}" From 38d0dfc4034644421369238eccdd2908fb4e8f0c Mon Sep 17 00:00:00 2001 From: said Date: Sat, 11 Jul 2026 22:31:10 +0100 Subject: [PATCH 04/30] finish with implementing phase 0 --- .../wrapper-plan-migration-checklist.md | 53 +++++- .../test_phase0e_backend_foundation.py | 132 +++++++++++++ x2py/wrapper_codegen/__init__.py | 50 +++++ x2py/wrapper_codegen/assembly.py | 37 ++++ x2py/wrapper_codegen/names.py | 119 ++++++++++++ x2py/wrapper_codegen/nodes.py | 173 ++++++++++++++++++ x2py/wrapper_codegen/source_printers.py | 159 ++++++++++++++++ 7 files changed, 715 insertions(+), 8 deletions(-) create mode 100644 tests/wrapper_codegen/test_phase0e_backend_foundation.py create mode 100644 x2py/wrapper_codegen/assembly.py create mode 100644 x2py/wrapper_codegen/names.py create mode 100644 x2py/wrapper_codegen/nodes.py create mode 100644 x2py/wrapper_codegen/source_printers.py diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 2c5437e01..0a50431ef 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -1297,6 +1297,43 @@ bridge ABI slots, native slots, and lifecycle order. It must not render backend nodes, C/Fortran source, CPython reference-counting mechanics, or printer output. +### Phase 0E Minimal Scalar Backend Foundation Contract + +Phase 0E introduces isolated backend syntax, naming, source assembly, and +printer infrastructure only. It does not add scalar plan emitters, binding or +bridge handlers, runtime helper implementations, compilation, route selection, +or generated artifact parity checks. + +The legacy inventory for the first scalar backend foundation is: + +| Backend concern | Legacy Python source owner | Consumed behavior | Isolated Phase 0E choice | +| --- | --- | --- | --- | +| Generated symbol naming | `x2py/codegen/scope.py::Scope.get_new_name` | deterministic unique names and collision avoidance | rewrite as `NameAllocator`; no semantic scopes, categories, or lookup ownership | +| Module/function contexts | `x2py/codegen/scope.py::Scope` plus binding/bridge generator-local state | per-module and per-function local names | rewrite as explicit `ModuleEmissionContext` and `FunctionEmissionContext` | +| Generic node structure | `x2py/codegen/models/core.py::Module`, `FunctionDef`, `FunctionDefArgument`, `Return`, `Import`, `Declare` | module imports, function signatures, parameters, declarations, statements, returns | rewrite minimal frozen C/Fortran node dataclasses with only printer-consumed fields | +| Scalar datatype/literal spelling | `x2py/codegen/models/datatypes.py` scalar `Numpy*Type` classes and `convert_to_literal` | C/Fortran scalar type spellings and literal text | rewrite as explicit `BackendScalarType` and expression/literal text nodes | +| CPython/NumPy include/API concepts | `x2py/codegen/bindings/cpython_api.py`, `numpy_cpython_api.py`, and `CPythonCodePrinter` | `Python.h`, `numpy/arrayobject.h`, `PyObject*`, `PyArg_ParseTupleAndKeywords`, scalar conversion helper names, `PyMethodDef`, `PyModuleDef`, `import_array` | represent only include/API references as isolated node data; Phase 1 handlers decide which primitives to emit | +| C and header printing | `x2py/codegen/printers/ccode.py::CCodePrinter` and `cpythoncode.py::CPythonCodePrinter` | include lines, header guards, prototypes, function signatures, declarations, expression statements, returns | rewrite `CSourcePrinter` over isolated C nodes only | +| Fortran bridge printing | `x2py/codegen/printers/fcode.py::FCodePrinter` | module/use/contains, `bind(c, name=...)`, scalar declarations, assignments, calls | rewrite `FortranSourcePrinter` over isolated Fortran nodes only | +| Source assembly orchestration | `x2py/codegen/binding_pipeline.py::BindingPipeline.generate/write` | create complete C source, C header, and Fortran module objects, then print modules | rewrite `BackendSourceAssembly.rendered_sources()` for in-memory source strings only | + +Every Phase 0E node field is consumed by a Phase 0E printer, source assembly, +or context test. Phase 1 scalar emitters will be the first production +consumers of these nodes; until then tests construct small representative +modules directly. This phase intentionally keeps CPython reference counting, +argument parsing, scalar conversion helper bodies, bridge call bodies, runtime +support installation, file writes, compiler/linker calls, and route eligibility +outside the implementation. + +Structural guarantees for this phase: + +- no `x2py.wrapper_codegen` backend module imports `x2py.codegen`; +- C and Fortran source printers accept only isolated backend nodes and do not + import or inspect wrapper-plan models; +- generated-source assertions stay focused on naming, declarations, + signatures, header guards, includes, `bind(c)` spelling, and + module-to-printer orchestration rather than full runtime wrapper snapshots. + ### Intermediate Test Contract Add intermediate tests only where they protect a stable boundary or a failure @@ -1601,28 +1638,28 @@ value review rather than continuing automatically. ### Phase 0E — Minimal Scalar Backend Foundation -- [ ] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, +- [x] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, datatype/literal behavior, CPython and NumPy API primitives, naming behavior, helper concepts, and printer cases required for Phase 1. Record each legacy source path and consumed field/method before implementation. -- [ ] Implement a minimal `NameAllocator` and module/function emission contexts; +- [x] Implement a minimal `NameAllocator` and module/function emission contexts; do not copy legacy `Scope` or its semantic lookup/categories. -- [ ] For every required node/API/helper class, choose explicitly between a +- [x] For every required node/API/helper class, choose explicitly between a small unchanged copy and a rewritten minimal class. Each new field and method must have a current Phase 1 emitter or printer consumer. -- [ ] Do not import, alias, subclass, or adapt legacy model classes. Preserve +- [x] Do not import, alias, subclass, or adapt legacy model classes. Preserve required behavior through the isolated implementation and later compiled parity evidence. -- [ ] Add focused tests only for nontrivial naming/node/printer mechanics that +- [x] Add focused tests only for nontrivial naming/node/printer mechanics that the selected existing wrapper fixture cannot isolate. Do not add exhaustive node tests or full generated-source snapshots. -- [ ] Reproduce the legacy module/header assembly and +- [x] Reproduce the legacy module/header assembly and `module -> doprint(module)` orchestration needed to create complete `c_module` and `fortran_module` objects before writing source. -- [ ] Implement only the C/Fortran printer cases needed by the isolated scalar +- [x] Implement only the C/Fortran printer cases needed by the isolated scalar nodes. Verify structurally that the printers consume only isolated nodes and cannot import or inspect wrapper-plan models. -- [ ] Do not add scalar emitters, compilation, or production route selection in +- [x] Do not add scalar emitters, compilation, or production route selection in this phase. ## Phase 1 — Scalar Function Inputs diff --git a/tests/wrapper_codegen/test_phase0e_backend_foundation.py b/tests/wrapper_codegen/test_phase0e_backend_foundation.py new file mode 100644 index 000000000..dc0e1af5b --- /dev/null +++ b/tests/wrapper_codegen/test_phase0e_backend_foundation.py @@ -0,0 +1,132 @@ +"""Phase 0E tests for isolated scalar backend foundations.""" + +from __future__ import annotations + +import ast +from pathlib import Path + +import pytest + +from tests.wrapper.fortran._support import REPO_ROOT +from x2py.wrapper_codegen import ( + ApiReference, + BackendScalarType, + BackendSourceAssembly, + CDeclaration, + CExpressionStatement, + CFunction, + CFunctionPrototype, + CHeader, + CInclude, + CModule, + CParameter, + CReturn, + CSourcePrinter, + CodeExpression, + FortranAssignment, + FortranFunction, + FortranModule, + FortranParameter, + FortranSourcePrinter, + FortranUse, + HandlerRegistryPlan, + ModuleEmissionContext, + ModulePlan, + NameAllocator, + UnsupportedWrapperCodegenNodeError, +) + + +def test_name_allocator_sanitizes_keywords_and_reuses_module_context_names(): + allocator = NameAllocator(("x",)) + context = ModuleEmissionContext("demo", allocator) + + assert allocator.allocate("return") == "return_" + assert allocator.allocate("x") == "x_1" + assert allocator.allocate("2-value") == "x_2_value" + + function_context = context.function_context("demo.add") + assert function_context.local_name("x") == "x_2" + assert "x_2" in allocator.used_names + + +def test_backend_source_assembly_renders_complete_c_header_and_fortran_modules(): + float64 = BackendScalarType( + semantic_name="Float64", + c_spelling="double", + fortran_spelling="real(c_double)", + python_parse_unit="d", + numpy_type_macro="NPY_FLOAT64", + ) + api = ApiReference("import_array", include="numpy/arrayobject.h") + parameters = (CParameter("self", "PyObject *"), CParameter("args", "PyObject *")) + c_function = CFunction( + name="wrap_add_r8", + return_type="PyObject *", + parameters=parameters, + storage="static", + body=( + CDeclaration("x", float64.c_spelling, CodeExpression("0.0")), + CExpressionStatement(CodeExpression(f"{CSourcePrinter().doprint(api)}()")), + CReturn(CodeExpression("NULL")), + ), + ) + assembly = BackendSourceAssembly( + module_name="fmath", + c_header=CHeader( + guard="FMATH_WRAPPER_H", + includes=(CInclude("Python.h"),), + prototypes=(CFunctionPrototype("wrap_add_r8", "PyObject *", parameters),), + ), + c_module=CModule( + name="fmath_wrapper", + includes=(CInclude("Python.h"), CInclude(api.include), CInclude("fmath_wrapper.h", system=False)), + functions=(c_function,), + ), + fortran_module=FortranModule( + name="bind_c_fmath_wrapper", + uses=(FortranUse("iso_c_binding", ("c_double",)),), + procedures=( + FortranFunction( + name="bind_c_add_r8", + parameters=(FortranParameter("x", float64.fortran_spelling, ("value",)),), + result_name="result", + result_type=float64.fortran_spelling, + bind_name="ADD_R8", + body=(FortranAssignment("result", CodeExpression("x")),), + ), + ), + ), + ) + + rendered = assembly.rendered_sources() + + assert "#ifndef FMATH_WRAPPER_H" in rendered.c_header + assert "PyObject * wrap_add_r8(PyObject * self, PyObject * args);" in rendered.c_header + assert '#include "fmath_wrapper.h"' in rendered.c_source + assert "static PyObject * wrap_add_r8(PyObject * self, PyObject * args)" in rendered.c_source + assert "double x = 0.0;" in rendered.c_source + assert "import_array();" in rendered.c_source + assert "use iso_c_binding, only: c_double" in rendered.fortran_source + assert 'function bind_c_add_r8(x) result(result) bind(c, name="ADD_R8")' in rendered.fortran_source + assert "real(c_double), value :: x" in rendered.fortran_source + + +def test_source_printers_reject_wrapper_plan_models(): + plan = ModulePlan(owner_path="demo", functions=(), handler_registry=HandlerRegistryPlan((), (), ())) + + with pytest.raises(UnsupportedWrapperCodegenNodeError): + CSourcePrinter().doprint(plan) + with pytest.raises(UnsupportedWrapperCodegenNodeError): + FortranSourcePrinter().doprint(plan) + + +def test_source_printers_do_not_import_wrapper_plan_models(): + path = REPO_ROOT / "x2py" / "wrapper_codegen" / "source_printers.py" + imports = { + node.module + for node in ast.walk(ast.parse(Path(path).read_text(encoding="utf-8"))) + if isinstance(node, ast.ImportFrom) and node.module is not None + } + + assert "x2py.wrapper_codegen.plan" not in imports diff --git a/x2py/wrapper_codegen/__init__.py b/x2py/wrapper_codegen/__init__.py index fcaee0903..6e085ede5 100644 --- a/x2py/wrapper_codegen/__init__.py +++ b/x2py/wrapper_codegen/__init__.py @@ -6,6 +6,29 @@ from __future__ import annotations +from .assembly import BackendSourceAssembly, RenderedBackendSources +from .names import FunctionEmissionContext, ModuleEmissionContext, NameAllocator +from .nodes import ( + ApiReference, + BackendScalarType, + CDeclaration, + CExpressionStatement, + CFunction, + CFunctionPrototype, + CHeader, + CInclude, + CModule, + CParameter, + CReturn, + CodeExpression, + FortranAssignment, + FortranCall, + FortranDeclaration, + FortranFunction, + FortranModule, + FortranParameter, + FortranUse, +) from .plan import ( ActionHandlerPlan, ArgumentTransferPlan, @@ -24,22 +47,49 @@ ) from .planner import WrapperPlanner from .renderer import WrapperPlanRenderer +from .source_printers import CSourcePrinter, FortranSourcePrinter from .support import WrapperPlanSupportAnalyzer from .validator import WrapperPlanValidator from .visitor import ClassVisitor, UnsupportedWrapperCodegenNodeError __all__ = ( "ActionHandlerPlan", + "ApiReference", "ArgumentTransferPlan", + "BackendScalarType", + "BackendSourceAssembly", "BindingHandoffPlan", "BridgeAbiPlan", "BridgeAbiSlotPlan", + "CDeclaration", + "CExpressionStatement", + "CFunction", + "CFunctionPrototype", + "CHeader", + "CInclude", + "CModule", + "CParameter", + "CReturn", + "CSourcePrinter", "ClassVisitor", + "CodeExpression", + "FortranAssignment", + "FortranCall", + "FortranDeclaration", + "FortranFunction", + "FortranModule", + "FortranParameter", + "FortranSourcePrinter", + "FortranUse", + "FunctionEmissionContext", "FunctionPlan", "HandlerRegistryPlan", "LifecycleActionPlan", + "ModuleEmissionContext", "ModulePlan", + "NameAllocator", "NativeCallSlotPlan", + "RenderedBackendSources", "ResultPlan", "UnsupportedWrapperCodegenNodeError", "WrapperPlanDiagnostic", diff --git a/x2py/wrapper_codegen/assembly.py b/x2py/wrapper_codegen/assembly.py new file mode 100644 index 000000000..9dc3404b1 --- /dev/null +++ b/x2py/wrapper_codegen/assembly.py @@ -0,0 +1,37 @@ +"""In-memory source assembly for isolated wrapper backend modules.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from x2py.wrapper_codegen.nodes import CHeader, CModule, FortranModule +from x2py.wrapper_codegen.source_printers import CSourcePrinter, FortranSourcePrinter + + +@dataclass(frozen=True) +class RenderedBackendSources: + """Rendered in-memory source strings for one wrapper module.""" + + c_source: str + c_header: str + fortran_source: str + + +@dataclass(frozen=True) +class BackendSourceAssembly: + """Complete isolated backend module set before file writing.""" + + module_name: str + c_module: CModule + c_header: CHeader + fortran_module: FortranModule + + def rendered_sources(self) -> RenderedBackendSources: + """Render complete C source, C header, and Fortran source strings.""" + c_printer = CSourcePrinter() + fortran_printer = FortranSourcePrinter() + return RenderedBackendSources( + c_source=c_printer.doprint(self.c_module), + c_header=c_printer.doprint(self.c_header), + fortran_source=fortran_printer.doprint(self.fortran_module), + ) diff --git a/x2py/wrapper_codegen/names.py b/x2py/wrapper_codegen/names.py new file mode 100644 index 000000000..b86dc8b59 --- /dev/null +++ b/x2py/wrapper_codegen/names.py @@ -0,0 +1,119 @@ +"""Minimal naming and emission contexts for isolated wrapper backends.""" + +from __future__ import annotations + +from dataclasses import dataclass, field + + +class NameAllocator: + """Allocate deterministic unique generated identifiers.""" + + _KEYWORDS = frozenset( + { + "auto", + "break", + "case", + "char", + "const", + "continue", + "default", + "do", + "double", + "else", + "enum", + "extern", + "float", + "for", + "goto", + "if", + "int", + "long", + "return", + "short", + "signed", + "sizeof", + "static", + "struct", + "switch", + "typedef", + "union", + "unsigned", + "void", + "volatile", + "while", + } + ) + + def __init__(self, reserved: tuple[str, ...] = ()): + """Create an allocator with optional reserved names.""" + self._used = {self._sanitize(name) for name in reserved} + + def reserve(self, name: str) -> str: + """Reserve an exact generated identifier.""" + sanitized = self._sanitize(name) + self._used.add(sanitized) + return sanitized + + def allocate(self, preferred: str, *, prefix: str = "x") -> str: + """Return a unique identifier derived from ``preferred``.""" + base = self._sanitize(preferred, prefix=prefix) + candidate = base + index = 1 + while candidate in self._used: + candidate = f"{base}_{index}" + index += 1 + self._used.add(candidate) + return candidate + + @property + def used_names(self) -> tuple[str, ...]: + """Return allocated and reserved names in deterministic order.""" + return tuple(sorted(self._used)) + + def _sanitize(self, value: str, *, prefix: str = "x") -> str: + """Return a valid C-like identifier.""" + raw = value.strip() or prefix + chars = [char if self._is_identifier_char(char) else "_" for char in raw] + name = "".join(chars) + if not self._is_identifier_start(name[0]): + name = f"{prefix}_{name}" + if name in self._KEYWORDS: + name = f"{name}_" + return name + + def _is_identifier_start(self, char: str) -> bool: + """Return whether ``char`` may start an identifier.""" + return char == "_" or char.isalpha() + + def _is_identifier_char(self, char: str) -> bool: + """Return whether ``char`` may appear in an identifier.""" + return char == "_" or char.isalnum() + + +@dataclass +class ModuleEmissionContext: + """Backend-local state for one generated module.""" + + module_name: str + name_allocator: NameAllocator = field(default_factory=NameAllocator) + + def function_context(self, owner_path: str) -> FunctionEmissionContext: + """Return a function-local context sharing module names.""" + return FunctionEmissionContext( + module_name=self.module_name, + owner_path=owner_path, + name_allocator=self.name_allocator, + ) + + +@dataclass +class FunctionEmissionContext: + """Backend-local state for one generated function.""" + + module_name: str + owner_path: str + name_allocator: NameAllocator + + def local_name(self, preferred: str) -> str: + """Allocate one function-local generated name.""" + return self.name_allocator.allocate(preferred) diff --git a/x2py/wrapper_codegen/nodes.py b/x2py/wrapper_codegen/nodes.py new file mode 100644 index 000000000..c54109baa --- /dev/null +++ b/x2py/wrapper_codegen/nodes.py @@ -0,0 +1,173 @@ +"""Isolated backend syntax nodes for scalar wrapper source foundations.""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class BackendScalarType: + """Scalar type spelling consumed by C and Fortran printers.""" + + semantic_name: str + c_spelling: str + fortran_spelling: str + python_parse_unit: str | None = None + numpy_type_macro: str | None = None + + +@dataclass(frozen=True) +class CodeExpression: + """Raw expression text owned by a backend emitter or focused test.""" + + text: str + + +@dataclass(frozen=True) +class ApiReference: + """Named CPython, NumPy, or runtime helper API concept.""" + + name: str + include: str | None = None + + +@dataclass(frozen=True) +class CInclude: + """C include directive.""" + + header: str + system: bool = True + + +@dataclass(frozen=True) +class CParameter: + """C function parameter.""" + + name: str + type_name: str + + +@dataclass(frozen=True) +class CFunctionPrototype: + """C function prototype for generated headers.""" + + name: str + return_type: str + parameters: tuple[CParameter, ...] = () + + +@dataclass(frozen=True) +class CDeclaration: + """C local or global declaration.""" + + name: str + type_name: str + initializer: CodeExpression | None = None + + +@dataclass(frozen=True) +class CExpressionStatement: + """C expression statement.""" + + expression: CodeExpression + + +@dataclass(frozen=True) +class CReturn: + """C return statement.""" + + expression: CodeExpression | None = None + + +@dataclass(frozen=True) +class CFunction: + """C function definition.""" + + name: str + return_type: str + parameters: tuple[CParameter, ...] = () + body: tuple[CDeclaration | CExpressionStatement | CReturn, ...] = () + storage: str | None = None + + +@dataclass(frozen=True) +class CHeader: + """Generated C header module.""" + + guard: str + includes: tuple[CInclude, ...] = () + prototypes: tuple[CFunctionPrototype, ...] = () + + +@dataclass(frozen=True) +class CModule: + """Generated C source module.""" + + name: str + includes: tuple[CInclude, ...] = () + declarations: tuple[CDeclaration, ...] = () + functions: tuple[CFunction, ...] = () + + +@dataclass(frozen=True) +class FortranUse: + """Fortran use statement.""" + + module: str + only: tuple[str, ...] = () + + +@dataclass(frozen=True) +class FortranParameter: + """Fortran procedure argument declaration.""" + + name: str + type_name: str + attributes: tuple[str, ...] = () + + +@dataclass(frozen=True) +class FortranDeclaration: + """Fortran local or result declaration.""" + + name: str + type_name: str + attributes: tuple[str, ...] = () + + +@dataclass(frozen=True) +class FortranAssignment: + """Fortran assignment statement.""" + + target: str + expression: CodeExpression + + +@dataclass(frozen=True) +class FortranCall: + """Fortran call statement.""" + + function_name: str + arguments: tuple[CodeExpression, ...] = () + + +@dataclass(frozen=True) +class FortranFunction: + """Fortran function or subroutine body used by scalar bridges.""" + + name: str + parameters: tuple[FortranParameter, ...] = () + result_name: str | None = None + result_type: str | None = None + bind_name: str | None = None + declarations: tuple[FortranDeclaration, ...] = () + body: tuple[FortranAssignment | FortranCall, ...] = () + + +@dataclass(frozen=True) +class FortranModule: + """Generated Fortran module.""" + + name: str + uses: tuple[FortranUse, ...] = () + procedures: tuple[FortranFunction, ...] = () diff --git a/x2py/wrapper_codegen/source_printers.py b/x2py/wrapper_codegen/source_printers.py new file mode 100644 index 000000000..abeb3f169 --- /dev/null +++ b/x2py/wrapper_codegen/source_printers.py @@ -0,0 +1,159 @@ +"""Source printers for isolated scalar backend nodes.""" + +from __future__ import annotations + +from x2py.wrapper_codegen.nodes import ( + ApiReference, + CDeclaration, + CExpressionStatement, + CFunction, + CFunctionPrototype, + CHeader, + CInclude, + CModule, + CParameter, + CReturn, + FortranAssignment, + FortranCall, + FortranDeclaration, + FortranFunction, + FortranModule, + FortranParameter, + FortranUse, +) +from x2py.wrapper_codegen.visitor import ClassVisitor + + +class CSourcePrinter(ClassVisitor): + """Print isolated C source and header nodes.""" + + def doprint(self, node: object) -> str: + """Render one isolated C backend node.""" + return self.visit(node) + + def _visit_CModule(self, node: CModule) -> str: + """Render a complete C source module.""" + parts = [self.visit(include) for include in node.includes] + parts.extend(self.visit(declaration) for declaration in node.declarations) + parts.extend(self.visit(function) for function in node.functions) + return "\n\n".join(part for part in parts if part) + + def _visit_CHeader(self, node: CHeader) -> str: + """Render a complete C header module.""" + lines = [f"#ifndef {node.guard}", f"#define {node.guard}"] + lines.extend(self.visit(include) for include in node.includes) + lines.extend(self.visit(prototype) for prototype in node.prototypes) + lines.append(f"#endif /* {node.guard} */") + return "\n".join(lines) + + def _visit_CInclude(self, node: CInclude) -> str: + """Render one C include directive.""" + if node.system: + return f"#include <{node.header}>" + return f'#include "{node.header}"' + + def _visit_CFunction(self, node: CFunction) -> str: + """Render one C function definition.""" + prefix = f"{node.storage} " if node.storage else "" + body = "\n".join(f" {self.visit(statement)}" for statement in node.body) + return f"{prefix}{self._signature(node.return_type, node.name, node.parameters)} {{\n{body}\n}}" + + def _visit_CFunctionPrototype(self, node: CFunctionPrototype) -> str: + """Render one C function prototype.""" + return f"{self._signature(node.return_type, node.name, node.parameters)};" + + def _visit_CParameter(self, node: CParameter) -> str: + """Render one C parameter.""" + return f"{node.type_name} {node.name}" + + def _visit_CDeclaration(self, node: CDeclaration) -> str: + """Render one C declaration.""" + if node.initializer is None: + return f"{node.type_name} {node.name};" + return f"{node.type_name} {node.name} = {node.initializer.text};" + + def _visit_CExpressionStatement(self, node: CExpressionStatement) -> str: + """Render one C expression statement.""" + return f"{node.expression.text};" + + def _visit_CReturn(self, node: CReturn) -> str: + """Render one C return statement.""" + if node.expression is None: + return "return;" + return f"return {node.expression.text};" + + def _visit_ApiReference(self, node: ApiReference) -> str: + """Render one C API symbol reference.""" + return node.name + + def _signature(self, return_type: str, name: str, parameters: tuple[CParameter, ...]) -> str: + """Render a C function signature.""" + rendered = ", ".join(self.visit(parameter) for parameter in parameters) or "void" + return f"{return_type} {name}({rendered})" + + +class FortranSourcePrinter(ClassVisitor): + """Print isolated Fortran source nodes.""" + + def doprint(self, node: object) -> str: + """Render one isolated Fortran backend node.""" + return self.visit(node) + + def _visit_FortranModule(self, node: FortranModule) -> str: + """Render a complete Fortran module.""" + lines = [f"module {node.name}"] + lines.extend(f" {self.visit(use)}" for use in node.uses) + lines.extend([" implicit none", "contains"]) + lines.extend(self._indented(self.visit(procedure)) for procedure in node.procedures) + lines.append(f"end module {node.name}") + return "\n".join(lines) + + def _visit_FortranUse(self, node: FortranUse) -> str: + """Render one Fortran use statement.""" + if node.only: + return f"use {node.module}, only: {', '.join(node.only)}" + return f"use {node.module}" + + def _visit_FortranFunction(self, node: FortranFunction) -> str: + """Render one Fortran function.""" + signature = self._function_signature(node) + lines = [signature] + lines.extend(f" {self.visit(parameter)}" for parameter in node.parameters) + if node.result_name is not None and node.result_type is not None: + lines.append(f" {node.result_type} :: {node.result_name}") + lines.extend(f" {self.visit(declaration)}" for declaration in node.declarations) + lines.extend(f" {self.visit(statement)}" for statement in node.body) + lines.append(f"end function {node.name}") + return "\n".join(lines) + + def _visit_FortranParameter(self, node: FortranParameter) -> str: + """Render one Fortran parameter declaration.""" + return self._declaration(node.type_name, node.name, node.attributes) + + def _visit_FortranDeclaration(self, node: FortranDeclaration) -> str: + """Render one Fortran declaration.""" + return self._declaration(node.type_name, node.name, node.attributes) + + def _visit_FortranAssignment(self, node: FortranAssignment) -> str: + """Render one Fortran assignment.""" + return f"{node.target} = {node.expression.text}" + + def _visit_FortranCall(self, node: FortranCall) -> str: + """Render one Fortran call statement.""" + return f"call {node.function_name}({', '.join(argument.text for argument in node.arguments)})" + + def _function_signature(self, node: FortranFunction) -> str: + """Render a Fortran function signature.""" + args = ", ".join(parameter.name for parameter in node.parameters) + suffix = f" result({node.result_name})" if node.result_name is not None else "" + bind = f' bind(c, name="{node.bind_name}")' if node.bind_name is not None else "" + return f"function {node.name}({args}){suffix}{bind}" + + def _declaration(self, type_name: str, name: str, attributes: tuple[str, ...]) -> str: + """Render a Fortran declaration.""" + suffix = f", {', '.join(attributes)}" if attributes else "" + return f"{type_name}{suffix} :: {name}" + + def _indented(self, text: str) -> str: + """Indent a rendered procedure inside a module body.""" + return "\n".join(f" {line}" for line in text.splitlines()) From 8036c5455bd7df2e536662b95558a8e7ed2c26ee Mon Sep 17 00:00:00 2001 From: said Date: Mon, 13 Jul 2026 09:28:09 +0100 Subject: [PATCH 05/30] implement stage 2 and 3 --- .github/workflows/quality.yml | 2 + .../wrapper-plan-migration-checklist.md | 2117 +++++------------ docs/user/guide/fortran-wrapper.md | 5 +- tests/_shared/ownership_policy_support.py | 4 +- .../pyi_builds/test_contract_fixtures.py | 25 + .../test_rendered_wrapper_artifact_build.py | 155 ++ tests/pipeline/test_wrapper_plan_replay.py | 50 + .../test_wrapper_plan_route_selection.py | 475 ++++ .../test_accessor_and_storage_policy.py | 4 +- .../policy/test_scalar_wrapper_policy.py | 124 - tests/semantics/policy/test_wrapper_policy.py | 440 ++++ tests/wrapper/CHECKLIST_COVERAGE.md | 14 +- tests/wrapper/fortran/_support.py | 67 + .../test_contract_package_runtime.py | 66 +- .../wrapper/fortran/function_calls/README.md | 2 +- .../function_calls/test_optional_arguments.py | 91 +- .../test_scalar_writeback_plan.py | 86 + .../layout_rules/test_wrapper_guide_layout.py | 22 + tests/wrapper/fortran/module_state/README.md | 2 +- .../test_scalar_module_variable_plan.py | 164 ++ .../fortran/scalars/test_verified_baseline.py | 70 +- .../wrapper_codegen/test_phase0b_contracts.py | 58 +- .../wrapper_codegen/test_phase0d_plan_core.py | 341 +-- .../test_phase0e_backend_foundation.py | 105 +- .../test_phase1a_wrapper_assembly.py | 155 ++ .../test_phase1b_scalar_input_conversion.py | 33 + .../test_phase2a_scalar_results.py | 30 + .../test_phase2b_hidden_scalar_outputs.py | 29 + ...st_phase3_scalar_presence_and_writeback.py | 187 ++ .../test_phase4_scalar_module_variables.py | 137 ++ tools/check_wrapper_codegen_complexity.py | 20 + tools/replay_wrapper_plan.py | 298 +++ tools/wrapper_plan_staged_walkthrough.py | 183 ++ x2py/codegen/bind_c.py | 6 +- x2py/codegen/bridges/fortran_to_c.py | 27 +- x2py/pipeline/build.py | 496 +++- x2py/pipeline/wrapper_artifacts.py | 34 +- x2py/semantics/fortran2ir.py | 15 +- x2py/semantics/ir2ast.py | 36 +- x2py/semantics/models.py | 3 +- x2py/semantics/policy_completion.py | 12 +- x2py/semantics/scalar_wrapper_policy.py | 403 ---- x2py/semantics/wrapper_exports.py | 99 + x2py/semantics/wrapper_policy.py | 933 ++++++++ x2py/stage_values.py | 50 + x2py/wrapper_codegen/__init__.py | 89 +- x2py/wrapper_codegen/assembly.py | 37 - x2py/wrapper_codegen/c/__init__.py | 1 + x2py/wrapper_codegen/c/binding.py | 995 ++++++++ x2py/wrapper_codegen/checks.py | 97 +- x2py/wrapper_codegen/fortran/__init__.py | 1 + x2py/wrapper_codegen/fortran/bridge.py | 586 +++++ x2py/wrapper_codegen/generator.py | 610 +++++ x2py/wrapper_codegen/names.py | 119 - x2py/wrapper_codegen/nodes.py | 206 +- x2py/wrapper_codegen/plan.py | 274 ++- x2py/wrapper_codegen/planner.py | 450 +++- .../wrapper_codegen/primitive_scalar_types.py | 95 + x2py/wrapper_codegen/renderer.py | 101 - x2py/wrapper_codegen/source_printers.py | 240 +- x2py/wrapper_codegen/support.py | 119 +- x2py/wrapper_codegen/validator.py | 180 -- 62 files changed, 8770 insertions(+), 3105 deletions(-) create mode 100644 tests/pipeline/test_rendered_wrapper_artifact_build.py create mode 100644 tests/pipeline/test_wrapper_plan_replay.py create mode 100644 tests/pipeline/test_wrapper_plan_route_selection.py delete mode 100644 tests/semantics/policy/test_scalar_wrapper_policy.py create mode 100644 tests/semantics/policy/test_wrapper_policy.py create mode 100644 tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py create mode 100644 tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py create mode 100644 tests/wrapper_codegen/test_phase1a_wrapper_assembly.py create mode 100644 tests/wrapper_codegen/test_phase1b_scalar_input_conversion.py create mode 100644 tests/wrapper_codegen/test_phase2a_scalar_results.py create mode 100644 tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py create mode 100644 tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py create mode 100644 tests/wrapper_codegen/test_phase4_scalar_module_variables.py create mode 100644 tools/check_wrapper_codegen_complexity.py create mode 100644 tools/replay_wrapper_plan.py create mode 100644 tools/wrapper_plan_staged_walkthrough.py delete mode 100644 x2py/semantics/scalar_wrapper_policy.py create mode 100644 x2py/semantics/wrapper_exports.py create mode 100644 x2py/semantics/wrapper_policy.py create mode 100644 x2py/stage_values.py delete mode 100644 x2py/wrapper_codegen/assembly.py create mode 100644 x2py/wrapper_codegen/c/__init__.py create mode 100644 x2py/wrapper_codegen/c/binding.py create mode 100644 x2py/wrapper_codegen/fortran/__init__.py create mode 100644 x2py/wrapper_codegen/fortran/bridge.py create mode 100644 x2py/wrapper_codegen/generator.py delete mode 100644 x2py/wrapper_codegen/names.py create mode 100644 x2py/wrapper_codegen/primitive_scalar_types.py delete mode 100644 x2py/wrapper_codegen/renderer.py delete mode 100644 x2py/wrapper_codegen/validator.py diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index bcc575b83..4385610ee 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -49,6 +49,8 @@ jobs: run: python -m ruff check . - name: Ruff format run: python -m ruff format --check . + - name: Wrapper-plan generator contracts + run: python tools/check_wrapper_codegen_complexity.py - name: Bandit security scan run: python -m bandit -c pyproject.toml -r c_parser fortran_parser semantics x2py --severity-level medium --confidence-level medium - name: Vulture dead-code scan diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 0a50431ef..ce2fe472b 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -8,807 +8,361 @@ status: active-roadmap # Wrapper Plan Migration Checklist -This checklist replaces the broad "freeze every stage" idea with a smaller, -more useful target: build wrappers from an explicit, readable wrapper plan -instead of sending semantic IR through the current generic lowering/codegen -model first. +This file is the canonical implementation contract for wrapper-plan migration. +It replaces the generic semantic-IR wrapper lowering route one eligible module +at a time. The migration changes representation and generation organization; it +does not intentionally change the established Python, native ABI, ownership, or +build behavior of a migrated lane. -This file is the canonical implementation prompt for the migration. Route -granularity, plan structure, supported-lane definitions, validation rules, -emitter ownership, and cutover state must be recorded here before the related -code is changed. A phase heading is not by itself an implementation-ready -specification; the expansion rules below apply to the broader later phases. - -The long-term target is: +## Canonical Pipeline ```text -semantic IR +Semantic IR -> post-IR policy completion - -> wrapper-generation route selection - -> wrapper-plan route - -> mechanically project completed policy into wrapper plan - -> validate wrapper plan structure and handoffs - -> binding emitter + bridge emitter - -> complete generated wrapper artifact set - -> temporary legacy route - -> semantic_ir_to_codegen_ast() - -> existing bridge/binding generation - -> complete generated wrapper artifact set - -> shared compilation and link orchestration - -> importable extension + -> WrapperPlanner.build(module) + -> editable ModulePlan + -> WrapperCodeGenerator.generate(plan) + -> freeze and validate the received plan + -> recursively synthesize C binding nodes + -> recursively synthesize Fortran bridge nodes + -> print backend nodes + -> RenderedGeneratedWrapperArtifacts + -> existing build/link orchestration ``` -The route selects the complete runtime-wrapper generation path. It does not -select only argument lowering, one function, the binding, or the bridge. Both -routes must produce everything the shared build orchestration needs for one -extension: generated bridge source, generated C/CPython binding source and -headers, module initialization, required imports/includes/runtime support, -generated source compile requirements, and the names and files passed to the -existing compilation/link stage. - -Semantic `.pyi` generation and printing are not part of this route and remain -on their current path. The wrapper plan may consume semantic IR loaded from a -`.pyi` contract, but it does not replace or alter `.pyi` emission. - -The wrapper plan is the contract between the Python binding, the generated -native bridge, and the native call. A maintainer should be able to open the plan -for one wrapper function and see: - -- the Python-visible arguments and result order, -- decorator effects such as `@native_call`, `@bind`, `@raises`, overload - metadata, hidden native literals, and reordered native arguments, -- the binding action selected for each Python argument/result, -- the bridge handoff produced by binding and consumed by bridge, -- the bridge action selected for the native call, -- copy-in, copy-out, writeback, cleanup, and result projection phases, -- the exact dispatch key that selects each binding and bridge implementation - method. - -The plan should be simple enough for maintainers to read and reproduce by hand. -The supported way to change generated behavior is to change the semantic -contract or completed policy and rebuild the plan, not to patch the plan or add -a backend exception. Validation must fail before any C or Fortran source is -emitted when a completed policy or its mechanically derived plan is -inconsistent. - -## Migration Source Of Truth - -This is a representation and ownership migration, not a wrapper-behavior -redesign. The current production path through `semantic_ir_to_codegen_ast()`, -the existing bridge and binding generators, the source printers, and the build -orchestration is the behavioral source of truth for each lane until that lane -has completed parity evidence. - -- Derive each new plan action and handler from an audited current code path and - its wrapper tests. Do not invent a new conversion, ABI, call order, cleanup - order, public/generated-symbol naming contract, runtime helper protocol, - error behavior, or build artifact layout merely because the new - representation could support one. -- Preserve settled semantic/public contracts and previously documented bug - fixes when they are stricter than accidental current implementation behavior. - Any other intentional behavior change is separate work: document and test it - independently before changing the migration baseline. -- Recreate the minimum dependency-closed behavior needed by each lane inside the - isolated generator. For each CPython/NumPy API model, C/Fortran node, - datatype/literal, naming mechanism, or printer case, either copy a small - already-suitable implementation or rewrite a smaller class containing only - fields and methods the new emitters/printers use. These are independent - implementations, not imports, aliases, subclasses, or adapters around legacy - model classes. -- The legacy implementation is the behavioral source, not the required class - design. Record the legacy origin, consumed fields, emitted source behavior, - and parity evidence before simplifying. Do not preserve unused constructors, - properties, inheritance, mutation APIs, lookup categories, or future-facing - fields merely because the legacy class has them. -- Reuse the smallest coherent legacy method-body snippets and helper logic when - they remain straightforward after switching to the validated plan and - isolated context. When the legacy method depends on unrelated class state or - abstractions, rewrite a smaller direct handler from the recorded behavior - instead of copying that structure. Preserve observable call order, error - handling, cleanup, and generated-call behavior either way. -- Shared policy-completed semantic input, stable runtime APIs, the generated- - artifact handoff, and compilation/link orchestration may remain common. If a - new route needs changed runtime behavior, add a separately named helper used - only by that route until cutover; do not change the meaning of a helper still - used by the legacy generator. -- Keep the legacy route runnable and behaviorally unchanged while a lane is - introduced. Production route selection is explicit, and tests must be able to - invoke both routes for an eligible generation unit without a public - compatibility flag. -- A lane is not migrated merely because the new route compiles. It must match - the existing route's Python-visible results, native argument/result order, - mutation and writeback, exceptions, ownership/lifetime behavior, generated - artifact requirements, and success/failure cleanup behavior. -- Generated source text does not need to be byte-for-byte identical when the - same behavior and ABI are preserved. Backend-local names and equivalent - mechanical control flow may differ. Material differences must be explained by - the new mechanical organization, not by an unplanned semantic change. -- Until final cutover, reverting a lane means deliberately routing its whole - generation unit through the still-maintained legacy route. It does not mean - catching a new-route failure and silently retrying legacy generation. - -### Legacy Replay Procedure - -The working legacy pipeline makes this a reproduction exercise rather than a -greenfield generator project. Use it as an executable oracle every time a test -or semantic lane is migrated. - -1. Select an existing passing `tests/wrapper` test and run its complete - generation unit through the legacy route with generated artifacts retained. -2. Record the generated C binding source/header, Fortran bridge source, - generated artifact names and requirements, native-call order, runtime - behavior, and failure behavior relevant to the lane. -3. Trace those artifacts back through the current Python lowering, binding, - bridge, node/API-model, printer, runtime-helper, and build-orchestration - methods. Record the exact source methods and consumed state in this - checklist before porting them. -4. Complete any semantic decisions exposed by that trace in post-IR policy. - Do not move an old local decision into the planner or emitter merely because - that is the quickest textual copy. -5. Copy a small proven implementation unchanged when all of it is needed, or - rewrite the smallest independent equivalent when the old implementation - carries unrelated state. Start from the old method bodies and generated - artifacts; do not invent a replacement mechanism from memory. -6. Generate the same complete artifact set through the new plan route, inspect - differences against the retained legacy artifacts, then compile and run the - same existing assertions through both routes. - -Generated source is diagnostic evidence, not necessarily a byte-for-byte -golden file. Equivalent backend-local names or mechanical formatting are -allowed, but every ABI, conversion, ownership, cleanup, call-order, and build- -artifact difference must be explained. When a parity failure occurs, rerun and -inspect the working legacy route before changing the new implementation. - -Most wrapper tests share conversion, node, printer, and build mechanisms. Once -one mechanism has been reproduced and validated, later tests should reuse the -same new handler or primitive and add only their completed policy/plan mapping -or genuinely new mechanical behavior. - -## Route Selection Contract - -The initial migration route is atomic for one extension generation unit. In -the current build orchestration that unit is the merged semantic module used to -build one importable extension. - -- Route selection runs after post-IR policy completion and before - `semantic_ir_to_codegen_ast()`. -- The support check recursively inspects every runtime-visible or - runtime-required element in the merged module: functions, arguments, - results, hidden native arguments/results, decorator effects, module - variables, classes, constructors, properties, methods, overload dispatch, - cleanup, and build requirements. -- The wrapper-plan route is selected only when every such element is covered by - implemented plan actions, validators, binding handlers, and bridge handlers. -- No accepted decorator, native-call projection kind, or implicit call behavior - may be ignored by the support check. It must be implemented by a completed - lane or reported as an unsupported owner path that selects the legacy route. -- If any element is not covered, the whole generation unit uses the temporary - legacy route. Initially, one extension must not mix plan-generated functions - with legacy-generated functions, or a plan-generated binding with a - legacy-generated bridge. -- The route decision returns a structured support report with the selected - route and owner paths/reasons for every unsupported element. It must be - inspectable in tests and maintainer diagnostics. -- Unsupported migration coverage may select the legacy route. Invalid or - inconsistent completed policy must fail before route selection; an error in - plan construction, validation, emission, or compilation after the plan route - is selected must fail the build and must not silently retry the legacy route. -- Source-driven and semantic-`.pyi`-driven extension builds use the same route - selector and the same wrapper-plan builder after producing the merged, - policy-completed semantic module. - -Function-level or mixed-route generation can be considered only after the -module-level migration is complete and only if a concrete need justifies its -extra composition and validation rules. It is not part of this checklist. - -## Policy Authority And Planner Boundary - -The wrapper planner receives policy-completed semantic owners and converts them -into a wrapper plan. It is not another policy stage. +There is no public or wrapper-domain representation between `ModulePlan` and +backend syntax nodes. `CModule`, `CHeader`, `CFunction`, `FortranModule`, and +`FortranFunction` are direct printer inputs, not another wrapper planning +stage. -```text -semantic contract + semantic datatype facts - -> post-IR policy completion - -> complete module/function/class/variable/argument/result policies - -> WrapperPlanner.build(policy-completed module) - -> frozen WrapperPlan +The public generation boundary is deliberately small: + +```python +complete_semantic_policies(module) +plan = WrapperPlanner().build(module) +artifacts = WrapperCodeGenerator().generate(plan) ``` -- Datatype states what an object is: scalar family, precision, rank, shape, or - other representation facts. Completed policy states what wrapper generation - must do: Python conversion action, bridge/native action, ownership, transfer, - destruction, mutability/writeback, nullability, output projection, release, - storage mode, getter/setter behavior, native-call order, and lifecycle order. -- The planner copies those completed decisions and datatype facts into readable - plan records, assigns stable owner paths and symbolic references, and wires - already-decided producers to consumers. It may traverse owners, preserve - declared order, and build deterministic tuples and lookup references. -- The planner must not select an action, ownership mode, ABI behavior, call - order, writeback, cleanup, result projection, or handler from datatype, - `intent`, decorators, raw metadata, `is_alias`, dotted-variable shape, local - memory checks, or a missing policy field. Such a branch means policy - completion is incomplete and must be moved there. -- Planner complexity is presumed to indicate an incomplete policy boundary. - A planner branch is allowed only for structural traversal or deterministic - wiring that cannot change wrapper semantics. Any other exception requires a - concrete reason recorded in this checklist before implementation and a - focused test proving that it is structural rather than a hidden decision. -- Every runtime-visible or runtime-required owner must expose a complete typed - policy before planning. If an argument, result, function, class, module - variable, decorator effect, or native projection is still represented only - by scattered facts that the planner would need to interpret, extend post-IR - policy completion with a typed completed-policy record first. -- The plan retains the completed policy values or stable typed references to - them so validation and rendering can show why each action was selected. Plan - records must not use untyped `object` or free-form dictionaries for policy. -- Completed policy is the customization authority. A maintainer who wants - different generated behavior changes the contract/policy, reruns post-IR - policy completion when applicable, and rebuilds the plan. There is no direct - plan-transformation API and no backend-specific customization path. -- The validator checks that the plan is a faithful and structurally consistent - projection of completed policy. It diagnoses missing policy, unsupported - completed actions, broken handoffs, and producer/consumer mismatches; it does - not fill defaults or choose replacement behavior. - -## Design Rules - -- Post-IR policy completion decides semantic actions. The planner projects those - decisions into the plan, and binding and bridge emitters consume them. None of - these stages reconstruct policy from datatype, `intent`, `is_alias`, local - memory handling, dotted-variable shape, or missing policy. -- The wrapper plan records action keys, handoff expectations, and native-call - ordering. It does not contain raw generated C, Fortran, or CPython text. -- Binding and bridge emitters are dispatch tables plus small implementation - methods. Reuse completed policy actions directly when they already identify - the behavior, such as `PythonBarrierAction.SCALAR_VALUE` and - `NativeBarrierAction.PASS_VALUE`; do not create a parallel action vocabulary - merely to rename them in the plan. -- Each action has exactly one registered handler for its emitter. Validation and - plan rendering resolve that registry to the exact method name; emitters do not - dynamically construct method names or run a second policy-selection tree. -- Initial isolated handlers keep the current dispatch method names when practical, - so a plan action can be traced directly to the audited legacy method. Add a - new completed policy action in post-IR policy completion when no existing - action expresses the required semantic step; never invent a plan-only action - to avoid completing policy. -- Dispatch should be by semantic lane and action, not by every concrete dtype. - For example, `Float64`, `Int32`, and `Bool` can share scalar-value handlers - while dtype remains plan data. -- Every binding output that the bridge consumes is represented by a handoff - spec in one end-to-end argument/result transfer record. Every bridge native - argument/result that the native call consumes is represented in that same - record and the function's `BridgeAbiPlan`/`NativeCallPlan`. -- Validation checks producer/consumer consistency before emission: - the transfer's Python-side action must produce its handoff, its bridge ABI - slot must consume that handoff, its native action must satisfy the native-call - slot, and writeback/result plans must consume values actually produced. -- Keep one readable `ArgumentTransferPlan` for the complete Python argument -> C - handoff -> bridge ABI -> native argument path. Do not force maintainers to - join separate binding and bridge subplans. C and Fortran emitters remain - separate implementations that consume different fields of the same transfer - record. -- Local variable names may be generated by a plan context, but the plan should - carry stable symbolic roles such as `value`, `descriptor`, `shape`, `status`, - `message`, or `writeback`. -- The old lowering/codegen path may remain temporarily for unsupported lanes, - but the route must be explicit and tracked by lane and owner path. Do not hide - a semantic fallback inside binding, bridge, or printer code. -- Plan construction and validation are separate from source emission. Emitters - may create backend-local names and helper temporaries, but they may not add, - remove, reorder, or reinterpret semantic actions from the validated plan. -- Backend-specific resource mechanics do not belong in the wrapper plan. - CPython borrowed/new/stolen-reference rules, `Py_INCREF`/`Py_DECREF`, and - partial-failure reference cleanup are private binding-emitter mechanics. - Equivalent compiler/runtime details remain private to their backend. -- A plan may record backend-neutral lifetime relationships such as owned, - borrowed, transferred, retained owner, destruction responsibility, or - call-scoped cleanup. The selected backend method mechanically maps those - relationships and the target API's contract to its local resource handling; - doing so is implementation, not a new semantic decision. -- Isolation begins after post-IR policy completion. The new generator must not - import `x2py.codegen`, and legacy `x2py.codegen` modules must not import the - new generator. High-level pipeline orchestration is the only owner allowed to - select between them. -- Add backend primitives incrementally by lane, not as a wholesale clone of the - legacy codegen package. Each isolated slice includes only the dependency - closure needed to generate and print that lane, so its representation can - evolve without changing the legacy route. -- The binding and bridge emitters both consume the same validated plan. Their - generated runtime data flow is Python binding -> bridge -> native call, but - one emitter is not the semantic-policy input to the other. - -### Hierarchical Plan Ownership - -The plan mirrors semantic ownership. It is hierarchical rather than one flat -list and does not rely on one large planner method. +`WrapperCodeGenerator.generate` accepts `ModulePlan` only. It does not accept +semantic modules, build a plan itself, select an alternate lowering route, or +retry a prior route after direct generation begins. + +Semantic `.pyi` generation remains outside this route. A semantic `.pyi` +contract can supply the semantic module consumed by planning, but planning does +not change `.pyi` emission. + +## One Shared Plan, Explicit Backend Views + +`ModulePlan` is one shared semantic-and-ABI contract. It is not a C plan joined +to a Fortran plan and it does not contain backend nodes or source text. + +Every owner that crosses or coordinates the boundary has binding and bridge +child plans in the same editable tree: ```text -WrapperPlan - ModulePlan - VariablePlan ... - FunctionPlan - ArgumentPlan -> ArgumentTransferPlan ... - ResultPlan ... - BridgeAbiPlan - NativeCallPlan - WritebackPlan/CleanupPlan ... - ClassPlan - ConstructorPlan ... - MethodPlan ... - PropertyPlan ... - VariablePlan ... +ModulePlan + binding: BindingModulePlan + bridge: BridgeModulePlan + functions: FunctionPlan ... + binding: BindingFunctionPlan + bridge: BridgeFunctionPlan + arguments: ArgumentTransferPlan ... + binding: BindingArgumentPlan + bridge: BridgeArgumentPlan + result: ResultPlan | None + binding: BindingResultPlan + bridge: BridgeResultPlan + lifecycle: LifecycleActionPlan ... + binding: BindingLifecyclePlan | None + bridge: BridgeLifecyclePlan | None ``` -- Create one frozen plan record for each runtime-visible or runtime-required - semantic owner. The record carries its completed typed policy, datatype facts - where applicable, stable owner path, and references to its child records. -- The module planner preserves declared member order and visits module - variables, functions, and classes. The function planner visits arguments and - results and assembles the already-completed call, ABI, writeback, cleanup, and - projection records. The class planner does the same for constructors, - methods, properties, and fields. -- Hidden native arguments/results, decorator projections, reordered native - slots, and function-local lifecycle steps belong under their owning - `FunctionPlan`. They are not detached module-level plan objects. -- Each planner visitor method returns one plan node. Only the owning parent - assembles child nodes into ordered tuples; child visitors do not mutate a - shared plan or inspect sibling backend output. -- Cross-level and cross-backend relationships use typed references or stable - owner paths. Do not duplicate a child plan to make it available to both - emitters. -- Owners that do not affect runtime wrapper generation may be absent only when - the support analyzer explicitly classifies them as ignorable. An unsupported - runtime owner selects the whole legacy route before planning. - -### Visitor And Class Ownership - -Tree traversal in the isolated generator is class-based and follows one -visitor protocol. Production behavior is owned by named classes and methods, -not a collection of module-level orchestration functions. - -- Implement a minimal independent `wrapper_codegen.visitor.ClassVisitor` with - deterministic MRO dispatch and a configurable method prefix. Audit the - existing visitor algorithm as the behavioral source, but include only the - behavior the isolated planner, validators, emitters, and printers need. -- `WrapperPlanner(ClassVisitor)` traverses policy-completed semantic owners with - methods such as `_visit_SemanticModule`, `_visit_SemanticFunction`, - `_visit_SemanticClass`, and `_visit_SemanticVariable`. These methods only - copy completed policy/datatype facts and assemble deterministic plan records. -- `WrapperPlanSupportAnalyzer`, `WrapperPlanValidator`, and - `WrapperPlanRenderer` use the same visitor protocol for semantic or plan-node - traversal. The C binding and Fortran bridge emitters use visitors to assemble - module/function/argument/result structure. -- Isolated printers use the same protocol with - `visitor_method_prefix = "_print"` and explicit `_print_` methods. - Unsupported node types fail through the visitor's explicit default path. -- Visitor dispatch answers "which model/node type is this?" Completed-action - registries answer "which already-decided mechanical implementation runs?" - Do not replace action registries with `isinstance` ladders inside visitor - methods, and do not make visitor dispatch another policy selector. -- New production modules expose class APIs: `WrapperPlanner.build(...)`, - `WrapperPlanSupportAnalyzer.analyze(...)`, - `WrapperPlanValidator.validate(...)`, `WrapperPlanRenderer.render(...)`, and - `WrapperCodeGenerator.generate(...)`. Do not add equivalent module-level - builder, validator, renderer, support, emitter, printer, or generator - functions. -- A module-level production function is permitted only when a concrete Python - protocol or external entrypoint requires it and the reason is recorded in - this checklist before implementation. Dataclasses, enums, constants, test - functions, and a tool script's `main()` are not orchestration alternatives. -- Structural checks reject undeclared module-level functions in - `x2py.wrapper_codegen` and reject top-level `isinstance`/`match` traversal - ladders that bypass the visitor protocol. - -### Dispatch Size And Splitting - -- A primary emitter dispatcher is keyed by a completed action such as - `SCALAR_VALUE` or `PASS_VALUE`. Its selected method should implement one - understandable mechanical case and may delegate repeated node construction - to small private helpers. -- If a selected method grows because scalar families genuinely use different - APIs or generated control flow, add a visible secondary dispatcher keyed by - an explicit backend datatype family such as logical, integer, real, or - complex. Keep all precisions of a family together when precision changes only - type data or a conversion-table entry. -- Split again by precision or concrete datatype only when the emitted API, - range/error handling, declarations, or control flow actually differs. Do not - create one handler per dtype merely to keep methods short. -- Handler registries and plan rendering expose the complete selected chain, for - example `_convert_python_scalar_value_argument -> _convert_python_real_value`. - Missing primary or secondary combinations fail validation/emission rather - than falling back to a general method. -- When a method becomes difficult to follow, first separate independent phases - such as declaration, conversion, call argument, and cleanup into node-fragment - helpers. Use another dispatcher only when there is a real behavioral axis to - dispatch on. - -### Emitter Traceability And Complexity Gates - -The repository-wide Ruff/Radon limits protect legacy code but are too permissive -for the new emitter contract. The general changed-block Radon limit is 20 and -the Ruff McCabe limit is 45; neither means a maintainer can reproduce an emitter -step easily. Add a stricter static checker for `x2py.wrapper_codegen` before the -first emitter handlers are implemented. - -- Every registered primary handler, secondary handler, and private - node-fragment helper in `c/binding.py` or `fortran/bridge.py` has Radon - cyclomatic complexity at most 5 (grade A), at most 25 statements, and control- - flow nesting depth at most 2. -- Every `WrapperPlanner` visitor method has the same complexity, statement, and - nesting limits. It may use an explicit completed-policy variant only to - choose the corresponding plan-record shape. It may not dispatch on datatype, - decorators, or raw metadata to derive behavior; completed policy already - contains the selected actions and ordering. -- A registered handler accepts one validated transfer/result plan plus an - explicit backend emission context and returns an `EmissionFragment`. It must - not require a maintainer to recreate hidden mutable emitter state before - calling it in a focused test. -- Module/function assemblers and the top-level artifact orchestrator may have - cyclomatic complexity at most 10 and at most 50 statements because they - combine already-selected fragments; they may not perform policy or datatype - dispatch. -- New node, API-model, naming, and context methods are also kept at Radon - complexity at most 5 and contain only behavior required by current migrated - lanes. Copied printer methods may remain outside the stricter handler limit - when preserving complex legacy formatting; they remain subject to the - repository Radon non-regression policy and baseline rendering tests. -- Do not add a permanent complexity allowlist for a registered emitter handler. - A copied legacy handler that exceeds a limit may exist during isolated - baseline work, but the new route cannot become eligible for that action until - the handler is split and passes the strict gate. -- Complexity is a trigger for review, not an instruction to add a dispatcher - blindly. If complexity comes from independent emission phases, extract small - fragment helpers. If it comes from different datatype-family behavior, add a - secondary family dispatcher. If precision only selects a type/API table - value, keep one family handler and use data rather than another dispatcher. -- The checker also verifies that every registry target exists, every registered - handler is measured, no handler calls a printer, and every secondary - dispatcher has a total explicit mapping for the supported plan combinations. - It also enforces the declared module-level-function and visitor rules for the - isolated package. -- Plan rendering resolves and prints the full primary/secondary handler chain. - Registry checks verify that every rendered handler exists and is covered by a - supported plan combination. Compiled parity through existing wrapper tests is - the default behavioral proof; direct handler tests are reserved for - mechanical failures that existing fixtures cannot isolate. - -Implement this as a dedicated checker such as -`python3 tools/check_wrapper_codegen_complexity.py`. It may reuse Radon's -`cc_visit` and the repository's existing Radon helper code, while adding AST- -based statement count, nesting, registry, and forbidden-printer-call checks. -Run it as a blocking static check for every wrapper-codegen implementation -change. - -## Intended Ownership And Files - -The exact names may evolve during Phase 0, but responsibilities should remain -separated along these lines: +`ArgumentTransferPlan` remains the only argument-owner record; do not add a +generic duplicate `ArgumentPlan`. Its backend-facing child plans are +deliberately distinct and directly editable: + +- `binding: BindingArgumentPlan` describes the Python input, its C conversion + action, and + the C handoff value/role it produces; +- `bridge: BridgeArgumentPlan` describes the C ABI slot, value-versus-address + convention, + native action, and Fortran value that the bridge consumes; +- `native_call_slot` records the exact native-call position and source; +- result and lifecycle records identify later producers, consumers, ordering, + and responsibility through their own binding and bridge views. + +The binding input and bridge input may have different representations: a C +binding commonly receives `PyObject *`, produces a C scalar or address, and +the bridge then consumes a value or pointer according to its ABI slot. The plan +therefore records the producer/consumer handoff contract explicitly; it does +not assume identical types or actions. This keeps both backend plans in one +coherent editable tree while preventing them from drifting into disconnected +top-level plans. `CBindingGenerator` reads only binding views plus shared +handoff/order facts needed to create C nodes; `FortranBridgeGenerator` reads +only bridge views plus shared native-call order needed to create Fortran nodes. +Neither backend output is an input to the other backend. + +An owner may have only one active backend side when completed policy places the +behavior entirely in one backend, but that ownership must be explicit in the +plan rather than inferred during lowering. + +The plan includes every fact required for mechanical lowering: + +- owner path and plan-node kind; +- typed Python, native, and result actions; +- semantic datatype, datatype family, and precision/type facts; +- Python/native handoffs and bridge ABI slots; +- native-call slots in their exact order; +- result and output projection; +- ownership, transfer, destruction, mutability, writeback, release + responsibility, storage mode, nullability, and lifecycle ordering; and +- every typed lowering choice required by a supported backend. + +It contains decisions and facts, never generator method names. In particular, +there are no handler-name fields, handler records, or plan-owned handler +registries. The planner uses the completed policy actions already represented +by `PythonBarrierAction`, `NativeBarrierAction`, and `CodegenAction`; a new +typed action is added only when none of those can identify a necessary +mechanical behavior. Free-form string actions are forbidden. + +## Ownership Boundary + +Post-IR policy completion decides object kind, ownership, transfer, +destruction, mutability, writeback, nullability, output projection, release +responsibility, contract-value storage (`stack`, `heap`, or `alias`), getter +behavior, native setter assignment, Python setter exposure, ABI order, and +lifecycle order before planning starts. + +`WrapperPlanner` only projects those completed decisions and datatype facts +into readable editable records. It may traverse owners, preserve declared +order, assign stable owner paths, and wire already-decided producers to +consumers. It must not derive or replace policy from a datatype, `intent`, +decorator spelling, `is_alias`, dotted owner shape, local memory observation, +or a missing field. + +No code after planning may infer, replace, or override semantic policy. Backend +contexts may allocate temporary names and create declarations, error paths, +reference-count operations, and local cleanup statements after selecting a +typed lowering case. Those are emitted-code mechanics, not plan policy. + +`WrapperPlanner.build(module)` returns an editable `ModulePlan`. Maintainers +may edit its ordinary fields to inspect an experiment before generation. A +permanent behavior change belongs in the semantic contract and completed policy +rather than in a backend exception. + +## Generator-Owned Freezing and Validation + +Every consumer freezes the exact object it receives: ```text -x2py/wrapper_codegen/ - visitor.py minimal independent class visitor used by isolated models - names.py minimal deterministic NameAllocator; no legacy Scope copy - types.py minimal datatype/literal nodes needed by migrated lanes - - plan/ - models.py frozen readable plan records - specs.py handoff, bridge-ABI, native-slot, and structural specs - build.py WrapperPlanner visitor: completed policy -> WrapperPlan - validate.py WrapperPlanValidator visitor - support.py WrapperPlanSupportAnalyzer visitor and route report - render.py WrapperPlanRenderer visitor - - c/ - nodes.py minimal C declarations/statements needed by migrated lanes - cpython_api.py minimal CPython API nodes needed by migrated lanes - numpy_api.py minimal NumPy C API nodes needed by migrated lanes - concepts.py minimal C/CPython helper nodes needed by migrated lanes - context.py module/function names, local values, and cleanup state - binding.py CPythonBindingEmitter visitor - printer.py CPythonCodePrinter visitor with _print_ methods - - fortran/ - nodes.py minimal Fortran declarations/statements needed by lanes - context.py module/function names and local values - bridge.py FortranBridgeEmitter visitor - printer.py FCodePrinter visitor with _print_ methods - - generate.py WrapperCodeGenerator class and artifact orchestration - artifacts.py complete new-route generated-wrapper artifact result - -x2py/pipeline/ - shared route selection and compile/link orchestration +editable ModulePlan -- WrapperCodeGenerator --> frozen ModulePlan +editable backend modules -- source printers --> frozen backend modules +editable generated artifacts -- build integration --> frozen artifacts +``` + +At the start of `WrapperCodeGenerator.generate(plan)`, the generator must: + +1. recursively freeze that exact plan object; +2. run the complete structural validation on the final edited plan; and +3. reject inconsistent handoffs, ABI slots, native ordering, lifecycle roles, + or unsupported lowering combinations before it creates backend nodes. + +Later mutation of the received plan raises `FrozenStageRecordError`. Backend +nodes remain editable until their printer consumes them. Generated artifacts +remain editable until `_build_rendered_wrapper_extension(...)` consumes them. + +`WrapperPlanner` does not validate its output. It mechanically projects an +editable plan, which may temporarily be inconsistent while a maintainer edits +it. `WrapperCodeGenerator` owns the private structured validation methods and +is the only validation consumer. There is no standalone validator class or +public validation operation. + +Structural validation preserves these invariants: + +- binding producer and bridge consumer roles agree; +- bridge ABI coverage, positions, and owner roles are complete; +- native-call slot coverage, exact ordering, hidden literals, and hidden + results agree; +- direct and hidden result producer/consumer roles agree; +- writeback, cleanup, and release actions use available source roles in their + declared order; +- positions and symbolic roles are neither duplicate nor missing; and +- external and bind-target requirements are complete. + +## Direct Recursive Lowering + +`WrapperCodeGenerator` owns two private backend visitors: + +```python +c_module, c_header = CBindingGenerator().visit(plan) +fortran_module = FortranBridgeGenerator().visit(plan) ``` -This is a permanent responsibility-based package name, not a temporary -`new_codegen` package that would need another migration after cutover. Files -under `c/` and `fortran/` are added only as a migrated lane needs them; the -layout does not authorize copying the entire legacy package up front. - -Within the generation boundary there is no shared model identity between the -routes. If a migrated handler needs behavior currently provided by a legacy -`Variable`, datatype, literal, API model, scope, or printer-dispatch base, audit -which parts are actually consumed and implement the smallest independent class -that reproduces those parts. Copy a class unchanged only when it is already -small and all of it is required. - -### Naming Without Legacy Scope - -Do not copy the general legacy `Scope`. The wrapper plan already resolves -public names, native names, bridge symbols, argument positions, and ABI slots; -the new backend only needs deterministic collision-free names for generated -modules, functions, and local temporaries. - -- `NameAllocator` owns `reserve(name)` and `new_name(base)` with deterministic - suffixing. It does not store semantic variables, classes, decorators, - symbolic aliases, loops, dotted symbols, imports, or original-name lookup. -- A C or Fortran module emission context owns one module-level allocator for - generated functions, module objects, helper symbols, and public/ABI names - reserved from the validated plan. -- Each emitted function receives an explicit function context with its own - local allocator for arguments, result storage, temporaries, and cleanup - values. It may reserve module symbols but does not search a parent semantic - scope. -- Later class support may add a class emission context only when Phase 9 proves - it is needed. Do not add class/loop/program scope categories during scalar - phases. -- Public and ABI names come from the plan and cannot be renamed by the backend - allocator. The allocator handles only backend-local collisions. -- Focused naming tests copy the relevant legacy collision cases and prove stable - names across repeated generation, but the new allocator API and internal data - structures remain minimal. - -`ir2ast.py` remains the entry to the temporary legacy route; it must not choose -the route or partially invoke the new emitters. The source-driven and `.pyi`- -driven build entrypoints should call one shared orchestration helper rather -than implement different support rules. - -The two routes should return one pipeline-owned internal generated-artifact -result shape so the existing compiler/link orchestration does not need to -understand plan actions. That result is an internal build handoff, not a -compatibility API and not a second semantic model. - -The isolated C and Fortran layers are thin mechanical backend layers. For each -primitive, preserve the proven legacy behavior and tests needed by the lane -while choosing the smallest new representation that can express it. The -original node, printer, scope, and generator implementations remain untouched -and runnable. Shared compilation receives generated files through the artifact -result and does not import either route's internal models. - -## Node Construction And Printing Contract - -Emitters construct backend nodes; printers render backend nodes. Do not mix -these responsibilities. +They are private implementation organization inside direct generation, not +public stages. Both visitors recursively traverse the same plan tree and +return actual C or Fortran nodes (or tuples of actual nodes where a child needs +multiple declarations or statements). + +The recursive shape is: ```text -validated WrapperPlan - -> Fortran bridge emitter - -> isolated Fortran module/function/declaration/statement nodes - -> isolated Fortran printer - -> bridge source text - -> CPython binding emitter - -> isolated C module/function/declaration/statement nodes - -> isolated C source/header printers - -> binding source and header text - -> artifact assembler - -> complete generated-wrapper artifact result +ModulePlan + -> binding and bridge module contexts and backend nodes + -> NamespacePlan + -> directly owned FunctionPlan and ModuleVariablePlan records + -> binding/bridge argument transfers, result projection, lifecycle actions + -> complete backend function and namespace nodes + -> complete backend module node +``` + +An argument visitor returns the C or Fortran declarations/statements/parameters +needed for that backend. A result visitor returns the backend result nodes. +Lifecycle visitors return backend writeback, cleanup, or release nodes. Parent +visitors assemble these concrete child results directly into complete syntax +nodes. Do not introduce another wrapper-specific transport model. + +The public orchestration stays visibly direct: + +```python +class WrapperCodeGenerator: + def generate(self, plan: ModulePlan) -> RenderedGeneratedWrapperArtifacts: + plan.freeze() + self._validate_plan(plan) + self._c_generator.require_supported(plan) + self._fortran_generator.require_supported(plan) + + c_module, c_header = self._c_generator.visit(plan) + fortran_module = self._fortran_generator.visit(plan) + + c_source = self._c_printer.doprint(c_module) + c_header_source = self._c_printer.doprint(c_header) + fortran_source = self._fortran_printer.doprint(fortran_module) + return self._rendered_artifacts( + plan.owner_path, + c_source, + c_header_source, + fortran_source, + ) ``` -- Both emitters consume the same validated wrapper plan and its explicit bridge - ABI specification. The binding emitter must not consume the generated - Fortran AST, and the bridge emitter must not discover information that the C - binding needs. If either occurs, the shared plan or ABI specification is - incomplete. -- Preserve the legacy high-level pattern in isolated form: - `WrapperCodeGenerator.generate(plan)` constructs complete `fortran_module` - and `c_module` objects, then passes them to isolated - `FCodePrinter.doprint(...)` and `CPythonCodePrinter.doprint(...)` - implementations. Preserve this orchestration and the required module/header - behavior from legacy codegen rather than inventing a second source-writing - mechanism. -- Do not copy the legacy sequential dependency where the binding generator - learns its ABI by consuming the bridge generator's AST. In the new route, - both complete module trees are independently constructed from the same - validated `WrapperPlan` and `BridgeAbiPlan`. -- Each dispatched emitter method returns backend-node fragments such as - declarations, setup statements, call arguments, result statements, - success/failure cleanup, and produced symbolic values. A module assembler - combines those fragments into a complete C or Fortran module node. -- Emitters do not concatenate source text. CPython conversion and reference- - counting operations are represented by isolated C/API call nodes; Fortran - declarations, assignments, calls, and control flow use isolated Fortran - nodes. -- Printers accept only their backend nodes. They own syntax, indentation, - punctuation, fixed syntax templates, and mechanical rendering of represented - includes/imports. They must not accept `WrapperPlan`, inspect plan actions, - choose conversions, add lifecycle behavior, or repair incomplete modules. -- Includes, imports, public/generated symbols, function signatures, and header - declarations are selected by plan-driven emission and represented as nodes - before printing. A printer may deduplicate or order them mechanically. -- C source/header and Fortran source printers are independently testable against - the baseline nodes before plan emitters use them. Each isolated printer - implements only the `_print_` cases required by currently migrated - nodes and fails explicitly for unsupported node types; do not copy unused - printer methods in anticipation of later lanes. -- `plan/` does not import `c/` or `fortran/`; the C and Fortran backends do not - import each other; and backend printers import their nodes/types but not plan - builders, actions, validators, emitters, or pipeline routing. Add structural - tests for these internal dependency directions with the package skeleton. - -## Core Plan Shape - -The exact class names can evolve, but the first implementation should stay close -to this shape: +The generator constructs `RenderedGeneratedWrapperArtifacts` directly from the +printed source plus artifact metadata. It does not duplicate native build plans, +compiler selection, link ordering, runtime-support installation, or compilation +policy; those remain in existing build/link orchestration. + +## Direct Lowering Methods + +Each backend maps a completed plan action to one directly named method. The +single visible rule is: ```python -@dataclass(frozen=True) -class WrapperPlan: - extension_name: str - module: ModulePlan - requirements: WrapperArtifactRequirements - - -@dataclass(frozen=True) -class ModulePlan: - public_name: str - owner_path: OwnerPath - policy: CompletedModulePolicy - functions: tuple[FunctionPlan, ...] - variables: tuple[VariablePlan, ...] - classes: tuple[ClassPlan, ...] - - -@dataclass(frozen=True) -class FunctionPlan: - public_name: str - native_name: str - owner_path: OwnerPath - policy: CompletedFunctionPolicy - python_arguments: tuple[ArgumentPlan, ...] - bridge_abi: BridgeAbiPlan - native_call: NativeCallPlan - results: tuple[ResultPlan, ...] - writebacks: tuple[WritebackPlan, ...] - - -@dataclass(frozen=True) -class ArgumentPlan: - public_name: str - owner_path: OwnerPath - datatype: SemanticType - policy: OwnershipDecision - python_position: int | None - transfer: ArgumentTransferPlan - writeback: WritebackPlan | None = None - - -@dataclass(frozen=True) -class ArgumentTransferPlan: - python_action: PythonBarrierAction - handoff: HandoffSpec - bridge_slot: BridgeArgumentRef - native_action: NativeBarrierAction - native_slot: NativeArgumentRef - - -@dataclass(frozen=True) -class NativeCallPlan: - native_name: str - arguments: tuple[NativeArgumentRef, ...] - results: tuple[NativeResultRef, ...] +method_name = f"_lower_{subject}_{action.value}" ``` -The plan validator owns consistency diagnostics. Binding and bridge emitters -should be able to trust a validated plan and focus on emitted-code mechanics. -`CompletedModulePolicy` and `CompletedFunctionPolicy` stand for typed post-IR -policy records, not plan-owned decisions. Phase 0D must replace these sketch -names with the actual completed semantic policy types and define equivalent -typed policy fields for results, variables, and classes before those owners are -migrated. If policy completion cannot provide such a record, that semantic -stage must be completed before the corresponding planner visitor is written. +For example, an argument whose completed optional mode is `required` uses +`_lower_argument_required`; a module getter action `nullable_snapshot` uses +`_lower_module_getter_nullable_snapshot`. `lowering_method_name(subject, +action)` exposes that exact selection for inspection. Generator entry verifies +that every selected method exists before lowering begins, and unsupported +actions fail explicitly. No dispatcher dictionary or method-name string is +stored in the plan. + +Primitive dtype spelling and converter differences live in the intentionally +scalar-specific `PrimitiveScalarTypeRegistry`; they do not duplicate control +flow methods or select semantic policy. + +## Migration and Route Rules + +The legacy route remains the behavioral oracle until a lane has direct-plan +parity. Route selection is atomic per merged extension: a generation unit uses +either the direct wrapper-plan route or the legacy route. It never combines one +backend from one route with the other backend from the other route. -The first implementation also needs two non-semantic orchestration records: +An unsupported owner may select the legacy route before planning. Once the plan +route is selected, planning, validation, lowering, printing, or compilation +failure fails the build; it must not fall back to legacy generation. -- a route support report naming the generation unit, selected route, covered - lanes, and unsupported owner paths/reasons; -- a generated wrapper artifact result naming the complete bridge/binding - sources, headers, imports/includes/runtime requirements, generated source - compilation requirements, and extension initialization name. +For each lane: + +1. replay an existing passing `tests/wrapper` case through the legacy route and + retain its generated artifacts; +2. record the relevant legacy source paths, ABI/call order, ownership and + cleanup behavior, artifact requirements, and runtime assertions; +3. complete every missing semantic decision before planning; +4. add the smallest required plan record and directly named lowering method; +5. produce the same complete artifact set through the direct route; +6. inspect differences, compile both routes, and run the existing assertions; +7. update checklist evidence only after direct-route parity is proven. -These records must not duplicate the native object/library/link plan already -owned by build orchestration. +Generated source is diagnostic evidence, not a byte-for-byte golden. Backend +temporary names and equivalent control flow may differ, but ABI, conversion, +ownership, cleanup, call order, and artifact requirements must remain proven. -## Worked Scalar Trace +During this migration the full real-library BLAS/LAPACK wrapper corpus is +excluded locally and in CI until final cutover. General native-bundle coverage +remains active. -For a Python-visible scalar procedure equivalent to: +## Staged Walkthrough + +`tools/wrapper_plan_staged_walkthrough.py` is the maintained hand-inspection +path. It shows only the source/contract entry, policy completion, plan creation, +a direct edit, direct generation, artifact inspection, build, and runtime use: ```python -def f(x: Float64) -> None: ... +module = ... +complete_semantic_policies(module) + +plan = WrapperPlanner().build(module) +namespace = next(item for item in plan.namespaces if item.python_path == ()) +function = namespace.functions[0] +function.bridge.native_name = "SUB_R8" + +binding = CBindingGenerator() +bridge = FortranBridgeGenerator() +print(binding.lowering_method_name("argument", function.arguments[0].binding.optional_mode)) +print(bridge.lowering_method_name("argument", function.arguments[0].bridge.optional_mode)) + +artifacts = WrapperCodeGenerator( + c_generator=binding, + fortran_generator=bridge, +).generate(plan) + +# inspect generated files +# build and run ``` -assume post-IR policy completion produces the existing actions -`PythonBarrierAction.SCALAR_VALUE` and -`NativeBarrierAction.PASS_VALUE`. The maintainer-visible plan rendering should -stay approximately this small: +It does not expose standalone validation. Printed plan inspection uses the +actual namespace and owner records, and `lowering_method_name(subject, action)` +shows the exact directly named implementation method selected in each backend. -```text -function f(x: Float64) -> None - argument x - binding action scalar_value - binding handler CPythonBindingEmitter._convert_python_scalar_value_argument - produces x.value : Float64 - bridge ABI f_bridge.x consumes x.value - bridge action pass_value - bridge handler FortranBridgeEmitter._convert_native_value_argument - native slot f argument 0 <- x.value - result none -``` +## Required Evidence -The exact class-owned call path should be equally direct: +Focused tests must prove: -```text -completed = complete_semantic_policies(module) existing semantic stage -plan = WrapperPlanner().build(completed[0]) completed policy -> WrapperPlan -WrapperPlanValidator().validate(plan) structural consistency only -WrapperCodeGenerator().generate(plan) - CPythonBindingEmitter.emit_function(f) - -> _convert_python_scalar_value_argument(x) - -> isolated C node fragments - FortranBridgeEmitter.emit_function(f) - -> _convert_native_value_argument(x) - -> isolated Fortran node fragments - CPythonCodePrinter.doprint(c_module) nodes -> C source/header - FCodePrinter.doprint(fortran_module) nodes -> Fortran source -create_shared_library(...) existing compilation/link entrypoint -``` +- `WrapperPlanner.build(module)` returns a directly mutable plan; +- direct edits to binding and bridge views change the relevant generated C and + Fortran source; +- `WrapperCodeGenerator.generate(plan)` freezes the exact consumed plan; +- module visitors recursively include generated function nodes; +- function visitors recursively include argument, result, and lifecycle nodes; +- directly named backend lowering methods cover every supported plan action; +- unsupported combinations fail explicitly; +- source printers freeze backend module nodes; +- generated artifacts remain editable until build consumption, which freezes + them; +- source and semantic-`.pyi` entries preserve compiled runtime parity; and +- backend lowering does not reconstruct semantic policy. + +Use package-export inspection and focused migration checks to prove removal of +obsolete internal representations; do not preserve tests whose only assertion +is that a removed API is absent. -`WrapperPlanner._visit_SemanticFunction(f)` visits `x`, copies its `Float64` -datatype and completed `SCALAR_VALUE`/`PASS_VALUE` policy actions, and wires the -pre-decided positions into `ArgumentTransferPlan`. It does not decide that a -`Float64` should use those actions. A different valid completed policy produces -a different plan through the same visitor without changing planner or emitter -code. - -At runtime, the generated CPython binding converts the Python argument into the -scalar C handoff, calls the generated bridge symbol, and the bridge invokes the -native procedure using the completed `PASS_VALUE` behavior. CPython reference -counting, concrete temporary names, C declarations, Fortran declarations, and -printer formatting are deliberately absent from the rendered plan. - -A scalar plan that requires a maintainer to inspect backend nodes or printer -code to discover either selected handler has failed the readability goal. The -rendered plan is the normal trace; backend nodes and printers are inspected only -when debugging how a selected handler emits source. - -## Decorator And Native-Projection Coverage - -Decorator and projection handling is part of route support, not an emitter -detail. Maintain a matrix in this section as implementation proceeds. Each row -must eventually name its exact plan representation, validation rules, binding -handler, bridge handler, and focused tests. - -| Contract effect | Owning phase | Initial route rule | -| --- | --- | --- | -| Direct scalar call with implicit native order | Phase 1 | Eligible after scalar input actions are complete | -| `@bind(...)` native symbol selection | Phase 1 | Legacy route until symbol selection is explicit in `NativeCallPlan` | -| `@external` native target selection | Phase 1 | Legacy route until target/source-language requirements are explicit | -| `@hold_gil` call behavior | Phase 1 | Legacy route until GIL behavior is an explicit binding call phase | -| `@native_call` scalar `Arg(...)` reordering and `Addr(Arg(...))` | Phase 1 | Legacy route until every native slot and address handoff validates | -| `@native_call` typed numeric/logical hidden literals | Phase 1 | Legacy route until literal type, value, and native slot validate | -| `@native_call` scalar `Return(...)`, `Work(...)`, and direct native result projection | Phase 2 | Legacy route until result/workspace production and consumption validate | -| `@raises(...)` status/message projection | Phase 2 | Legacy route until status, message, success rule, and Python error path validate | -| Optionality and `IsPresent(...)` | Phase 3 | Legacy route until omitted, explicit `None`, present, and presence-token paths validate | -| String `Len(...)` and typed string literals | Phase 5 | Legacy route until the expanded string sub-lanes are complete | -| Array shape/stride/size/itemsize and conversion projections | Phase 6 | Legacy route until the expanded ordinary-array sub-lanes are complete | -| `Allocatable(...)` and `Pointer(...)` native projections | Phase 7 | Legacy route until the expanded descriptor/handle sub-lanes are complete | -| Derived/native type metadata | Phases 8-9 | Legacy route until the relevant derived-type and class sub-lanes are complete | -| `Pass()`, methods, constructors, properties, and `@overload(...)` | Phase 9 | Legacy route until the expanded class sub-lanes are complete | -| Callback decorators, adapters, and trampoline behavior | Phase 10 | Legacy route until the expanded callback sub-lanes are complete | - -When the live parser or semantic model accepts an effect missing from this -matrix, add it before implementing or routing that case. Do not treat the table -as proof that every current syntax spelling has already been audited; Phase 0 -owns that live inventory. +## Recovered Roadmap Scope + +The detailed migration queue below is retained from the original roadmap. The +obsolete Phase 0-2 emitter/fragment architecture is replaced by the simplified +direct-plan checklist later in this file; all later semantic lanes, matrix rows, +verification gates, and completion records remain explicit. ## Existing Wrapper Suite As The Migration Queue @@ -819,8 +373,8 @@ route easier to exercise. - Phase 0A adds a maintained migration matrix to this file covering every Python test node under `tests/wrapper`. Each row records whether the test - generates a wrapper, the source/contract generation unit it uses, the lanes - that currently block the wrapper-plan route, and one status: + generates a wrapper, the source/contract generation unit it uses, its + relevant feature lanes, and one status: `not-applicable`, `deferred-real-library`, `legacy`, `dual-route`, or `wrapper-plan`. - Existing source files, contract fixtures, build helpers, runtime assertions, @@ -867,11 +421,40 @@ new wrapper test node must either match an existing row intentionally or add a new row here before later implementation starts. Statuses have the meanings defined above: `legacy` still uses the current -`semantic_ir_to_codegen_ast()` route, `not-applicable` does not generate a -runtime wrapper, and `deferred-real-library` is reserved for the full BLAS and -LAPACK corpus until Phase 12. - -| Pytest selector | Generation unit | Blocking lanes | Status | +`semantic_ir_to_codegen_ast()` route, `dual-route` runs the same generation +unit and runtime assertions through both implementations, `wrapper-plan` uses +only `WrapperPlan -> WrapperCodeGenerator`, `not-applicable` does not generate +a runtime wrapper, and `deferred-real-library` is reserved for the full BLAS +and LAPACK corpus until Phase 12. + +#### Current Wrapper Route Counts + +These are collected pytest-node counts, not matrix-row counts. The structural +layout test derives them from live `tests/wrapper` collection and fails if this +summary, the exhaustive matrix, and the test tree disagree. + +| Status | Collected nodes | +| --- | ---: | +| `wrapper-plan` | 0 | +| `dual-route` | 17 | +| `legacy` | 178 | +| `not-applicable` | 95 | +| `deferred-real-library` | 2 | + +Migration is complete only when `legacy`, `dual-route`, and +`deferred-real-library` are all zero. At that point every runtime-generating +node must be `wrapper-plan`; `not-applicable` may remain only for tests that do +not generate a wrapper. Until then, moving a node from `legacy` to `dual-route` +records proven parity, and moving it from `dual-route` to `wrapper-plan` +records final removal of its legacy execution. + +#### Complete Route Ledger + +For a `legacy` row, the feature-lane column identifies what still blocks the +new route. For `dual-route` and `wrapper-plan` rows, it identifies the behavior +already covered by the new generator. + +| Pytest selector | Generation unit | Feature lanes / blockers | Status | | --- | --- | --- | --- | | `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | | `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | @@ -885,6 +468,7 @@ LAPACK corpus until Phase 12. | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_cycles_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_missing_relative_contract_before_native_validation` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_both_routes[*]` | forced legacy/direct-plan parity | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `dual-route` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_rejects_colliding_wildcard_exports` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | @@ -960,8 +544,10 @@ LAPACK corpus until Phase 12. | `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | direct wrapper/build route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | deliberate legacy/direct-plan replay using the existing `foptional_fixed.f` generation unit and shared runtime/failure assertions | optional/presence; scalar inputs/results; build/artifact integration | `dual-route` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | deliberate legacy/direct-plan replay of one semantic-.pyi descriptor contract against the same native module | optional/presence; nullable scalar descriptor; build/artifact integration | `dual-route` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | coverage-gap legacy/direct-plan replay for immutable scalar replacement return | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `dual-route` | | `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | @@ -970,6 +556,7 @@ LAPACK corpus until Phase 12. | `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route[*]` | coverage-gap legacy/direct-plan replay for a whole module containing only Phase 1-4 scalar owners | scalar inputs/results; scalar module variables/state; build/artifact integration | `dual-route` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | @@ -1001,6 +588,7 @@ LAPACK corpus until Phase 12. | `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | | `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes[*]` | deliberate legacy/direct-plan replay using the existing `fmath.f` and `fmath_f90.f90` generation units and shared runtime assertions | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `dual-route` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | @@ -1011,365 +599,6 @@ LAPACK corpus until Phase 12. | `tests/wrapper/fortran/strings/test_character_edge_cases.py::*` | source/generated-.pyi parity or parametrized route | strings; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/strings/test_string_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -### Phase 0A Current Scalar Baseline - -The first migration baseline is -`tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]`. -It is the smallest existing runtime generation unit that directly covers the -initial scalar lane without arrays, strings, module variables, classes, hidden -native outputs, native result projections, or build-bundle behavior. Its -callables do use the current scalar `@bind(...)`, `@external`, and -`@native_call([Addr(Arg(...))])` contract, so the first scalar lane must treat -bind names, external status, scalar address projection, native-call slot order, -and scalar result ownership as completed post-IR policy. - -That unit must remain atomic: - -- Source mode builds the current source fixture named by `SCALAR_LEGACY_SOURCE` - through the normal source wrapper route. -- Generated-`.pyi` mode first checks the generated semantic contract fixture at - `tests/wrapper/fortran/scalars/contracts/fmath`, compiles the native object, - then builds the same runtime wrapper surface from the checked contract. -- Both modes assert the same public runtime behavior through - `_assert_fmath_examples(...)`: lower-case scalar functions accept scalar - Python/NumPy values and return scalar Python/NumPy-compatible results for - real, integer, complex, and logical families. -- The expected generated artifact set is - `bind_c_fmath_wrapper.f90`, `fmath_wrapper.c`, and `fmath_wrapper.h`, plus - shared runtime support installed by the compilation pipeline. - -This fixture is not Phase 1-only. The Python arguments are scalar value inputs, -but every callable also has a scalar result. The whole generation unit cannot -move to `dual-route` until Phase 1 scalar inputs and Phase 2 scalar result -projection are both represented, validated, emitted, compiled, and compared -against the legacy route. - -#### Current scalar route inventory - -| Current behavior or path | Legacy source owner | Proposed wrapper-plan record/action | Required backend behavior | Baseline evidence | -| --- | --- | --- | --- | --- | -| Complete source or generated-`.pyi` semantic module before runtime generation | `x2py/pipeline/build.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/policy_completion.py`, `x2py/semantics/readiness.py` | Route selector receives one policy-completed generation unit | No route selection before policy completion and readiness-owned blockers | `test_fortran_wrapper_pipeline_builds_importable_extension[*]` | -| Lower policy-completed semantic functions to the current codegen AST | `x2py/semantics/ir2ast.py::semantic_ir_to_codegen_ast` | `WrapperPlanner` copies completed module/function/argument/result policy into `ModulePlan`, `FunctionPlan`, `ArgumentTransferPlan`, and later `ResultPlan` | Planner must not infer scalar behavior from datatype or intent | same scalar fixture plus Phase 0C policy tests | -| Python scalar argument conversion dispatches from completed policy | `x2py/codegen/bindings/c_to_python.py::CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER`; `_convert_python_scalar_value_argument` | `ArgumentTransferPlan.python_action = PythonBarrierAction.SCALAR_VALUE` | Isolated binding handler keeps the audited scalar-value conversion behavior and records the binding-to-bridge value handoff | same scalar fixture | -| Bridge scalar address-projected argument dispatches from completed policy | `x2py/codegen/bridges/fortran_to_c.py::FortranToCBridgeGenerator._NATIVE_BARRIER_DISPATCHER`; `_convert_native_call_local_address_argument` | `ArgumentTransferPlan.native_action = NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS` and bridge ABI call-local address slot | Isolated bridge handler passes the address of call-local scalar storage into the native call without storage/writeback policy inference | same scalar fixture | -| Scalar native call uses completed `@native_call([Addr(Arg(...))])` order | `FortranToCBridgeGenerator._visit_FunctionDef` after `_convert_function_arguments(...)` and native projection lowering | `NativeCallPlan` with deterministic native slots copied from completed scalar native-call slot policy | Validator rejects duplicate, missing, or reordered slots not justified by completed policy | same scalar fixture; hidden/native-only projections remain later lanes | -| Scalar result is returned to Python | legacy bridge result conversion and `CPythonBindingGenerator` result wrapping paths | Phase 2 `ResultPlan` and Python result projection action | Result creation consumes a produced native result; Phase 1 must not fake this as input support | same scalar fixture; Phase 2 expansion owns details | -| Generate bridge and binding source/header artifacts | `x2py/codegen/binding_pipeline.py::BindingPipeline.generate` and `BindingPipeline.write`; `FCodePrinter`; `CPythonCodePrinter` | Phase 0E/1A isolated module/header assembly and printers | New route emits complete Fortran bridge, C binding source, C header, additional imports, and runtime requirements before compilation | expected artifact names asserted by the scalar fixture | -| Compile and link the importable extension | `x2py/compiling/python_wrapper.py::create_shared_library` | Shared generated-wrapper artifact handoff reused by both routes | Compilation/link orchestration stays shared; no new route-specific build policy in emitters | existing wrapper build assertions | - -#### Decorator and feature audit reconciliation - -The live wrapper suite currently covers direct scalar calls, `@bind(...)`, -`@external`, `@hold_gil`, `@native_call` argument and result projections, -typed hidden literals, `@raises(...)`, optional and presence-token behavior, -strings, ordinary arrays, native array handles/descriptors, module variables, -derived types, snapshots, constructors, methods, properties, overloads, -generic dispatch, visibility/naming policy, callbacks, multiple-source builds, -semantic-`.pyi` replay, native bundles, and full BLAS/LAPACK real-library -corpora. The migration matrix above reconciles those features to the broad -lanes in the existing phase order. - -No accepted decorator or native projection is considered migrated by this -audit. Rows remain `legacy` unless their complete generation unit has passed -the required dual-route parity evidence. The full BLAS/LAPACK corpus remains -`deferred-real-library`; general native-bundle tests stay active and legacy -because they cover shared build mechanics independently of the full real -library corpus. - -#### Maintained first-lane wrapper-plan contract - -For the selected scalar baseline, the planned records must expose these phases -without backend policy inference: - -- Python surface: one `FunctionPlan` per public scalar function, lower-case - Python name, ordered Python scalar value arguments, and one scalar result. -- Binding handoff: each scalar argument has one `ArgumentTransferPlan` using - `PythonBarrierAction.SCALAR_VALUE`, the existing scalar value conversion - behavior, and one symbolic value handoff to the bridge. -- Bridge ABI: each transfer consumes that value handoff through the completed - scalar native action. The selected scalar fixture uses - `NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS` because every argument is - address-projected with `Addr(Arg(...))`. -- Native call: each scalar function has completed native-call slots copied from - `@native_call([Addr(Arg(...))])`, including native position, Python argument - position, Python/native names, value kind, and the native barrier action. - Implicit declared-order scalar calls may use the same policy shape, but they - are not required for the selected baseline evidence. -- Result projection: scalar native results are represented only in Phase 2 - records; Phase 1 planning may record that the selected baseline is blocked - by scalar result support but must not emit an incomplete runtime wrapper. -- Cleanup and writeback: scalar value inputs have no writeback, destructor, or - ownership transfer phase. Backend-local error cleanup remains private to the - binding/bridge implementation methods and is not a plan policy decision. -- Node construction and printing: Phase 0E may reuse or rewrite only the - dependency-closed legacy node, header/source assembly, printer, and helper - behavior required by the scalar baseline. It must not import, alias, - subclass, or adapt legacy codegen model classes in `x2py.wrapper_codegen`. -- Legacy entrypoint: the independent baseline entrypoint is the existing - `semantic_ir_to_codegen_ast()` route into `BindingPipeline` and - `create_shared_library()`. Phase 0A records it only; it does not modify - legacy lowering, nodes, generators, printers, or compilation. - -### Phase 0B Isolated Package Contract - -Phase 0B introduces infrastructure only. It does not add a route selector, -planner, backend nodes, emitters, printers, compilation, or production -wrapper-plan entrypoint. - -The implemented package boundary is: - -- `x2py.wrapper_codegen` is an isolated package. It may depend on stable Python - infrastructure and shared semantic/pipeline value objects when needed later, - but it must not import `x2py.codegen`. -- Legacy `x2py.codegen` must not import `x2py.wrapper_codegen`. -- A source module that imports both route families must live under - `x2py.pipeline`, because route selection and shared orchestration belong - there. No current production module imports both. -- The package currently exports only its independent `ClassVisitor` protocol - and unsupported-node error. Production wrapper builds do not import the - package. - -The independent visitor contract is intentionally smaller than the legacy -utility visitor: - -- dispatch is through `visit(node, ...)`; -- handler names are deterministic `_` lookups over the - node class MRO; -- the default prefix is `_visit`, with an instance-level override for renderer - or emitter protocols; -- missing support raises `UnsupportedWrapperCodegenNodeError` with the visitor - type, node type, and prefix. - -The blocking `x2py.wrapper_codegen.checks` package checker enforces Phase 0B -static contracts before any emitter handlers exist: - -- wrapper-codegen production modules must not import `x2py.codegen`; -- production module-level functions are rejected so generation behavior stays - on owning classes; -- production `Analyzer`, `Emitter`, `Planner`, `Renderer`, and `Validator` - classes must inherit `ClassVisitor`; -- each function/method must stay within the wrapper-codegen complexity, - statement-count, and nesting limits; -- class-level `*_REGISTRY`, `*_DISPATCHER`, and `*_HANDLERS` mappings that - name handler methods must point at methods on the same class, including - nested secondary dispatch dictionaries; -- registered handlers and handler-like methods may not call source printers - directly through `doprint(...)` or `write(...)`. - -The pipeline-owned generated-wrapper handoff is -`x2py.pipeline.wrapper_artifacts.GeneratedWrapperArtifacts`. It records only -generated wrapper source files, generated headers, the generated module name, -and runtime-support requirement keys. Native source objects, prebuilt native -artifacts, libraries, include directories, library directories, link order, and -compile/link execution remain owned by the existing build plan and compiler -orchestration. - -No new runtime helper API is introduced in Phase 0B. Later lanes may share an -existing runtime helper only when the behavior is unchanged. If the -wrapper-plan route needs different runtime behavior, that helper must have a -separately named generated caller and cleanup contract recorded in the relevant -lane before use. - -CPython reference-counting conventions, new/borrowed/stolen-reference rules, -`Py_INCREF`/`Py_DECREF`, and partial-failure cleanup remain binding-emitter -implementation mechanics. They are absent from plan models, rendered plans, -and cross-backend validation. - -### Phase 0C Scalar Policy Completion Contract - -Phase 0C is a semantic policy phase only. It adds no wrapper-plan records, -planner, backend nodes, emitters, printers, compilation path, or route -selection. - -The selected existing scalar semantic fixture is -`tests/wrapper/fortran/scalars/contracts/fmath/__init__.pyi`. Its current -contract surface is: - -- public module functions only; no methods, classes, overloads, module - variables, arrays, strings, hidden outputs, optional arguments, result - projections, build bundles, or runtime helper selection; -- every callable is marked `@external`; -- every callable has a `@bind(...)` native symbol and lower-case Python - function name; -- every Python argument is a primitive scalar value, completed by policy as - `PythonBarrierAction.SCALAR_VALUE`; -- every native argument slot is `Addr(Arg(i))`, completed by policy as an - address-projected scalar call-local value with - `NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`; -- every callable has one scalar value result, completed by policy as - `CodegenAction.DIRECT_VALUE`, `PythonBarrierAction.NONE`, and - `NativeBarrierAction.NONE`; -- scalar inputs have no writeback, release, or public setter phase; backend - cleanup of call-local temporaries remains local to the eventual selected - handler. - -Post-IR policy completion must attach one typed -`ScalarWrapperFunctionPolicy` to each semantic function. A supported scalar -record contains: - -- stable owner path, Python name, native symbol, external flag, and bind target; -- one `ScalarWrapperArgumentPolicy` per declared Python argument, including - owner path, Python position, native position, Python/native names, - semantic scalar type name, rank, optional flag, completed ownership decision, - completed codegen action, completed Python barrier action, completed native - barrier action, storage mode, boundary storage mode, projection flag, and - Python visibility; -- one `ScalarWrapperResultPolicy` for scalar results, including owner path, - semantic scalar type name, rank, completed ownership decision, completed - codegen action, completed Python barrier action, completed native barrier - action, storage mode, and boundary storage mode; -- one `ScalarWrapperNativeCallSlotPolicy` per native slot, including native - position, source kind (`implicit` or `projection`), Python position, - Python/native names, projection value kind, owner path, completed native - barrier action, and completed codegen action; -- empty writeback, cleanup, and release action tuples for the selected scalar - fixture. - -Unsupported functions still receive a typed scalar-wrapper policy record with -`supported=False` and stable blocker text. Missing policy or blocked policy -must fail through the semantic accessor before planning starts. Later planner -code may consume the typed record, but it must not inspect bridge, binding, or -backend-local implementation state to recover any field listed above. - -### Phase 0D Hierarchical Plan-Core Contract - -Phase 0D converts completed first-lane scalar policy into route-neutral wrapper -plans only. It adds no backend nodes, source emitters, source printers, -compilation path, route selector, fallback route, or production wrapper-plan -entrypoint. - -The plan-core package owns these records: - -- `ModulePlan`: one generation-unit plan with the module owner path, tuple of - function plans, and the completed action-handler registry used by validation - and rendering. -- `FunctionPlan`: one semantic function owner with Python/native names, - external/bind metadata, tuple of argument transfer plans, one optional result - plan, one bridge ABI plan, available symbolic roles, and explicit writeback, - cleanup, and release lifecycle tuples. -- `ArgumentTransferPlan`: the single cross-boundary transfer record for one - argument. It contains the completed `PythonBarrierAction`, one binding - handoff, one bridge ABI slot, the completed `NativeBarrierAction`, and one - native-call slot reference. Binding and bridge subplans must not be split into - separate argument policies. -- `BridgeAbiPlan` and `BridgeAbiSlotPlan`: deterministic bridge ABI positions - and symbolic roles copied from completed scalar policy. -- `NativeCallSlotPlan`: deterministic native-call slot references copied from - completed scalar native-call slot policy. No native slot may be invented from - datatype, `intent`, or backend-local state. -- `ResultPlan` and `LifecycleActionPlan`: symbolic result/writeback/cleanup - consumers. The selected scalar baseline has a scalar result and empty - lifecycle tuples. -- `HandlerRegistryPlan`: primary Python-barrier handlers, secondary - native-barrier handlers, and result handlers keyed by existing completed - `PythonBarrierAction`, `NativeBarrierAction`, and `CodegenAction` values. The - plan core must not define duplicate action enums. - -`WrapperPlanner(ClassVisitor)` owns deterministic structural wiring: - -- `_visit_SemanticModule` returns one `ModulePlan` and recursively visits only - module functions after the support analyzer confirms the whole module is in - the completed first scalar lane. -- `_visit_SemanticFunction` consumes - `completed_scalar_wrapper_policy(function)` and returns one `FunctionPlan`. - Missing or blocked scalar-wrapper policy is an error before planning. -- `_visit_ScalarWrapperArgumentPolicy` and - `_visit_ScalarWrapperResultPolicy` copy only completed policy values into - transfer/result records and attach handler names from the class-owned - registries. - -`WrapperPlanValidator(ClassVisitor)` owns pre-emission invariants: - -- unknown primary Python action handlers, unknown secondary native action - handlers, and unknown result handlers; -- missing bridge ABI slots or missing native-call slots on argument transfers; -- inconsistent handoff, bridge-slot, and native-slot symbolic roles; -- duplicate symbolic roles within one function; -- results, writebacks, cleanups, or releases consuming unavailable symbolic - values. - -`WrapperPlanSupportAnalyzer(ClassVisitor)` reports whole-generation-unit -eligibility with stable owner paths and reasons. It does not select routes. - -`WrapperPlanRenderer(ClassVisitor)` renders deterministic maintainer text with -owner paths, completed policy action values, handler names, symbolic handoffs, -bridge ABI slots, native slots, and lifecycle order. It must not render backend -nodes, C/Fortran source, CPython reference-counting mechanics, or printer -output. - -### Phase 0E Minimal Scalar Backend Foundation Contract - -Phase 0E introduces isolated backend syntax, naming, source assembly, and -printer infrastructure only. It does not add scalar plan emitters, binding or -bridge handlers, runtime helper implementations, compilation, route selection, -or generated artifact parity checks. - -The legacy inventory for the first scalar backend foundation is: - -| Backend concern | Legacy Python source owner | Consumed behavior | Isolated Phase 0E choice | -| --- | --- | --- | --- | -| Generated symbol naming | `x2py/codegen/scope.py::Scope.get_new_name` | deterministic unique names and collision avoidance | rewrite as `NameAllocator`; no semantic scopes, categories, or lookup ownership | -| Module/function contexts | `x2py/codegen/scope.py::Scope` plus binding/bridge generator-local state | per-module and per-function local names | rewrite as explicit `ModuleEmissionContext` and `FunctionEmissionContext` | -| Generic node structure | `x2py/codegen/models/core.py::Module`, `FunctionDef`, `FunctionDefArgument`, `Return`, `Import`, `Declare` | module imports, function signatures, parameters, declarations, statements, returns | rewrite minimal frozen C/Fortran node dataclasses with only printer-consumed fields | -| Scalar datatype/literal spelling | `x2py/codegen/models/datatypes.py` scalar `Numpy*Type` classes and `convert_to_literal` | C/Fortran scalar type spellings and literal text | rewrite as explicit `BackendScalarType` and expression/literal text nodes | -| CPython/NumPy include/API concepts | `x2py/codegen/bindings/cpython_api.py`, `numpy_cpython_api.py`, and `CPythonCodePrinter` | `Python.h`, `numpy/arrayobject.h`, `PyObject*`, `PyArg_ParseTupleAndKeywords`, scalar conversion helper names, `PyMethodDef`, `PyModuleDef`, `import_array` | represent only include/API references as isolated node data; Phase 1 handlers decide which primitives to emit | -| C and header printing | `x2py/codegen/printers/ccode.py::CCodePrinter` and `cpythoncode.py::CPythonCodePrinter` | include lines, header guards, prototypes, function signatures, declarations, expression statements, returns | rewrite `CSourcePrinter` over isolated C nodes only | -| Fortran bridge printing | `x2py/codegen/printers/fcode.py::FCodePrinter` | module/use/contains, `bind(c, name=...)`, scalar declarations, assignments, calls | rewrite `FortranSourcePrinter` over isolated Fortran nodes only | -| Source assembly orchestration | `x2py/codegen/binding_pipeline.py::BindingPipeline.generate/write` | create complete C source, C header, and Fortran module objects, then print modules | rewrite `BackendSourceAssembly.rendered_sources()` for in-memory source strings only | - -Every Phase 0E node field is consumed by a Phase 0E printer, source assembly, -or context test. Phase 1 scalar emitters will be the first production -consumers of these nodes; until then tests construct small representative -modules directly. This phase intentionally keeps CPython reference counting, -argument parsing, scalar conversion helper bodies, bridge call bodies, runtime -support installation, file writes, compiler/linker calls, and route eligibility -outside the implementation. - -Structural guarantees for this phase: - -- no `x2py.wrapper_codegen` backend module imports `x2py.codegen`; -- C and Fortran source printers accept only isolated backend nodes and do not - import or inspect wrapper-plan models; -- generated-source assertions stay focused on naming, declarations, - signatures, header guards, includes, `bind(c)` spelling, and - module-to-printer orchestration rather than full runtime wrapper snapshots. - -### Intermediate Test Contract - -Add intermediate tests only where they protect a stable boundary or a failure -that the existing compiled wrapper tests cannot isolate. - -- Test policy completeness for each newly migrated owner: missing or blocked - policy fails before planning. -- Test policy-to-plan projection with an existing semantic fixture. Assert the - owner hierarchy, completed action keys, call/ABI order, handoffs, and one - concise maintainer rendering for the lane; do not snapshot every plan field. -- Test validator failures with small directly constructed plans. Use - table-driven cases for invariant categories such as missing/duplicate slots, - incompatible producer/consumer handoffs, unavailable results/writebacks, and - unsupported completed actions. Do not add one test per implementation branch. -- Test whole-module support and route selection with existing generation units: - one fully supported unit and representative units blocked by later lanes. - Prove that no unit mixes routes and no new-route failure silently retries - legacy generation. -- Test the isolated visitor, name allocator, dependency boundary, registry - completeness, and complexity checker directly because they are standalone - contracts not observable through wrapper runtime behavior. -- Do not require one direct unit test per emitter handler or helper. Registry - checks prove that the rendered handler exists; the reused compiled wrapper - tests prove the selected chain's behavior. Add a direct emitter test only for - a failure or mechanical contract that cannot be reached through an existing - wrapper fixture. -- Do not add full generated C/Fortran source snapshots. Keep or add focused - source assertions only when exact source structure is the observable ABI or - build contract and runtime behavior cannot prove it. -- Add focused node/printer tests only for nontrivial mechanical behavior such as - precedence, escaping, declaration syntax, or header/source partitioning. - Do not create one test per node or copied printer method. - -The governing rule is one intermediate test per stable contract or failure -category, not one test per class, method, node, or branch. - ## Incremental Protocol For each lane: @@ -1385,15 +614,15 @@ For each lane: completion; do not start its planner while semantic decisions remain scattered or implicit. 5. Implement the lane's hierarchical plan records, planner visitors, - action registries, ABI/handoff specs, validator, renderer, and support-report - coverage. + ABI/handoff specs, generator-owned structural checks, directly named backend + lowering methods, source-printer support, and support-report coverage. 6. Implement the minimum dependency-closed backend slice in `x2py.wrapper_codegen`: copy small suitable pieces, rewrite oversized legacy classes as minimal equivalents, and add only the intermediate tests required by the contract above. -7. Generate the plan from policy-completed semantic IR, dispatch binding and - bridge handlers into isolated node fragments, assemble complete backend - modules, and print complete internal artifacts. +7. Generate the plan from policy-completed semantic IR, invoke the directly + named binding and bridge lowering methods, assemble complete backend modules, + and print complete internal artifacts. 8. Compare the new generated artifacts with the retained legacy artifacts and explain every material difference before compilation. 9. Compile the internal artifacts before changing production route selection. @@ -1443,356 +672,109 @@ expanded sub-lanes individually, then close the phase only after all live cases in its audited support matrix are either migrated or explicitly removed from the product contract. -## Required Execution Order - -The checklist order is mandatory. Do not wire the new route into production -build selection while its scalar artifacts exist only as models or uncompiled -source. Complete each subphase and its focused evidence before starting the -next one: - -```text -0A tests/wrapper inventory and current-behavior baseline - -> 0B isolated package and dependency boundary - -> 0C complete scalar policy - -> 0D hierarchical wrapper-plan core and validation - -> 0E minimal scalar backend nodes, names, and printers - -> 1A internal scalar plan emission - -> 1B compiled dual-route parity - -> 1C production route selection - -> later semantic lanes in numbered order - -> 11 cross-cutting tests/wrapper completion - -> 12 cutover and legacy removal -``` - -Within later lanes, follow the same dependency order: identify the existing -`tests/wrapper` coverage, audit current behavior, expand the lane checklist, -complete policy, add plan records/planner/validation, implement only the -required backend behavior, emit and print nodes internally, compile and compare -the same existing tests through both routes, update the migration matrix, and -only then widen production route eligibility. - -### Session Continuation Protocol - -Progress is evidence-driven, not calendar-driven. Do not use a time target to -skip a prerequisite, weaken parity, reduce validation, invent a smaller test -fixture, or mark partially completed work as complete. - -The minimal user prompt for a new implementation session is: - -```text -Continue implementing the wrapper-plan migration checklist. -``` - -That prompt means the agent must follow this resume procedure before editing: - -1. Read this checklist and the repository instructions, then inspect the dirty - worktree, recent relevant commits, current checklist state, and wrapper-test - migration matrix. Work with existing user changes; do not reset them. -2. Audit the first unchecked dependency-closed group whose prerequisites are - genuinely complete. Reconcile stale checkbox state against live code and - tests before choosing work. -3. If the next item is a broad phase or lacks exact policy fields, legacy source - paths, plan records, handlers, invariants, and existing test coverage, expand - it in this file before implementation. -4. Use the legacy replay procedure for the selected existing generation unit. - Start from its passing test, retained generated artifacts, and traced Python - implementation rather than designing from memory. -5. Implement one coherent group through its required intermediate checks. Do - not stop after adding models or copied code when the group's next required - validation can be completed in the same session. -6. Run the focused existing `tests/wrapper` nodes, required intermediate - contract tests, wrapper-codegen checker, static-analysis commands, and other - verification required by `AGENTS.md` for the files changed. -7. Update checkboxes and migration-matrix rows only for behavior proven by the - required evidence. A legacy test passing does not prove the new route; a row - becomes `wrapper-plan` only when route diagnostics and parity requirements - prove it. -8. End the session with the exact completed group, changed pipeline stages, - legacy paths reused, tests and command results, remaining unsupported paths, - and the next dependency-ordered unchecked item. - -Do not search for a shortcut around an unmet gate. Missing policy goes back to -post-IR completion, unsupported mechanics remain on the explicit legacy route, -and failed parity is investigated against retained legacy artifacts. If the new -route starts recreating most of legacy codegen without materially improving -policy traceability, plan readability, or handler size, stop for an explicit -value review rather than continuing automatically. - -## Phase 0 — Foundation Before Production Routing - -### Phase 0A — Wrapper Test Inventory And Current Scalar Baseline - -- [x] Enumerate every Python test node under `tests/wrapper` and add the - migration matrix required above. Do not begin implementation with untracked - wrapper tests. -- [x] Classify each test as non-generating or map it to its complete wrapper - generation unit and all semantic lanes needed before that unit can use the - wrapper-plan route. -- [x] Select the first existing generation unit whose coverage best matches the - initial scalar lane. Record every additional feature in that unit that delays - atomic route eligibility; do not replace it with a new narrower fixture. -- [x] Inventory the current end-to-end wrapper behavior and implementation paths - for the first scalar lane, including lowering branches, binding/bridge helper - methods, CPython/NumPy API primitives, source printers, generated artifacts, - build integration, and focused runtime fixtures. -- [x] Create a maintained baseline matrix mapping each first-lane current code - path and observable behavior to its proposed plan action, handler, required - backend behavior, and parity evidence. Do not define an action from a - hypothetical implementation. -- [x] Audit every decorator, native-call projection kind, implicit call - behavior, and generated module/class feature accepted by the live semantic - model; reconcile the coverage matrix with that audit. -- [x] Confirm the legacy generator's independent entrypoint and baseline tests; - do not modify legacy lowering, nodes, generators, or printers in this phase. -- [x] Update the maintained wrapper-plan contract with the audited Python - surface, binding handoff, bridge ABI, native call, result, cleanup, writeback, - node-construction, and printing phases. - -### Phase 0B — Isolated Package Boundary - -- [x] Create the `x2py.wrapper_codegen` package skeleton without connecting it - to production build selection. -- [x] Define and enforce the package boundary: `x2py.wrapper_codegen` cannot - import `x2py.codegen`, legacy `x2py.codegen` cannot import - `x2py.wrapper_codegen`, and only pipeline orchestration may eventually import - both route entrypoints. -- [x] Add dependency tests for that boundary before isolated backend code is - introduced. -- [x] Implement and test the minimal independent `ClassVisitor` used throughout - the package, including deterministic MRO lookup, configurable method prefixes, - and explicit unsupported-node failure. -- [x] Add structural checks requiring visitor-based traversal and rejecting - undeclared module-level production functions in `x2py.wrapper_codegen`. -- [x] Add the blocking wrapper-codegen complexity/traceability checker before - emitter handlers are introduced. Cover Radon complexity, statement count, - nesting depth, registry completeness, secondary-dispatch completeness, and - forbidden printer calls from handlers. -- [x] Add focused tests for the checker, including one failure fixture for each - enforced limit and registry/dependency rule. -- [x] Define the pipeline-owned generated-wrapper artifact result shared by both - routes without duplicating native object/library/link-plan ownership. -- [x] Keep runtime helper APIs shared only when their behavior is unchanged. Add - separately named new-route helpers when different behavior is required, and - record their generated callers and cleanup contract. -- [x] Document that CPython reference counting and API ownership conventions are - binding-emitter-local mechanics and are absent from plan models, rendered - plans, and cross-backend plan validation. - -### Phase 0C — Complete Scalar Policy - -- [x] Audit the selected existing scalar generation unit and list every - module/function/argument/result/decorator/native-projection decision the - planner would otherwise need to infer. -- [x] Define or complete typed post-IR policy records for the owners needed by - the first scalar lane. Do not use free-form metadata or planner defaults as a - substitute for a completed policy field. -- [x] Move any remaining scalar action, call/ABI order, ownership, lifecycle, - projection, writeback, or cleanup decisions into - `complete_semantic_policies(...)` before implementing the planner. -- [x] Verify the policy-completed module contains every datatype fact and - completed policy value needed to reproduce the audited legacy behavior - without reading bridge, binding, or backend-local state. -- [x] Add only the intermediate policy tests required by the contract above, - reusing the selected existing semantic fixture and covering missing/blocked - policy failure before planning. -- [x] Do not add plan models, backend nodes, emitters, printers, compilation, or - production route selection in this phase. - -### Phase 0D — Hierarchical Wrapper Plan Core - -- [x] Define the first frozen plan data classes and tuple collections. -- [x] Implement `WrapperPlanner(ClassVisitor)` with explicit `_visit_` - methods that each return one plan record, recursively visit only that owner's - children, and perform only deterministic structural wiring. Keep each method - within the strict planner complexity gate. -- [x] Define one `ArgumentTransferPlan` containing the existing completed Python - action, one binding-to-bridge handoff, bridge ABI slot, completed native - action, and native-call slot. Do not create separate binding and bridge - subplans for one argument. -- [x] Define `BridgeAbiPlan`, native-call refs, handoff specs, primary/secondary - handler registries, and validation errors around existing completed - `PythonBarrierAction` and `NativeBarrierAction` values. Do not add duplicate - plan action enums for behavior already represented by completed policy. -- [x] Add a plan validator that catches inconsistent transfer handoffs, - missing bridge/native-call slots, unknown primary or secondary handlers, - duplicate symbolic roles, and writebacks/results that consume unavailable - values. -- [x] Define the whole-generation-unit support report, including stable - owner-path reasons for unsupported elements, without changing production - route selection yet. -- [x] Implement class-owned `WrapperPlanSupportAnalyzer`, - `WrapperPlanValidator`, and `WrapperPlanRenderer` visitor APIs; do not add - equivalent module-level functions. -- [x] Add deterministic plan rendering that includes symbolic owner paths, - completed policy values, dispatch handler names, handoffs, bridge ABI slots, - native slots, and lifecycle order without backend nodes or CPython-specific - mechanics. -- [x] Verify the planner fails on missing/incomplete policy rather than deriving - defaults, and the validator produces owner-path diagnostics before node - emission for policy/plan inconsistency. -- [x] Add one policy-to-plan projection/rendering test for the selected existing - scalar fixture and table-driven validator tests by invariant category. Do not - add one test per planner method or plan field. -- [x] Do not add backend nodes, emitters, printers, compilation, or production - route selection in this phase. - -### Phase 0E — Minimal Scalar Backend Foundation - -- [x] Inventory the minimum dependency-closed set of scalar C/Fortran nodes, - datatype/literal behavior, CPython and NumPy API primitives, naming behavior, - helper concepts, and printer cases required for Phase 1. Record each legacy - source path and consumed field/method before implementation. -- [x] Implement a minimal `NameAllocator` and module/function emission contexts; - do not copy legacy `Scope` or its semantic lookup/categories. -- [x] For every required node/API/helper class, choose explicitly between a - small unchanged copy and a rewritten minimal class. Each new field and method - must have a current Phase 1 emitter or printer consumer. -- [x] Do not import, alias, subclass, or adapt legacy model classes. Preserve - required behavior through the isolated implementation and later compiled - parity evidence. -- [x] Add focused tests only for nontrivial naming/node/printer mechanics that - the selected existing wrapper fixture cannot isolate. Do not add exhaustive - node tests or full generated-source snapshots. -- [x] Reproduce the legacy module/header assembly and - `module -> doprint(module)` orchestration needed to create complete - `c_module` and `fortran_module` objects before writing source. -- [x] Implement only the C/Fortran printer cases needed by the isolated scalar - nodes. Verify structurally that the printers consume only isolated nodes and - cannot import or inspect wrapper-plan models. -- [x] Do not add scalar emitters, compilation, or production route selection in - this phase. - -## Phase 1 — Scalar Function Inputs - -Scope: free functions with scalar numeric/logical arguments that are -Python-visible inputs. Scalar call-target decorators and scalar native-call -argument projections are included as separate checklist items; unsupported -projection kinds keep the whole module on the legacy route. - -### Phase 1A — Internal Plan Emission - -- [ ] Generate plans for scalar value arguments such as `f(x: Float64)`. -- [ ] Populate one end-to-end `ArgumentTransferPlan` per scalar argument rather - than constructing separate binding and bridge plan objects. -- [ ] Represent Python argument position and `@native_call` native argument order - explicitly, including reordered arguments. -- [ ] Represent implicit native order, `@bind(...)`, `@external`, and - `@hold_gil` explicitly; none may be inferred or ignored by the emitters. -- [ ] Represent `Arg(...)`, `Addr(Arg(...))`, and typed numeric/logical hidden - literals as native argument sources with exact native positions. -- [ ] Reject duplicate, missing, or out-of-range Python/native positions and - bridge/native-call slots during plan validation. -- [ ] Register the isolated `_convert_python_scalar_value_argument` binding - handler for `SCALAR_VALUE` and isolated native handlers for `PASS_VALUE`, - `PASS_CALL_LOCAL_ADDRESS`, or other scalar actions already supported by - completed policy. -- [ ] Reuse audited legacy method-body snippets where they remain simple; - otherwise write smaller direct handlers that reproduce the baseline behavior - using the isolated nodes and explicit contexts. -- [ ] If scalar-family mechanics make a handler difficult to follow, introduce - a secondary logical/integer/real/complex dispatcher; keep precision as data - unless it changes emitted APIs, checks, declarations, or control flow. -- [ ] Keep every scalar emitter handler/helper within the strict complexity, - statement-count, and nesting limits; expose any secondary handler chain in - plan rendering and registry checks. -- [ ] Validate the transfer handoff, bridge ABI slot, and native-call slot as one - chain before either emitter runs. -- [ ] Make each scalar binding/bridge handler return isolated node fragments; - assemble complete C and Fortran module nodes outside individual handlers. -- [ ] Construct complete `c_module` and `fortran_module` objects, then pass them - to isolated `CPythonCodePrinter.doprint(...)` and `FCodePrinter.doprint(...)` - implementations to produce the scalar binding source/header and bridge - source. -- [ ] Assemble module initialization and generated-source requirements with the - printed files into the complete new-route artifact result. - -### Phase 1B — Internal Compilation And Parity - -- [ ] Select existing `tests/wrapper` nodes whose complete generation units are - now covered; do not introduce a new scalar source fixture for parity. -- [ ] Provide test-only orchestration that sends each selected existing module - directly through legacy and wrapper-plan routes without a public - compatibility option or production selector change. -- [ ] Compile and import new-route scalar artifacts through the shared compiler - and linker before making any production module eligible. -- [ ] Reuse the selected tests' existing build helpers and assertions for both - routes; do not duplicate their behavioral assertions in a new test file. -- [ ] Compare both routes for Python calls/results, native argument order, - pass-by-value/address behavior, conversion failures, exception state, - backend-local cleanup, generated artifact requirements, compilation, import, - and runtime behavior. -- [ ] Resolve every unexplained parity difference or document a separately - approved behavior correction before proceeding. -- [ ] Change the selected tests' migration-matrix status from `legacy` to - `dual-route` only after both compiled routes pass. - -### Phase 1C — Production Route Integration - -- [ ] Add one shared pipeline selector used by source-driven and semantic-`.pyi`- - driven builds after policy completion and before any legacy - `semantic_ir_to_codegen_ast()` call. -- [ ] Select the wrapper-plan route only for complete generation units whose - recursively inspected elements are all covered by completed scalar-input - actions, validators, emitters, printers, and parity evidence. -- [ ] Route a generation unit containing any unsupported element entirely - through the existing path. -- [ ] Add route-selector tests proving one module cannot mix plan and legacy - functions, bindings, bridges, nodes, or printers. -- [ ] Prove plan construction, validation, emission, printing, compilation, or - linking failures do not silently retry the legacy route. -- [ ] Keep explicit internal selection of the legacy route available for - rollback and dual-route tests. -- [ ] Move every newly eligible existing test from `dual-route` to - `wrapper-plan` in the migration matrix after production selection passes. - -## Phase 2 — Scalar Results And Hidden Outputs - -Scope: scalar direct returns, hidden scalar outputs, and scalar projected -results. - -- [ ] Audit and record the legacy result, hidden-output, result-packaging, - `@raises`, cleanup, printer, and runtime paths that define this lane's - baseline. -- [ ] Add or rewrite only the additional result variables, CPython creation - calls, C/Fortran statements, and printer cases required by this lane, with - baseline tests. -- [ ] Represent direct native return, hidden output, identity output, and - projected result lanes in `ResultPlan`. -- [ ] Add bridge actions for scalar result assignment and hidden scalar output - storage without relying on `is_alias`. -- [ ] Add binding actions for scalar Python result creation. -- [ ] Validate that every Python result consumes a native result or writeback - that the bridge produces. -- [ ] Cover `@raises` status/message outputs so runtime-status validation is - represented in the plan, not rediscovered in binding. -- [ ] Emit and print complete result-capable C/Fortran modules internally, then - compile and compare both routes for values, status/error paths, cleanup, ABI, - and artifact requirements. -- [ ] Widen whole-module route eligibility to scalar results only after that - parity evidence passes. +## Dependency-Ordered Checklist + +### Foundation and semantic authority + +- [x] Establish the isolated `x2py.wrapper_codegen` package boundary and + visitor infrastructure. +- [x] Complete the first primitive lane in general wrapper policy before + planning, including native-call order, result projection, ownership, and + lifecycle facts. +- [x] Build editable `ModulePlan`, `FunctionPlan`, transfer, result, ABI, + native-slot, and lifecycle records from completed wrapper policy. +- [x] Refactor each cross-boundary owner into explicit binding and bridge child + plans, including module/function/result/lifecycle scope as well as arguments. +- [x] Remove plan-owned method names and handler registries; add typed + datatype-family facts required by direct lowering. +- [x] Keep structural plan validation private to `WrapperCodeGenerator`, with + no planner-time validation or standalone validator class, and verify every + listed invariant after direct plan edits. + +### Direct generator boundary + +- [x] Change `WrapperCodeGenerator.generate` to consume only `ModulePlan`, + freeze it, validate it, validate lowering support, recursively generate + backend nodes, print them, and return artifacts directly. +- [x] Implement recursive `CBindingGenerator` synthesis of complete C modules, + headers, and functions from plan nodes. +- [x] Implement recursive `FortranBridgeGenerator` synthesis of complete + Fortran modules and functions from plan nodes. +- [x] Replace plan-selected method names with directly named backend lowering + methods selected by the visible `_lower_{subject}_{action.value}` rule. +- [x] Ensure backend node printers, artifact construction, and build + consumption retain their distinct freezing boundaries. + +### Scalar parity and route evidence + +- [x] Replay the existing scalar source and semantic-`.pyi` baseline through + the direct generator and compare generated artifacts and runtime behavior. +- [x] Update the staged walkthrough to use only plan editing and the public + generator boundary. +- [x] Retire superseded internal representations, their package exports, + orchestration, validation, documentation, and focused tests. +- [x] Run focused wrapper-codegen and pipeline tests; the walkthrough for both + supported entry choices where practical; `tests/wrapper` excluding LAPACK; + documentation checks; `git diff --check`; the required static-analysis suite; + and `tools/check_wrapper_codegen_complexity.py`. ## Phase 3 — Scalar Inout, Optional, And Descriptor-Like Scalars Scope: scalar copy-in/copy-out, optional arguments, present-but-null descriptor values, and scalar allocatable/pointer descriptor boundaries. -- [ ] Audit and record the legacy copy-in/out, optional presence, nullable +Phase 3 legacy replay audit: + +- `foptional_fixed.f` uses one nullable value pointer at the Bind-C ABI. + Omission and explicit `None` both pass a null pointer because both mean that + the ordinary optional dummy is absent; a concrete scalar uses call-local + storage, and the bridge branches on `c_associated(...)` before calling the + native function with or without the optional keyword. +- The existing optional allocatable-scalar contract uses two independent ABI + pointers. The value pointer is null for explicit `None`, while the presence + pointer is non-null for both `None` and a concrete value. Omission leaves both + null. The bridge therefore distinguishes absent, present-unallocated, and + present-with-value states without inferring presence from the value pointer. +- Immutable scalar replacement uses copy-in storage, native mutation of that + storage, copy-out to a new Python scalar, and scope-owned stack cleanup. The + caller's original NumPy scalar remains unchanged. The audit found no existing + runtime wrapper test for this primitive-scalar `Returns["argument", T]` + contract, so `test_scalar_writeback_plan.py` is the recorded coverage-gap + fixture for this lane. +- The first legacy replay of that coverage gap exposed a duplicate declaration + of the mutable scalar result. The legacy bridge now promotes the copy-in + temporary to the Bind-C function result and removes it from the ordinary + local-declaration set. Both routes compile, and incompatible Python values + fail with `TypeError` before the native call. Stack temporaries and local + allocatable descriptors require no explicit release action; their procedure + scope owns cleanup on normal return. + +The Phase 3 plan records optional mode, nullable value and presence handoffs, +and four ordered scalar replacement phases: `copy_in`, `native_mutation`, +`copy_out`, and `cleanup`. Generator preflight requires the complete phase set, +an existing source handoff, the correct binding/bridge owner for each phase, +and a Python result target for copy-out. Forced whole-module route selection +accepts these completed lanes after the dual-route evidence below. Automatic +production selection still remains on the legacy route under the independent +GIL parity deferral recorded for Phase 2D. + +- [x] Audit and record the legacy copy-in/out, optional presence, nullable scalar descriptor, cleanup, and failure-path behavior for this lane. -- [ ] Add or rewrite only the additional optional/descriptor nodes, API +- [x] Add or rewrite only the additional optional/descriptor nodes, API primitives, local-state helpers, and printer cases required by this lane, with baseline tests. -- [ ] Represent copy-in, native mutation, copy-out, and cleanup as explicit +- [x] Represent copy-in, native mutation, copy-out, and cleanup as explicit writeback phases. -- [ ] Preserve the three-state optional rule: omitted argument, explicit `None`, +- [x] Preserve the three-state optional rule: omitted argument, explicit `None`, and present concrete value are distinct when the native ABI needs them. -- [ ] Represent scalar descriptor presence tokens and nullable value handoffs in +- [x] Represent scalar descriptor presence tokens and nullable value handoffs in the plan. -- [ ] Validate that a writeback consumes an existing binding/bridge handoff and +- [x] Validate that a writeback consumes an existing binding/bridge handoff and writes to a Python-visible target or result slot. -- [ ] Emit and print complete inout/optional/descriptor-capable modules +- [x] Emit and print complete inout/optional/descriptor-capable modules internally, then compile and compare both routes for all three presence states, mutation, writeback, cleanup, ABI, and failures. -- [ ] Widen whole-module route eligibility to this lane only after parity, and +- [x] Widen whole-module route eligibility to this lane only after parity, and complete it before moving arrays or handles to the plan path. ## Phase 4 — Scalar Module Variables @@ -1801,25 +783,188 @@ Scope: scalar module variables. Derived-type fields remain in Phases 8 and 9 because their wrapper instance, owner, and property lifecycle must already be represented before field access can use the plan route. -- [ ] Audit and record the legacy scalar module-variable getter, setter, +Phase 4 legacy replay audit: + +- `fmodule_vars_f90.f90` establishes the ordinary scalar state contract. Its + legacy bridge emits value-returning getters and value-argument setters; + binding accessors run with the GIL held, aliases route to the same native + storage, deletion fails, and contract initializers call the native setter at + import. A `parameter` is instead copied into the Python module dictionary, so + rebinding it is local to that module object and never mutates native storage. + This whole source remains legacy because it also owns `rgb_color` and derived + module objects from later phases. +- The scalar subset of `fscalar_descriptors_f90` establishes nullable + allocatable and pointer reads. The legacy bridge returns null for absent + storage or allocates and copies one detached scalar; the binding converts the + copy, frees it, and rejects descriptor replacement. Its whole source cannot + migrate in Phase 4 because it also contains nullable snapshot-result forms + from a later lane. Allocation failure is deliberately injected with + `X2PY_WRAPPER_FAIL_ALLOC` and preserves the legacy null/`None` surface. +- No existing generation unit contained only the already completed scalar + function lanes plus every Phase 4 getter, setter, constant, descriptor, + initialization, reload, and failure behavior. The bounded + `test_scalar_module_variable_plan.py` whole-module fixture records that + coverage gap; it contains no strings, arrays, classes, or later-phase owner. + +The plan keeps only completed typed facts: Python names, getter and setter +actions, initializer or constant value, datatype family, native name/module, +native assignment, descriptor kind, and handoff roles. Both backends invoke +directly named lowering methods with matching subject/action suffixes wherever +their behavior is shared; datatype and descriptor facts stay method inputs. +The generator validates the complete frozen plan before either backend emits +anything, including binding/bridge getter agreement and the rule that a Python +write-through setter must have a compatible bridge setter role. Forced +whole-module selection now accepts `scalar-module-variables`; automatic +production selection remains independently deferred by the Phase 2D GIL gate. + +- [x] Audit and record the legacy scalar module-variable getter, setter, rejected replacement, module initialization, and attribute-routing behavior. -- [ ] Add or rewrite only the additional module/type nodes, getter/setter API +- [x] Add or rewrite only the additional module/type nodes, getter/setter API primitives, initialization nodes, and printer cases required by this lane, with baseline tests. -- [ ] Represent getter behavior, setter exposure, native setter assignment, and +- [x] Represent getter behavior, setter exposure, native setter assignment, and rejected replacement behavior in module-variable plans. -- [ ] Add binding actions for Python attribute get/set around scalar values. -- [ ] Add bridge actions for scalar module-variable read/write. -- [ ] Validate getter/setter pair consistency: a Python setter cannot exist +- [x] Add binding actions for Python attribute get/set around scalar values. +- [x] Add bridge actions for scalar module-variable read/write. +- [x] Validate getter/setter pair consistency: a Python setter cannot exist without a compatible bridge setter handoff. -- [ ] Keep ordinary Python module-name rebinding semantics separate from native +- [x] Keep ordinary Python module-name rebinding semantics separate from native module-variable storage. -- [ ] Emit and print complete module-variable-capable modules internally, then +- [x] Emit and print complete module-variable-capable modules internally, then compile and compare both routes for get/set behavior, rejection paths, initialization, cleanup, ABI, and generated artifacts. -- [ ] Widen whole-module route eligibility to scalar module variables only after +- [x] Widen whole-module route eligibility to scalar module variables only after that parity evidence passes. +### Phase 3/4 whole-unit namespace correction + +The post-Phase 4 review found that scalar lowering itself matched the legacy +route, but the parity helper unwrapped a sole native child module before making +assertions. That hid a public-surface difference: the legacy route retained +Fortran modules as Python child namespaces while the plan route flattened their +members at the extension root. The support analyzer also accepted colliding +procedures from separate native modules and allowed the failure to reach the +Fortran compiler. + +This correction is part of the completed scalar foundation rather than a new +datatype lane: + +- [x] Add a concise `NamespacePlan` beneath `ModulePlan`; place functions and + variables in namespace nodes instead of flattening them into the module. +- [x] Complete Python export paths in post-IR export policy, including + namespace-local keyword and collision fixes, then mechanically group plan + owners by those paths without reconstructing namespace policy in either + backend. +- [x] Generate root, child, and nested Python modules while keeping native + module imports and generated bridge symbols unambiguous. +- [x] Support ordinary scalar subroutines with no projected result through the + existing native call plus Python `None` result path. +- [x] Reject duplicate Python exports and generated symbols before either + backend emits source. +- [x] Use one visible lowering naming rule in both backends: + `_lower_argument_`, `_lower_result_`, + `_lower_writeback_`, `_lower_module_getter_`, + and `_lower_module_setter_`. Do not store method-name strings + in the plan or hide these selections in backend dictionaries. +- [x] Remove scalar prefixes from general wrapper concepts, including the + function, argument, result, native-slot, lifecycle, node, and printer policy + surfaces. Retain scalar naming only for permanently scalar-specific ABI type + facts and actions. +- [x] Update the staged walkthrough to print the namespace tree, typed actions, + and the directly corresponding binding and bridge method names. +- [x] Compile both routes from the existing complete + `contract_mixed_module_external.f90`, `contract_import_graph.f90`, + `contract_multi_module.f90`, `contract_standalone_only.f90`, and + `contract_same_name.f90` fixtures; compare the real extension root and child + namespaces without `_sole_native_module` normalization. + +## Phase 2D — Native Call Runtime Envelope — Next + +This is the next dependency-closed migration lane. Complete it before Phase 5 +so the already proven scalar generation units can move from temporary +`dual-route` evidence to production `wrapper-plan` routing instead of adding +more datatype lanes behind the same runtime gate. + +Scope: the binding-owned runtime envelope around an otherwise completed native +call. This phase includes default GIL release, explicit `@hold_gil`, and native +status/message projection through `@raises(...)`. Status projection is included +because the existing `fruntime_policy_f90` generation unit tests it together +with both GIL modes and whole-generation-unit routing cannot split that module. + +Excluded from this phase: + +- strings, arrays, descriptors, derived types, and callbacks, which remain in + their datatype or callback phases; +- callback re-entry and callback exception/abort behavior, which remain in + Phase 10; +- OpenMP array execution and Makefile-specific behavior, which remain blocked + by the array and cross-cutting build lanes; +- general Python exception translation that is not selected by a completed + native status policy. + +The existing legacy/runtime oracle is +`tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py` in both its +source-driven and semantic-`.pyi`-driven forms. It proves that an ordinary +native pause releases the GIL, `@hold_gil` keeps it held, successful status +returns produce the declared Python result, failing status returns raise the +selected exception with the native message, and emitted C places +`Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` only around eligible native +calls. `test_recursive_native_runtime_calls` is the scalar regression unit to +run after the call envelope works. Do not use the OpenMP or callback fixtures +as the first parity unit. + +The plan and generators must follow these boundaries: + +- Post-IR policy completion owns `hold_gil` and the complete native status + error decision, including status source, message source, success value, and + Python exception kind. The planner only projects those completed facts. +- Keep the runtime facts concise and function-owned. Extend the existing + binding-facing function plan rather than adding a second function plan or a + backend dispatcher table. The bridge continues to lower native call slots + and result storage mechanically; it does not decide GIL or Python exception + policy. +- Argument parsing and conversion, Python result construction, status/message + conversion, exception creation, writeback, and Python-owned cleanup always + run with the GIL held. For the default policy, release the GIL immediately + before the bridge call and reacquire it immediately after that call. For + `hold_gil=True`, emit no release region. +- Perform status evaluation and raise the selected Python exception only after + the GIL has been reacquired. Validate before emission that every status and + message projection names an existing native result slot with a compatible + completed handoff. +- Use directly named lowering methods that follow the existing visible naming + rule. Do not infer runtime policy from result types, function names, emitted + locals, or the presence of status-like native arguments. + +- [ ] Audit and record the exact legacy GIL release/hold region, status/message + projection, exception construction, result suppression, cleanup, and failure + behavior from both existing runtime-policy tests. +- [ ] Complete the native status error decision in post-IR policy before + planning; retain the already completed `hold_gil` fact as its single source + of truth. +- [ ] Extend the concise function plan with only the binding-facing runtime + facts needed for GIL and status-error lowering, and validate all referenced + native result slots before either backend emits source. +- [ ] Add direct binding lowering for the released-call and held-call envelopes + plus post-call status projection. Keep the bridge call and result-slot + lowering on their existing paths. +- [ ] Replay both source and semantic-`.pyi` forms of + `test_runtime_policies.py` through legacy and wrapper-plan routes using the + same concurrency, exception, artifact, and generated-C assertions. +- [ ] Run `test_recursive_native_runtime_calls` through the wrapper-plan route + as the scalar recursion regression; leave OpenMP and callbacks in their + later lanes. +- [ ] After dual-route parity passes, remove the blanket Phase 2D production + deferral. Let whole-generation-unit support select `wrapper-plan` only for + units whose feature lanes are complete; do not add fallback or per-function + mixed routing. +- [ ] Move the eligible scalar matrix rows from `dual-route` or `legacy` to + `wrapper-plan`, update the live route counts, and prove their default builds + no longer invoke `semantic_ir_to_codegen_ast()`. +- [ ] Finish this phase only when the production `wrapper-plan` count is + nonzero and the already completed scalar baseline no longer depends on the + legacy route outside deliberate rollback diagnostics. + ## Phase 5 — Strings Scope: scalar character values, fixed-length strings, deferred-length strings, @@ -2041,3 +1186,17 @@ datatype lane. - [ ] The final report includes the changed-stage breakdown required by `AGENTS.md` and names every test file added or updated with the behavior it covers. +## Session Continuation Protocol + +The stable continuation prompt is: + +```text +Continue implementing the wrapper-plan migration checklist. +``` + +On continuation: read this checklist and `AGENTS.md`; inspect the dirty +worktree; choose the first unchecked dependency-closed item; replay the +existing passing wrapper test before extending a lane; implement code and tests +together; run required verification; and check items only from live evidence. +Do not reset unrelated user changes, infer missing policy in lowering, or use a +new fallback after direct plan generation starts. diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index ee494ac8f..3de9f996f 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -847,7 +847,8 @@ unallocated or unassociated state. Hidden scalar or derived-type `Return(...)` outputs are different: the wrapper requests them with native temporary storage, so they are present and returned on every call. -Runtime tests: [`test_optional_arguments.py`](../../../tests/wrapper/fortran/function_calls/test_optional_arguments.py). +Runtime tests: [`test_optional_arguments.py`](../../../tests/wrapper/fortran/function_calls/test_optional_arguments.py), +[`test_scalar_writeback_plan.py`](../../../tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py). frozen artifacts At the start of `WrapperCodeGenerator.generate(plan)`, the generator must: 1. recursively freeze that exact plan object; -2. run the complete structural validation on the final edited plan; and -3. reject inconsistent handoffs, ABI slots, native ordering, lifecycle roles, - or unsupported lowering combinations before it creates backend nodes. +2. run the complete binding/bridge plan-consistency validation on the final + edited plan; +3. ask each backend to preflight only its own implementation capability; and +4. recursively lower the validated plan into backend nodes before the printers + consume those nodes. Later mutation of the received plan raises `FrozenStageRecordError`. Backend nodes remain editable until their printer consumes them. Generated artifacts @@ -177,15 +179,38 @@ it. `WrapperCodeGenerator` owns the private structured validation methods and is the only validation consumer. There is no standalone validator class or public validation operation. +`WrapperCodeGenerator._validate_plan()` is the single plan-consistency gate. +It validates the complete binding/bridge graph after the editable plan has +been frozen and before either backend preflight or visitor runs. The gate stays +small by composing `_plan_diagnostics()` from typed private diagnostics for +namespaces, functions, arguments, results, lifecycle actions, and module +variable getter/setter action families. A cross-view invariant belongs to the +diagnostic for the lowest plan node that contains both views; for example, a +module-variable diagnostic validates Python setter exposure against its native +assignment and bridge setter role. + +`CBindingGenerator.require_supported()` and +`FortranBridgeGenerator.require_supported()` are later backend-local +capability preflights. They may reject a completed action, primitive type, +descriptor kind, or ABI combination that their own backend cannot implement, +but they do not establish whether binding and bridge views agree. Backend +visitors and `_lower_*` methods mechanically consume the decisions owned by +their view. Their exhaustive unmatched-action errors remain defensive +protection for direct backend use; the public generation path must report a +cross-view inconsistency from `_validate_plan()` first. No generator infers +consistency by reading the other backend's plan view. + Structural validation preserves these invariants: +- module getter actions and roles agree, and Python setter exposure agrees with + native assignment, bridge setter roles, descriptor kinds, and constant state; - binding producer and bridge consumer roles agree; - bridge ABI coverage, positions, and owner roles are complete; - native-call slot coverage, exact ordering, hidden literals, and hidden - results agree; + results agree, including hidden-result native and codegen actions; - direct and hidden result producer/consumer roles agree; - writeback, cleanup, and release actions use available source roles in their - declared order; + declared order, and advertised roles exactly match their plan producers; - positions and symbolic roles are neither duplicate nor missing; and - external and bind-target requirements are complete. @@ -252,20 +277,47 @@ policy; those remain in existing build/link orchestration. ## Direct Lowering Methods -Each backend maps a completed plan action to one directly named method. The -single visible rule is: +Each backend visitor dispatches plan nodes by class through +`_visit_`. A visitor method then calls a typed +`_lower_` helper for each completed action family owned by that +backend. The helper uses an explicit, exhaustive action match and calls one +concrete `_lower__` implementation method. For example: ```python -method_name = f"_lower_{subject}_{action.value}" +def _visit_ModuleVariablePlan(self, plan): + return ( + *self._lower_module_getter(plan), + *self._lower_module_setter(plan), + ) + +def _lower_module_getter(self, plan): + match plan.binding.getter_action: + case ModuleGetterAction.CONSTANT_VALUE: + return self._lower_module_getter_constant_value(plan) + case ModuleGetterAction.DIRECT_VALUE: + return self._lower_module_getter_direct_value(plan) + case ModuleGetterAction.NULLABLE_SNAPSHOT: + return self._lower_module_getter_nullable_snapshot(plan) + raise ValueError(...) ``` -For example, an argument whose completed optional mode is `required` uses -`_lower_argument_required`; a module getter action `nullable_snapshot` uses -`_lower_module_getter_nullable_snapshot`. `lowering_method_name(subject, -action)` exposes that exact selection for inspection. Generator entry verifies -that every selected method exists before lowering begins, and unsupported -actions fail explicitly. No dispatcher dictionary or method-name string is -stored in the plan. +The C binding dispatches only from binding-owned actions, and the Fortran +bridge dispatches only from bridge-owned actions. In particular, native module +setter generation consumes the completed bridge assignment action rather than +the Python setter-exposure action. Post-IR policy completion records +`AssignmentMode.NONE` when no native setter is exposed and +`AssignmentMode.VALUE_COPY` for supported scalar value write-through; bridge +lowering does not reconstruct that choice from the Python setter action. +Backend support checks retain genuine ABI and capability validation; action +dispatch itself raises explicitly for every unsupported value, including an +unsupported alias assignment. + +Do not synthesize implementation method names, use `getattr` to execute +lowering, retain a fallback behavior, or store dispatcher names in the plan. +Do not create extra getter or setter plan nodes solely to gain more +`_visit_` methods. Both visitor and lowering methods return backend +syntax nodes; printers remain the only layer that renders those nodes as source +text. Primitive dtype spelling and converter differences live in the intentionally scalar-specific `PrimitiveScalarTypeRegistry`; they do not duplicate control @@ -319,8 +371,8 @@ function.bridge.native_name = "SUB_R8" binding = CBindingGenerator() bridge = FortranBridgeGenerator() -print(binding.lowering_method_name("argument", function.arguments[0].binding.optional_mode)) -print(bridge.lowering_method_name("argument", function.arguments[0].bridge.optional_mode)) +print(function.arguments[0].binding.optional_mode) +print(function.arguments[0].bridge.optional_mode) artifacts = WrapperCodeGenerator( c_generator=binding, @@ -332,8 +384,9 @@ artifacts = WrapperCodeGenerator( ``` It does not expose standalone validation. Printed plan inspection uses the -actual namespace and owner records, and `lowering_method_name(subject, action)` -shows the exact directly named implementation method selected in each backend. +actual namespace, owner, and completed action records. The backend visitors +make the corresponding explicit action matches visible in their typed lowering +helpers. ## Required Evidence diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index 01ce49cd2..d3435c703 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -313,12 +313,15 @@ def test_scalar_module_variable_policy_completes_access_and_storage_before_plann assert policies["counter"].initializer == 3 assert policies["target_scale"].getter_action is ModuleGetterAction.DIRECT_VALUE assert policies["target_scale"].setter_action is SetterAction.WRITE_THROUGH + assert policies["target_scale"].native_assignment is AssignmentMode.VALUE_COPY assert policies["optional_scale"].getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT assert policies["optional_scale"].descriptor_kind == "allocatable" assert policies["optional_scale"].setter_action is SetterAction.REJECT_REPLACEMENT + assert policies["optional_scale"].native_assignment is AssignmentMode.NONE assert policies["selected_scale"].getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT assert policies["selected_scale"].descriptor_kind == "pointer" assert policies["selected_scale"].setter_action is SetterAction.REJECT_REPLACEMENT + assert policies["selected_scale"].native_assignment is AssignmentMode.NONE def test_wrapper_policy_records_primitive_hidden_literals(): diff --git a/tests/wrapper_codegen/test_phase0d_plan_core.py b/tests/wrapper_codegen/test_phase0d_plan_core.py index 098e07b84..bf8f084c2 100644 --- a/tests/wrapper_codegen/test_phase0d_plan_core.py +++ b/tests/wrapper_codegen/test_phase0d_plan_core.py @@ -38,6 +38,16 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... ) +def _hidden_result_plan(): + return _plan( + """ +@native_call([Int32(1), Arg(0), Bool(False), Return("result", 0)]) +def scale(x: Float64) -> Float64: ... +""", + module_name="hidden_values", + ) + + def _edit_first_function(plan, edit): root = plan.namespaces[0] functions = (edit(root.functions[0]), *root.functions[1:]) @@ -73,13 +83,7 @@ def test_planner_projects_one_shared_tree_with_explicit_backend_views(): def test_planner_records_hidden_literals_and_hidden_result_slots(): - plan = _plan( - """ -@native_call([Int32(1), Arg(0), Bool(False), Return("result", 0)]) -def scale(x: Float64) -> Float64: ... -""", - module_name="hidden_values", - ) + plan = _hidden_result_plan() function = plan.namespaces[0].functions[0] assert [(slot.source_kind, slot.literal_type, slot.literal_value) for slot in function.native_call_slots] == [ @@ -93,6 +97,58 @@ def scale(x: Float64) -> Float64: ... assert function.result.native_call_slot == function.native_call_slots[3] +def test_generator_rejects_hidden_result_native_action_disagreement(): + plan = _hidden_result_plan() + function = plan.namespaces[0].functions[0] + result = function.result + replacement = ( + NativeBarrierAction.PASS_VALUE + if result.bridge.native_action is not NativeBarrierAction.PASS_VALUE + else NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + ) + invalid = _edit_first_function( + plan, + lambda item: replace( + item, + result=replace(result, bridge=replace(result.bridge, native_action=replacement)), + ), + ) + + with pytest.raises(ValueError, match="inconsistent-result-native-action"): + WrapperCodeGenerator().generate(invalid) + + +def test_generator_rejects_hidden_result_slot_codegen_action_disagreement(): + plan = _hidden_result_plan() + function = plan.namespaces[0].functions[0] + result = function.result + edited_slot = replace(result.native_call_slot, codegen_action=CodegenAction.DIRECT_VALUE) + invalid = _edit_first_function( + plan, + lambda item: replace( + item, + result=replace(result, native_call_slot=edited_slot), + native_call_slots=tuple( + edited_slot if slot.native_position == edited_slot.native_position else slot + for slot in item.native_call_slots + ), + ), + ) + + with pytest.raises(ValueError, match="inconsistent-result-slot-codegen-action"): + WrapperCodeGenerator().generate(invalid) + + +def test_generator_rejects_advertised_role_without_a_plan_producer(): + invalid = _edit_first_function( + _scalar_plan(), + lambda function: replace(function, available_roles=(*function.available_roles, "invented:role")), + ) + + with pytest.raises(ValueError, match="inconsistent-available-roles"): + WrapperCodeGenerator().generate(invalid) + + def test_planner_groups_completed_exports_into_explicit_namespace_nodes(): module = parse_pyi_text( """ diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index 722c9c1ef..c2509f3b4 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -10,7 +10,7 @@ from tests._shared.ownership_policy_support import parse_pyi_text from x2py.semantics.ownership import CodegenAction from x2py.semantics.policy_completion import complete_semantic_policies -from x2py.semantics.wrapper_policy import OptionalMode +from x2py.semantics.wrapper_policy import WritebackPhase from x2py.stage_values import FrozenStageRecordError from x2py.wrapper_codegen import ( CBindingGenerator, @@ -70,15 +70,70 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... assert "result = SWAP_ARGS(y, x)" in fortran_source -def test_function_lowering_names_are_direct_and_identical_across_backends(): - for generator in (CBindingGenerator, FortranBridgeGenerator): - assert generator.lowering_method_name("argument", OptionalMode.REQUIRED) == "_lower_argument_required" - assert ( - generator.lowering_method_name("argument", OptionalMode.NULLABLE_VALUE) == "_lower_argument_nullable_value" - ) - assert generator.lowering_method_name("argument", OptionalMode.DESCRIPTOR) == "_lower_argument_descriptor" - assert generator.lowering_method_name("result", CodegenAction.DIRECT_VALUE) == "_lower_result_direct_value" - assert generator.lowering_method_name("result", CodegenAction.HIDDEN_OUTPUT) == "_lower_result_hidden_output" +@pytest.mark.parametrize( + ("source", "c_fragment", "fortran_fragment"), + [ + ( + "def required_value(x: Float64) -> Float64: ...", + "PyObject * x_obj;", + "result = native_required_value(x)", + ), + ( + "def optional_value(x: Int32 = ...) -> Int32: ...", + "PyObject * x_obj = Py_None;", + "if (c_associated(bound_x)) then", + ), + ( + """ +@native_call([Allocatable(Arg(0))]) +def descriptor_value(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... +""", + "PyObject * value_obj = NULL;", + "type(c_ptr), value :: bound_value_present", + ), + ( + """ +@native_call([Addr(Arg(0)), Return("result", 0)]) +def hidden_value(x: Float64) -> Float64: ... +""", + "void bind_c_hidden_value(double * x, double * result);", + "subroutine bind_c_hidden_value(x, result)", + ), + ], +) +def test_supported_function_actions_select_their_backend_behavior(source, c_fragment, fortran_fragment): + artifacts = WrapperCodeGenerator().generate(_plan(source, module_name="action_dispatch")) + + assert c_fragment in _rendered_source(artifacts, ".c") + assert fortran_fragment in _rendered_source(artifacts, ".f90") + + +@pytest.mark.parametrize( + "codegen_action", + (CodegenAction.COPY_IN_OUT, CodegenAction.IN_PLACE_ARGUMENT), +) +def test_supported_writeback_actions_select_scalar_result_behavior(codegen_action): + plan = _plan( + 'def bump(value: Annotated[Int32, Immutable]) -> Returns["value", Int32]: ...', + module_name="writeback_dispatch", + ) + function = plan.namespaces[0].functions[0] + actions = tuple( + replace(action, binding=replace(action.binding, codegen_action=codegen_action)) + if action.phase is WritebackPhase.COPY_OUT + else action + for action in function.writeback_actions + ) + root = plan.namespaces[0] + edited = replace( + plan, + namespaces=(replace(root, functions=(replace(function, writeback_actions=actions),)),), + ) + + c_source = _rendered_source(WrapperCodeGenerator().generate(edited), ".c") + + assert "bind_c_bump(&value);" in c_source + assert "PyObject * result_obj = Int32_to_PyLong(&value);" in c_source def test_direct_plan_edits_change_binding_and_bridge_generation_then_freeze_plan(): @@ -151,5 +206,5 @@ def scale(x: Float64) -> Float64: ... namespaces=(replace(root, functions=(replace(function, arguments=(invalid_argument,)),)),), ) - with pytest.raises(ValueError, match="Unsupported C lowering action"): + with pytest.raises(ValueError, match="Unsupported C argument optional mode"): WrapperCodeGenerator().generate(invalid) diff --git a/tests/wrapper_codegen/test_phase4_scalar_module_variables.py b/tests/wrapper_codegen/test_phase4_scalar_module_variables.py index e2abdbc1f..782baea74 100644 --- a/tests/wrapper_codegen/test_phase4_scalar_module_variables.py +++ b/tests/wrapper_codegen/test_phase4_scalar_module_variables.py @@ -3,6 +3,7 @@ from __future__ import annotations from dataclasses import replace +from unittest.mock import Mock import pytest @@ -36,6 +37,14 @@ def _source(artifacts, suffix: str) -> str: return next(item.text for item in artifacts.sources if item.path.name.endswith(suffix)) +def _replace_variable(plan, python_name: str, edit): + root = plan.namespaces[0] + variables = tuple( + edit(variable) if variable.binding.python_names == (python_name,) else variable for variable in root.variables + ) + return replace(plan, namespaces=(replace(root, variables=variables), *plan.namespaces[1:])) + + def test_module_variable_plan_contains_only_completed_dispatch_facts(): plan = _plan() variables = {variable.binding.python_names[0]: variable for variable in plan.namespaces[0].variables} @@ -48,23 +57,96 @@ def test_module_variable_plan_contains_only_completed_dispatch_facts(): assert variables["counter"].binding.setter_action is SetterAction.WRITE_THROUGH assert variables["counter"].bridge.native_assignment is AssignmentMode.VALUE_COPY assert variables["counter"].binding.initializer == 3 + assert variables["target_scale"].bridge.native_assignment is AssignmentMode.VALUE_COPY assert variables["optional_scale"].binding.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT assert variables["optional_scale"].bridge.descriptor_kind == "allocatable" assert variables["optional_scale"].binding.setter_action is SetterAction.REJECT_REPLACEMENT + assert variables["optional_scale"].bridge.native_assignment is AssignmentMode.NONE assert variables["selected_scale"].bridge.descriptor_kind == "pointer" + assert variables["selected_scale"].bridge.native_assignment is AssignmentMode.NONE -def test_module_variable_lowering_names_are_direct_and_identical_across_backends(): - for generator in (CBindingGenerator, FortranBridgeGenerator): - assert generator.lowering_method_name("module_getter", ModuleGetterAction.DIRECT_VALUE) == ( - "_lower_module_getter_direct_value" - ) - assert generator.lowering_method_name("module_getter", ModuleGetterAction.NULLABLE_SNAPSHOT) == ( - "_lower_module_getter_nullable_snapshot" - ) - assert generator.lowering_method_name("module_setter", SetterAction.WRITE_THROUGH) == ( - "_lower_module_setter_write_through" - ) +def test_module_variable_visitors_consume_their_backend_owned_actions(): + plan = _plan() + counter = next( + variable for variable in plan.namespaces[0].variables if variable.binding.python_names == ("counter",) + ) + split_actions = replace( + counter, + binding=replace( + counter.binding, + getter_action=ModuleGetterAction.CONSTANT_VALUE, + setter_action=SetterAction.OMIT, + ), + bridge=replace( + counter.bridge, + getter_action=ModuleGetterAction.DIRECT_VALUE, + native_assignment=AssignmentMode.VALUE_COPY, + ), + ) + + assert CBindingGenerator().visit(split_actions) == () + assert [procedure.name for procedure in FortranBridgeGenerator().visit(split_actions)] == [ + "bind_c_get_counter", + "bind_c_set_counter", + ] + + +def test_fortran_module_setter_rejects_unsupported_bridge_assignment(): + plan = _plan() + counter = next( + variable for variable in plan.namespaces[0].variables if variable.binding.python_names == ("counter",) + ) + invalid = replace(counter, bridge=replace(counter.bridge, native_assignment=AssignmentMode.ALIAS)) + + with pytest.raises(ValueError, match="Unsupported Fortran module setter assignment"): + FortranBridgeGenerator().visit(invalid) + + +@pytest.mark.parametrize( + ("python_name", "assignment"), + [ + ("counter", AssignmentMode.NONE), + ("counter", AssignmentMode.ALIAS), + ("optional_scale", AssignmentMode.VALUE_COPY), + ("optional_scale", AssignmentMode.ALIAS), + ("limit", AssignmentMode.VALUE_COPY), + ], +) +def test_module_setter_assignment_mismatch_fails_before_backend_preflight_or_lowering( + python_name, + assignment, +): + invalid = _replace_variable( + _plan(), + python_name, + lambda variable: replace( + variable, + bridge=replace(variable.bridge, native_assignment=assignment), + ), + ) + c_generator = Mock(spec=CBindingGenerator) + fortran_generator = Mock(spec=FortranBridgeGenerator) + c_printer = Mock() + fortran_printer = Mock() + generator = WrapperCodeGenerator( + c_generator=c_generator, + fortran_generator=fortran_generator, + c_printer=c_printer, + fortran_printer=fortran_printer, + ) + + with pytest.raises(ValueError, match="invalid-module-native-assignment") as error: + generator.generate(invalid) + + assert "Unsupported Fortran module setter assignment" not in str(error.value) + c_generator.require_supported.assert_not_called() + fortran_generator.require_supported.assert_not_called() + c_generator.visit.assert_not_called() + fortran_generator.visit.assert_not_called() + c_generator.requires_runtime_support.assert_not_called() + c_printer.doprint.assert_not_called() + fortran_printer.doprint.assert_not_called() def test_module_variable_generators_dispatch_get_set_and_rejection_from_plan(): @@ -90,6 +172,26 @@ def test_module_variable_generators_dispatch_get_set_and_rejection_from_plan(): assert "selected_scale = value" not in fortran_source +def test_module_variable_literal_families_select_their_c_spelling(): + module = parse_pyi_text( + """ +enabled: Bool = True +count: Int32 = 3 +scale: Float64 = 1.5 +phase: Complex128 = 1 + 2j +""", + module_name="literal_state", + ) + complete_semantic_policies(module) + + c_source = _source(WrapperCodeGenerator().generate(WrapperPlanner().build(module)), ".c") + + assert "bind_c_set_enabled(true);" in c_source + assert "bind_c_set_count(3);" in c_source + assert "bind_c_set_scale(1.5);" in c_source + assert "bind_c_set_phase((1.0 + 2.0 * I));" in c_source + + def test_generator_rejects_python_module_setter_without_bridge_handoff(): plan = _plan() counter = next( diff --git a/tools/wrapper_plan_staged_walkthrough.py b/tools/wrapper_plan_staged_walkthrough.py index 03d51c7f8..d1c2a6186 100644 --- a/tools/wrapper_plan_staged_walkthrough.py +++ b/tools/wrapper_plan_staged_walkthrough.py @@ -99,46 +99,24 @@ def calculate(x: Float64, y: Float64) -> Float64: ... print(f" Python name: {planned_function.binding.python_name}") print(f" native target: {planned_function.bridge.native_name}") for argument in planned_function.arguments: - binding_method = binding_generator.lowering_method_name( - "argument", - argument.binding.optional_mode, + print( + f" argument {argument.binding.python_name}: " + f"binding action={argument.binding.optional_mode!r}, " + f"bridge action={argument.bridge.optional_mode!r}" ) - bridge_method = bridge_generator.lowering_method_name( - "argument", - argument.bridge.optional_mode, - ) - print(f" argument {argument.binding.python_name}: binding={binding_method}, bridge={bridge_method}") - result_action = planned_function.result.bridge.codegen_action if planned_function.result is not None else "none" - bridge_method = bridge_generator.lowering_method_name("result", result_action) if planned_function.result is None: - print(f" result: binding=, bridge={bridge_method}") + print(" result: binding=, bridge=") else: - binding_method = binding_generator.lowering_method_name( - "result", - planned_function.result.binding.codegen_action, + print( + f" result: binding action={planned_function.result.binding.codegen_action!r}, " + f"bridge action={planned_function.result.bridge.codegen_action!r}" ) - print(f" result: binding={binding_method}, bridge={bridge_method}") for variable in item.variables: - binding_getter = binding_generator.lowering_method_name( - "module_getter", - variable.binding.getter_action, - ) - bridge_getter = bridge_generator.lowering_method_name( - "module_getter", - variable.bridge.getter_action, - ) - binding_setter = binding_generator.lowering_method_name( - "module_setter", - variable.binding.setter_action, - ) - bridge_setter = bridge_generator.lowering_method_name( - "module_setter", - variable.binding.setter_action, - ) print( f" variable {variable.binding.python_names}: " - f"binding getter={binding_getter}, setter={binding_setter}; " - f"bridge getter={bridge_getter}, setter={bridge_setter}" + f"binding getter={variable.binding.getter_action!r}, setter={variable.binding.setter_action!r}; " + f"bridge getter={variable.bridge.getter_action!r}, " + f"assignment={variable.bridge.native_assignment!r}" ) print("native slots:", function.native_call_slots) print("result plan:", function.result) diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index c53034c05..40db9feb7 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -204,7 +204,7 @@ def build_module_variable_policy( getter_action=_scalar_module_getter_action(getter, constant), getter=getter, setter_action=setter.setter_action if setter is not None else SetterAction.OMIT, - native_assignment=setter.assignment_mode if setter is not None else AssignmentMode.NONE, + native_assignment=_scalar_module_native_assignment(setter), setter=setter, descriptor_kind=descriptor_kind, initializer=( @@ -828,6 +828,15 @@ def _scalar_module_getter_action( return ModuleGetterAction.DIRECT_VALUE +def _scalar_module_native_assignment( + setter: OwnershipDecision | None, +) -> AssignmentMode: + """Project the completed native setter action for bridge lowering.""" + if setter is None or setter.setter_action is not SetterAction.WRITE_THROUGH: + return AssignmentMode.NONE + return setter.assignment_mode + + def _scalar_module_descriptor_kind(variable: models.SemanticVariable) -> str | None: metadata = variable.semantic_type.metadata if metadata.get("fortran_allocatable"): diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index 791f420f0..a05f0d20f 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -5,6 +5,7 @@ from dataclasses import dataclass from x2py.semantics.ownership import ( + CodegenAction, NativeBarrierAction, PythonBarrierAction, SetterAction, @@ -35,6 +36,7 @@ ) from x2py.wrapper_codegen.plan import ( ArgumentTransferPlan, + DatatypeFamily, FunctionPlan, LifecycleActionPlan, ModulePlan, @@ -65,31 +67,21 @@ class CBindingGenerator(ClassVisitor): """Recursively lower binding plan views directly into C syntax nodes.""" def require_supported(self, plan: ModulePlan) -> None: - """Reject actions without their directly named C lowering method.""" + """Reject unsupported C ABI actions and scalar types.""" for function in self._functions(plan): for argument in function.arguments: if argument.binding.python_action is not PythonBarrierAction.SCALAR_VALUE: raise ValueError( f"Unsupported C argument action for {argument.owner_path!r}: {argument.binding.python_action!r}" ) - self._require_lowering_method("argument", argument.binding.optional_mode, argument.owner_path) PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) if function.result is not None: - self._require_lowering_method( - "result", - function.result.binding.codegen_action, - function.result.owner_path, - ) PrimitiveScalarTypeRegistry.type_for(function.result.semantic_type_name) for action in function.writeback_actions: if action.phase is not WritebackPhase.COPY_OUT or action.binding is None: continue - self._require_lowering_method("writeback", action.binding.codegen_action, action.owner_path) PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) for variable in self._variables(plan): - self._require_lowering_method("module_getter", variable.binding.getter_action, variable.owner_path) - self._require_lowering_method("module_setter", variable.binding.setter_action, variable.owner_path) - self._require_lowering_method("module_literal", variable.datatype_family, variable.owner_path) PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: @@ -222,10 +214,23 @@ def _module_allocator_functions(self, required: bool) -> tuple[CFunction, ...]: ) def _visit_ModuleVariablePlan(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: - """Lower getter and setter actions through their visible naming rule.""" - getter = self._call_lowering("module_getter", plan.binding.getter_action, plan) - setter = self._call_lowering("module_setter", plan.binding.setter_action, plan) - return (*getter, *setter) + """Lower binding-owned getter and setter actions into C functions.""" + return ( + *self._lower_module_getter(plan), + *self._lower_module_setter(plan), + ) + + def _lower_module_getter(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: + """Dispatch one completed Python getter action explicitly.""" + action = plan.binding.getter_action + match action: + case ModuleGetterAction.CONSTANT_VALUE: + return self._lower_module_getter_constant_value(plan) + case ModuleGetterAction.DIRECT_VALUE: + return self._lower_module_getter_direct_value(plan) + case ModuleGetterAction.NULLABLE_SNAPSHOT: + return self._lower_module_getter_nullable_snapshot(plan) + raise ValueError(f"Unsupported C module getter action for {plan.owner_path!r}: {action!r}") def _lower_module_getter_constant_value(self, _plan: ModuleVariablePlan) -> tuple[CFunction, ...]: """Constants are materialized in the module dictionary at initialization.""" @@ -289,6 +294,18 @@ def _lower_module_getter_nullable_snapshot(self, plan: ModuleVariablePlan) -> tu ), ) + def _lower_module_setter(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: + """Dispatch one completed Python setter action explicitly.""" + action = plan.binding.setter_action + match action: + case SetterAction.WRITE_THROUGH: + return self._lower_module_setter_write_through(plan) + case SetterAction.REJECT_REPLACEMENT: + return self._lower_module_setter_reject_replacement(plan) + case SetterAction.OMIT: + return self._lower_module_setter_omit(plan) + raise ValueError(f"Unsupported C module setter action for {plan.owner_path!r}: {action!r}") + def _lower_module_setter_write_through(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: """Return a Python-to-native scalar write-through helper.""" scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) @@ -359,9 +376,25 @@ def _visit_ArgumentTransferPlan( plan: ArgumentTransferPlan, *, context: _CFunctionContext, - ) -> tuple[CDeclaration | CExpressionStatement, ...]: - """Lower one input through its directly named optional-mode method.""" - return self._call_lowering("argument", plan.binding.optional_mode, plan, context) + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Lower one input through its completed optional mode.""" + return self._lower_argument(plan, context) + + def _lower_argument( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Dispatch one completed binding optional mode explicitly.""" + mode = plan.binding.optional_mode + match mode: + case OptionalMode.REQUIRED: + return self._lower_argument_required(plan, context) + case OptionalMode.NULLABLE_VALUE: + return self._lower_argument_nullable_value(plan, context) + case OptionalMode.DESCRIPTOR: + return self._lower_argument_descriptor(plan, context) + raise ValueError(f"Unsupported C argument optional mode for {plan.owner_path!r}: {mode!r}") def _lower_argument_required( self, @@ -465,8 +498,23 @@ def _visit_ResultPlan( function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Lower one result through its directly named codegen-action method.""" - return self._call_lowering("result", plan.binding.codegen_action, plan, function, context) + """Lower one result through its completed binding action.""" + return self._lower_result(plan, function, context) + + def _lower_result( + self, + plan: ResultPlan, + function: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: + """Dispatch one completed binding result action explicitly.""" + action = plan.binding.codegen_action + match action: + case CodegenAction.DIRECT_VALUE: + return self._lower_result_direct_value(plan, function, context) + case CodegenAction.HIDDEN_OUTPUT: + return self._lower_result_hidden_output(plan, function, context) + raise ValueError(f"Unsupported C result action for {plan.owner_path!r}: {action!r}") def _lower_result_direct_value( self, @@ -547,7 +595,22 @@ def _writeback_nodes( if len(ordered) != 1: raise ValueError(f"{plan.owner_path!r} requires exactly one scalar writeback result") action = ordered[0] - return self._call_lowering("writeback", action.binding.codegen_action, plan, action, context) + return self._lower_writeback(plan, action, context) + + def _lower_writeback( + self, + plan: FunctionPlan, + action: LifecycleActionPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: + """Dispatch one completed binding writeback action explicitly.""" + codegen_action = action.binding.codegen_action + match codegen_action: + case CodegenAction.COPY_IN_OUT: + return self._lower_writeback_copy_in_out(plan, action, context) + case CodegenAction.IN_PLACE_ARGUMENT: + return self._lower_writeback_in_place_argument(plan, action, context) + raise ValueError(f"Unsupported C writeback action for {action.owner_path!r}: {codegen_action!r}") def _lower_writeback_copy_in_out( self, @@ -917,7 +980,18 @@ def _module_constant_nodes( return tuple(nodes) def _module_literal(self, plan: ModuleVariablePlan, value: object) -> str: - return self._call_lowering("module_literal", plan.datatype_family, value) + """Dispatch one completed datatype family to its C literal spelling.""" + family = plan.datatype_family + match family: + case DatatypeFamily.BOOL: + return self._lower_module_literal_bool(value) + case DatatypeFamily.INTEGER: + return self._lower_module_literal_integer(value) + case DatatypeFamily.REAL: + return self._lower_module_literal_real(value) + case DatatypeFamily.COMPLEX: + return self._lower_module_literal_complex(value) + raise ValueError(f"Unsupported C module literal family for {plan.owner_path!r}: {family!r}") def _lower_module_literal_bool(self, value: object) -> str: return "true" if value else "false" @@ -977,19 +1051,3 @@ def _namespace_object_name(self, plan: NamespacePlan) -> str: def _namespace_module_name(self, module: ModulePlan, namespace: NamespacePlan) -> str: return ".".join((module.binding.owner_path, *namespace.python_path)) - - @staticmethod - def lowering_method_name(subject: str, action: object) -> str: - """Return the exact implementation method selected by one plan action.""" - value = getattr(action, "value", action) - return f"_lower_{subject}_{value}" - - def _require_lowering_method(self, subject: str, action: object, owner_path: str) -> str: - method_name = self.lowering_method_name(subject, action) - if not callable(getattr(self, method_name, None)): - raise ValueError(f"Unsupported C lowering action for {owner_path!r}: {method_name}") - return method_name - - def _call_lowering(self, subject: str, action: object, *args): - method_name = self._require_lowering_method(subject, action, getattr(args[0], "owner_path", subject)) - return getattr(self, method_name)(*args) diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index bdb73d1aa..70b42c8af 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -2,7 +2,7 @@ from __future__ import annotations -from x2py.semantics.ownership import NativeBarrierAction +from x2py.semantics.ownership import AssignmentMode, CodegenAction, NativeBarrierAction from x2py.semantics.wrapper_policy import ModuleGetterAction, OptionalMode from x2py.wrapper_codegen.nodes import ( CodeExpression, @@ -33,7 +33,7 @@ class FortranBridgeGenerator(ClassVisitor): """Recursively lower bridge plan views directly into Fortran nodes.""" def require_supported(self, plan: ModulePlan) -> None: - """Reject actions without their directly named Fortran lowering method.""" + """Reject unsupported Fortran ABI actions and scalar types.""" for function in self._functions(plan): self._require_function_supported(function) for variable in self._variables(plan): @@ -46,8 +46,6 @@ def _require_function_supported(self, function: FunctionPlan) -> None: NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, NativeBarrierAction.PASS_STORAGE_ADDRESS, } - result_action = function.result.bridge.codegen_action if function.result is not None else "none" - self._require_lowering_method("result", result_action, function.owner_path) if self._has_optional_arguments(function) and any( slot.source_kind == "literal" for slot in function.native_call_slots ): @@ -58,13 +56,18 @@ def _require_function_supported(self, function: FunctionPlan) -> None: f"Unsupported Fortran argument action for {argument.owner_path!r}: " f"{argument.bridge.native_action!r}" ) - self._require_lowering_method("argument", argument.bridge.optional_mode, argument.owner_path) PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) def _require_variable_supported(self, variable: ModuleVariablePlan) -> None: """Reject unsupported actions in one planned module variable.""" - self._require_lowering_method("module_getter", variable.bridge.getter_action, variable.owner_path) - self._require_lowering_method("module_setter", variable.binding.setter_action, variable.owner_path) + if variable.bridge.native_assignment not in { + AssignmentMode.NONE, + AssignmentMode.VALUE_COPY, + }: + raise ValueError( + f"Unsupported Fortran module setter assignment for {variable.owner_path!r}: " + f"{variable.bridge.native_assignment!r}" + ) if ( variable.bridge.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT and variable.bridge.descriptor_kind not in {"allocatable", "pointer"} @@ -96,11 +99,7 @@ def _visit_NamespacePlan(self, plan: NamespacePlan) -> tuple[FortranFunction, .. def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: """Recursively assemble one complete bridge procedure.""" - result_parameters, result_name, result_type = self._call_lowering( - "result", - plan.result.bridge.codegen_action if plan.result is not None else "none", - plan, - ) + result_parameters, result_name, result_type = self._lower_result(plan) parameters = tuple( parameter for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) @@ -120,6 +119,21 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: is_subroutine=is_subroutine, ) + def _lower_result( + self, + plan: FunctionPlan, + ) -> tuple[tuple[FortranParameter, ...], str | None, str | None]: + """Dispatch one completed bridge result action explicitly.""" + if plan.result is None: + return self._lower_result_none(plan) + action = plan.result.bridge.codegen_action + match action: + case CodegenAction.DIRECT_VALUE: + return self._lower_result_direct_value(plan) + case CodegenAction.HIDDEN_OUTPUT: + return self._lower_result_hidden_output(plan) + raise ValueError(f"Unsupported Fortran result action for {plan.owner_path!r}: {action!r}") + def _lower_result_none( self, _plan: FunctionPlan, @@ -142,10 +156,23 @@ def _lower_result_hidden_output( return self._hidden_result_parameters(plan), None, None def _visit_ModuleVariablePlan(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: - """Lower getter and setter actions through their visible naming rule.""" - getter = self._call_lowering("module_getter", plan.bridge.getter_action, plan) - setter = self._call_lowering("module_setter", plan.binding.setter_action, plan) - return (*getter, *setter) + """Lower bridge-owned getter and setter actions into procedures.""" + return ( + *self._lower_module_getter(plan), + *self._lower_module_setter(plan), + ) + + def _lower_module_getter(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: + """Dispatch one completed bridge getter action explicitly.""" + action = plan.bridge.getter_action + match action: + case ModuleGetterAction.CONSTANT_VALUE: + return self._lower_module_getter_constant_value(plan) + case ModuleGetterAction.DIRECT_VALUE: + return self._lower_module_getter_direct_value(plan) + case ModuleGetterAction.NULLABLE_SNAPSHOT: + return self._lower_module_getter_nullable_snapshot(plan) + raise ValueError(f"Unsupported Fortran module getter action for {plan.owner_path!r}: {action!r}") def _lower_module_getter_constant_value(self, _plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: """Constants are materialized directly by the Python binding.""" @@ -216,7 +243,21 @@ def _lower_nullable_module_getter( ), ) - def _lower_module_setter_write_through(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: + def _lower_module_setter(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: + """Dispatch one completed native assignment action explicitly.""" + action = plan.bridge.native_assignment + match action: + case AssignmentMode.NONE: + return self._lower_module_setter_none(plan) + case AssignmentMode.VALUE_COPY: + return self._lower_module_setter_value_copy(plan) + raise ValueError(f"Unsupported Fortran module setter assignment for {plan.owner_path!r}: {action!r}") + + def _lower_module_setter_none(self, _plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: + """Return no native setter when the bridge assignment is omitted.""" + return () + + def _lower_module_setter_value_copy(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: """Return one value-copy native module assignment.""" scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) name = self._module_bridge_setter_name(plan) @@ -230,17 +271,21 @@ def _lower_module_setter_write_through(self, plan: ModuleVariablePlan) -> tuple[ ), ) - def _lower_module_setter_reject_replacement(self, _plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: - """Rejected replacement has no native setter procedure.""" - return () - - def _lower_module_setter_omit(self, _plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: - """Constants have no native setter procedure.""" - return () - def _visit_ArgumentTransferPlan(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Lower one argument through the completed optional-mode action.""" - return self._call_lowering("argument", plan.bridge.optional_mode, plan) + return self._lower_argument(plan) + + def _lower_argument(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: + """Dispatch one completed bridge optional mode explicitly.""" + mode = plan.bridge.optional_mode + match mode: + case OptionalMode.REQUIRED: + return self._lower_argument_required(plan) + case OptionalMode.NULLABLE_VALUE: + return self._lower_argument_nullable_value(plan) + case OptionalMode.DESCRIPTOR: + return self._lower_argument_descriptor(plan) + raise ValueError(f"Unsupported Fortran argument optional mode for {plan.owner_path!r}: {mode!r}") def _lower_argument_required(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: attributes = ("value",) if plan.bridge.native_action is NativeBarrierAction.PASS_VALUE else () @@ -542,22 +587,6 @@ def _functions(self, plan: ModulePlan) -> tuple[FunctionPlan, ...]: def _variables(self, plan: ModulePlan) -> tuple[ModuleVariablePlan, ...]: return tuple(variable for namespace in plan.namespaces for variable in namespace.variables) - @staticmethod - def lowering_method_name(subject: str, action: object) -> str: - """Return the exact implementation method selected by one plan action.""" - value = getattr(action, "value", action) - return f"_lower_{subject}_{value}" - - def _require_lowering_method(self, subject: str, action: object, owner_path: str) -> str: - method_name = self.lowering_method_name(subject, action) - if not callable(getattr(self, method_name, None)): - raise ValueError(f"Unsupported Fortran lowering action for {owner_path!r}: {method_name}") - return method_name - - def _call_lowering(self, subject: str, action: object, *args): - method_name = self._require_lowering_method(subject, action, getattr(args[0], "owner_path", subject)) - return getattr(self, method_name)(*args) - def _iso_symbol(self, semantic_type_name: str) -> str: symbols = { "Bool": "c_bool", diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index e30955ee3..ff818405e 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -70,13 +70,13 @@ def generate(self, plan: ModulePlan) -> RenderedGeneratedWrapperArtifacts: ) def _validate_plan(self, plan: ModulePlan) -> None: - """Reject structural inconsistencies in the final frozen plan.""" + """Reject complete-plan inconsistencies in the final frozen plan.""" diagnostics = self._plan_diagnostics(plan) if diagnostics: raise ValueError(self._diagnostic_summary(diagnostics)) def _plan_diagnostics(self, plan: ModulePlan) -> tuple[WrapperPlanDiagnostic, ...]: - """Return module and descendant diagnostics before lowering starts.""" + """Return binding/bridge graph diagnostics before backend preflight.""" diagnostics = [] if plan.binding.owner_path != plan.owner_path: diagnostics.append(self._diagnostic(plan.owner_path, "binding-module-owner", plan.binding.owner_path)) @@ -244,13 +244,21 @@ def _module_setter_diagnostics(self, plan: ModuleVariablePlan) -> tuple[WrapperP if role is None: diagnostics.append(self._diagnostic(plan.owner_path, "missing-module-setter-role", action.value)) return tuple(diagnostics) + diagnostics = [] + if assignment is not AssignmentMode.NONE: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-module-native-assignment", assignment)) if role is not None: - return (self._diagnostic(plan.owner_path, "setter-role-without-write-through", role),) - if action is SetterAction.REJECT_REPLACEMENT and plan.bridge.descriptor_kind is None: - return (self._diagnostic(plan.owner_path, "rejected-module-setter-without-descriptor", action.value),) + diagnostics.append(self._diagnostic(plan.owner_path, "setter-role-without-write-through", role)) + if action is SetterAction.REJECT_REPLACEMENT and plan.bridge.descriptor_kind not in { + "allocatable", + "pointer", + }: + diagnostics.append( + self._diagnostic(plan.owner_path, "rejected-module-setter-without-descriptor", action.value) + ) if action is SetterAction.OMIT and plan.binding.getter_action is not ModuleGetterAction.CONSTANT_VALUE: - return (self._diagnostic(plan.owner_path, "omitted-nonconstant-module-setter", action.value),) - return () + diagnostics.append(self._diagnostic(plan.owner_path, "omitted-nonconstant-module-setter", action.value)) + return tuple(diagnostics) def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Return ordering, handoff, result, and lifecycle diagnostics.""" @@ -268,6 +276,7 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost len(plan.native_call_slots), ), *self._duplicate_role_diagnostics(plan), + *self._available_role_diagnostics(plan), *self._function_output_diagnostics(plan), ] slots = {slot.native_position: slot for slot in plan.native_call_slots} @@ -308,6 +317,10 @@ def _argument_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-function-native-slot", plan.native_position) ) + if plan.native_call_slot.source_kind not in {"implicit", "projection"}: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-argument-native-slot", plan.native_call_slot.source_kind) + ) diagnostics.extend(self._optional_argument_diagnostics(plan)) return tuple(diagnostics) @@ -418,6 +431,14 @@ def _hidden_result_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-result-position", slot.result_position)) if slot.symbolic_role != plan.bridge.native_result_role: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-result-role", slot.symbolic_role)) + if slot.native_action is not plan.bridge.native_action: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-result-native-action", slot.native_action.value) + ) + if slot.codegen_action is not plan.bridge.codegen_action: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-result-slot-codegen-action", slot.codegen_action.value) + ) if function_slots.get(slot.native_position) != slot: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-function-result-slot", slot.native_position) @@ -427,6 +448,8 @@ def _hidden_result_diagnostics( def _native_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Return hidden literal and hidden result slot diagnostics.""" diagnostics = [] + if plan.source_kind not in {"implicit", "projection", "literal", "result"}: + diagnostics.append(self._diagnostic(plan.owner_path, "unknown-native-slot-source", plan.source_kind)) if plan.source_kind == "literal": if plan.literal_type is None: diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-type", plan.native_position)) @@ -577,6 +600,15 @@ def _duplicate_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDi if count > 1 ) + def _available_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require the advertised roles to match argument and result producers.""" + expected = [argument.binding.handoff_role for argument in plan.arguments] + if plan.result is not None: + expected.append(plan.result.bridge.native_result_role) + if Counter(plan.available_roles) != Counter(expected): + return (self._diagnostic(plan.owner_path, "inconsistent-available-roles", plan.available_roles),) + return () + def _diagnostic(self, owner_path: str, code: str, detail: object) -> WrapperPlanDiagnostic: return WrapperPlanDiagnostic(owner_path, code, str(detail)) From e2dc7f59cfab587fa3a57a0e0623be11ea8a7404 Mon Sep 17 00:00:00 2001 From: said Date: Mon, 13 Jul 2026 13:45:44 +0100 Subject: [PATCH 07/30] codex: complete Phase 2D runtime and scalar boundary migration - cut over completed GIL and status-error units to wrapper-plan routing - add scalar storage, raw-address, mapping, out, and inout parity coverage - enforce justified bridge copies and share editable native-slot records - align rendered-plan build integration and record remaining Phase 2F work --- .../wrapper-plan-migration-checklist.md | 244 +++++++--- docs/user/guide/fortran-wrapper.md | 27 +- tests/codegen/printers/_support.py | 1 + .../test_calls_and_policy_metadata.py | 3 +- .../test_rendered_wrapper_artifact_build.py | 2 + .../test_wrapper_plan_route_selection.py | 177 ++++++- tests/semantics/policy/test_wrapper_policy.py | 78 ++- tests/wrapper/CHECKLIST_COVERAGE.md | 3 +- .../test_external_procedures.py | 1 - .../layout_rules/test_wrapper_guide_layout.py | 1 + .../runtime_behavior/test_runtime_policies.py | 66 ++- tests/wrapper/fortran/scalars/README.md | 7 +- .../scalars/test_scalar_boundary_plan.py | 301 ++++++++++++ .../test_phase1a_wrapper_assembly.py | 3 +- .../test_phase1b_scalar_input_conversion.py | 5 +- .../test_phase2b_hidden_scalar_outputs.py | 9 +- .../test_phase2d_native_runtime_envelope.py | 163 +++++++ .../test_phase2e_scalar_boundaries.py | 128 +++++ ...st_phase3_scalar_presence_and_writeback.py | 4 +- x2py/codegen/bindings/c_to_python.py | 58 +-- x2py/pipeline/build.py | 117 ++++- x2py/semantics/ir2ast.py | 61 +-- x2py/semantics/models.py | 1 + x2py/semantics/policy_completion.py | 124 +++++ x2py/semantics/wrapper_policy.py | 385 ++++++++++++++- x2py/wrapper_codegen/__init__.py | 6 + x2py/wrapper_codegen/c/binding.py | 446 +++++++++++++++--- x2py/wrapper_codegen/fortran/bridge.py | 389 ++++++++++++--- x2py/wrapper_codegen/generator.py | 351 +++++++++++++- x2py/wrapper_codegen/nodes.py | 12 +- x2py/wrapper_codegen/plan.py | 26 + x2py/wrapper_codegen/planner.py | 156 ++++-- .../wrapper_codegen/primitive_scalar_types.py | 40 +- x2py/wrapper_codegen/source_printers.py | 10 + x2py/wrapper_codegen/support.py | 15 +- 35 files changed, 2994 insertions(+), 426 deletions(-) create mode 100644 tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py create mode 100644 tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py create mode 100644 tests/wrapper_codegen/test_phase2e_scalar_boundaries.py diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index e3261f55a..bf8baf559 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -88,6 +88,9 @@ deliberately distinct and directly editable: convention, native action, and Fortran value that the bridge consumes; - `native_call_slot` records the exact native-call position and source; +- an argument or hidden result's `native_call_slot` is the same mutable record + referenced from `FunctionPlan.native_call_slots`, not a copied record that a + maintainer must edit twice; - result and lifecycle records identify later producers, consumers, ordering, and responsibility through their own binding and bridge views. @@ -488,12 +491,26 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 0 | -| `dual-route` | 17 | -| `legacy` | 178 | +| `wrapper-plan` | 65 | +| `dual-route` | 0 | +| `legacy` | 134 | | `not-applicable` | 95 | | `deferred-real-library` | 2 | +#### Recorded Route Progression + +This history keeps phase movement visible instead of replacing the previous +snapshot with only the latest totals. Phase 2D moved all 17 dual-route nodes +and 44 legacy nodes to production plan routing, then added two parametrized +plan-route nodes. Phase 2E adds two scalar-only parity nodes; the original +mixed integration nodes retain their real array/string/object blockers. + +| Proven checkpoint | `wrapper-plan` | `dual-route` | `legacy` | `not-applicable` | `deferred-real-library` | Total | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Before Phase 2D | 0 | 17 | 178 | 95 | 2 | 292 | +| Phase 2D complete | 63 | 0 | 134 | 95 | 2 | 294 | +| Phase 2E scalar isolation | 65 | 0 | 134 | 95 | 2 | 296 | + Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating node must be `wrapper-plan`; `not-applicable` may remain only for tests that do @@ -515,25 +532,25 @@ already covered by the new generator. | `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | | `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::*` | non-generating: legacy model/printer/policy unit coverage | legacy model/printer mechanics; ordinary arrays; native handles/descriptors | `not-applicable` | | `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_cycles_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_missing_relative_contract_before_native_validation` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_both_routes[*]` | forced legacy/direct-plan parity | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `dual-route` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_both_routes[*]` | production plan route with deliberate legacy rollback comparison | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_rejects_colliding_wildcard_exports` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_matches_checked_in_fixture` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mixed_entry_exposes_externals_at_root_and_modules_as_children` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_leaf_can_be_the_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_variable_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mutable_module_variable_default_initializes_native_storage` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_one_entry_preserves_multiple_native_module_namespaces` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_one_entry_preserves_multiple_native_module_namespaces` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_requires_a_native_link_input` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; native handles/descriptors | `legacy` | @@ -543,17 +560,17 @@ already covered by the new generator. | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_projection_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_python_suffix_as_semantic_contract` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_extension_beside_source` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_extension_beside_source` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_source_build_result_records_structured_native_plan` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_custom_wrapper_flags` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_full_direct_build_commands` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_source_build_result_records_structured_native_plan` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_custom_wrapper_flags` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_full_direct_build_commands` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_empty_source_list` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_makefile_verbose_combination` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_missing_source` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | @@ -581,38 +598,38 @@ already covered by the new generator. | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; scalar module visibility and namespace projection | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | direct wrapper/build route | external symbols/native linkage; naming/visibility/dispatch | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bridge_uses_explicit_interface_and_no_module_use` | direct wrapper/build route | external symbols/native linkage | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_fixed_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_free_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | direct wrapper/build route | scalar external symbol; explicit bridge interface; renamed export | `wrapper-plan` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bridge_uses_explicit_interface_and_no_module_use` | direct wrapper/build route | scalar external symbol; explicit bridge interface | `wrapper-plan` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_fixed_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_free_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_generated_external_contracts_are_non_empty_root_fragments` | direct wrapper/build route | external symbols/native linkage | `legacy` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | direct wrapper/build route | external symbols/native linkage | `legacy` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_namespace_imported_module_rejects_external_marker_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | scalar external symbols; explicit bridge interfaces | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_package_entry_rejects_non_external_root_declaration_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | | `tests/wrapper/fortran/function_calls/test_function_call_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | deliberate legacy/direct-plan replay using the existing `foptional_fixed.f` generation unit and shared runtime/failure assertions | optional/presence; scalar inputs/results; build/artifact integration | `dual-route` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | deliberate legacy/direct-plan replay of one semantic-.pyi descriptor contract against the same native module | optional/presence; nullable scalar descriptor; build/artifact integration | `dual-route` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | production plan route with deliberate legacy rollback comparison | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | coverage-gap legacy/direct-plan replay for immutable scalar replacement return | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `dual-route` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | +| `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | scalar calls with internal common-block storage | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route[*]` | coverage-gap legacy/direct-plan replay for a whole module containing only Phase 1-4 scalar owners | scalar inputs/results; scalar module variables/state; build/artifact integration | `dual-route` | +| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route[*]` | production plan route with deliberate legacy rollback comparison | scalar inputs/results; scalar module variables/state; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | scalar multi-source build/link orchestration; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | scalar multi-source external symbols and link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_generated_contract_build_matches_source_runtime_and_link_order` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_modified_entry_preserves_modules_and_adds_documented_alias` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_pyi_out_writes_one_flat_combined_package` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | @@ -622,30 +639,31 @@ already covered by the new generator. | `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::*` | full BLAS/LAPACK wrapper generation unit | external symbols/native linkage; build/compile/link orchestration; broad wrapper corpus | `deferred-real-library` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_duplicate_native_definitions_report_linker_error` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_duplicate_native_definitions_report_linker_error` | direct wrapper/build route | scalar external symbols; linker failure propagation | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_imported_contracts_resolve_from_one_archive_or_shared_library[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage; build/compile/link orchestration | `legacy` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_incompatible_native_artifact_reports_linker_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_module_directory_reports_compile_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_symbol_reports_native_link_or_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_mixed_module_external_bundle_resolves_all_native_input_kinds` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_required_transitive_named_library_resolves_runtime_symbol` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_dependency_order_resolves_transitive_library` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_groups_resolve_cyclic_archive_dependencies` | direct wrapper/build route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_mixed_module_external_bundle_resolves_all_native_input_kinds` | direct wrapper/build route | scalar module/external symbols; ordered native inputs and library directories | `wrapper-plan` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_required_transitive_named_library_resolves_runtime_symbol` | direct wrapper/build route | scalar external symbol; transitive named library | `wrapper-plan` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_dependency_order_resolves_transitive_library` | direct wrapper/build route | scalar external symbol; ordered archive linkage | `wrapper-plan` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_groups_resolve_cyclic_archive_dependencies` | direct wrapper/build route | scalar external symbol; archive-group linkage | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_unavailable_dependent_shared_library_reports_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/runtime_behavior/test_openmp_runtime.py::*` | direct wrapper/build route | runtime policies/errors/GIL; build/compile/link orchestration | `legacy` | | `tests/wrapper/fortran/runtime_behavior/test_runtime_behavior_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::*` | direct wrapper/build route | runtime policies/errors/GIL | `legacy` | -| `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::*` | source and edited-.pyi production plan route with deliberate legacy rollback comparison | runtime policies/errors/GIL | `wrapper-plan` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | direct wrapper/build route | scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::*` | scalar-only copied native routines with deliberate legacy/direct-plan parity | primitive scalar kinds; value and `Addr(Arg(i))` inputs; hidden output; copy-in/copy-out; rank-zero storage; raw `Addr(T)`; native slot reordering | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | | `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `legacy` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes[*]` | deliberate legacy/direct-plan replay using the existing `fmath.f` and `fmath_f90.f90` generation units and shared runtime assertions | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `dual-route` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes[*]` | production plan route with deliberate legacy rollback comparison using the existing `fmath.f` and `fmath_f90.f90` generation units | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | direct wrapper/build route | strings | `legacy` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | @@ -931,7 +949,7 @@ datatype lane: `contract_same_name.f90` fixtures; compare the real extension root and child namespaces without `_sole_native_module` normalization. -## Phase 2D — Native Call Runtime Envelope — Next +## Phase 2D — Native Call Runtime Envelope This is the next dependency-closed migration lane. Complete it before Phase 5 so the already proven scalar generation units can move from temporary @@ -989,35 +1007,153 @@ The plan and generators must follow these boundaries: rule. Do not infer runtime policy from result types, function names, emitted locals, or the presence of status-like native arguments. -- [ ] Audit and record the exact legacy GIL release/hold region, status/message +The completed legacy audit found one binding-owned envelope in both oracle +builds. The legacy binding parsed and converted Python inputs with the GIL +held, emitted `Py_BEGIN_ALLOW_THREADS` immediately before the bridge call and +`Py_END_ALLOW_THREADS` immediately after it by default, and omitted both +macros for `@hold_gil`. Only after reacquiring the GIL did it convert hidden +status/message outputs, compare status with `success`, construct +`RuntimeError`, suppress those policy outputs from the declared Python result, +and decref converted result objects on both the failure and success paths. A +missing or incompatible status/message name was previously rediscovered from +raw decorator dictionaries and result datatypes in `ir2ast` and the legacy C +binding; Phase 2D moved that decision to typed post-IR completion and left the +legacy route as a dispatch consumer for rollback parity. + +The direct plan route now preserves that ordering with explicit released-call +and held-call lowering methods. Its fixed native message handoff is +bridge-owned null-terminated storage that the binding converts and frees after +the GIL is reacquired. Generated symbol spelling differs from the legacy +artifacts, but the same source and edited-`.pyi` concurrency, exception, +cleanup, and runtime assertions pass. Production cutover also reused the +existing rendered-artifact build path for inferred native module include +directories, native library directories, `.pyi` manifests, verbose timing, +and scalar external explicit interfaces; no legacy retry was added. + +- [x] Audit and record the exact legacy GIL release/hold region, status/message projection, exception construction, result suppression, cleanup, and failure behavior from both existing runtime-policy tests. -- [ ] Complete the native status error decision in post-IR policy before +- [x] Complete the native status error decision in post-IR policy before planning; retain the already completed `hold_gil` fact as its single source of truth. -- [ ] Extend the concise function plan with only the binding-facing runtime +- [x] Extend the concise function plan with only the binding-facing runtime facts needed for GIL and status-error lowering, and validate all referenced native result slots before either backend emits source. -- [ ] Add direct binding lowering for the released-call and held-call envelopes +- [x] Add direct binding lowering for the released-call and held-call envelopes plus post-call status projection. Keep the bridge call and result-slot lowering on their existing paths. -- [ ] Replay both source and semantic-`.pyi` forms of +- [x] Replay both source and semantic-`.pyi` forms of `test_runtime_policies.py` through legacy and wrapper-plan routes using the same concurrency, exception, artifact, and generated-C assertions. -- [ ] Run `test_recursive_native_runtime_calls` through the wrapper-plan route +- [x] Run `test_recursive_native_runtime_calls` through the wrapper-plan route as the scalar recursion regression; leave OpenMP and callbacks in their later lanes. -- [ ] After dual-route parity passes, remove the blanket Phase 2D production +- [x] After dual-route parity passes, remove the blanket Phase 2D production deferral. Let whole-generation-unit support select `wrapper-plan` only for units whose feature lanes are complete; do not add fallback or per-function mixed routing. -- [ ] Move the eligible scalar matrix rows from `dual-route` or `legacy` to +- [x] Move the eligible scalar matrix rows from `dual-route` or `legacy` to `wrapper-plan`, update the live route counts, and prove their default builds no longer invoke `semantic_ir_to_codegen_ast()`. -- [ ] Finish this phase only when the production `wrapper-plan` count is +- [x] Finish this phase only when the production `wrapper-plan` count is nonzero and the already completed scalar baseline no longer depends on the legacy route outside deliberate rollback diagnostics. +## Phase 2E — Scalar Boundary Completion and Test Isolation — Complete + +Complete the scalar public boundary before stopping this migration lane. This +phase does not begin strings or arrays. It separates scalar evidence from +mixed generation units so whole-unit routing cannot hide whether one scalar +policy is implemented. + +Scope: every supported primitive scalar kind; ordinary Python scalar values; +`Addr(Arg(i))` call-local address projection; projected scalar copy-in/copy-out; +caller-owned rank-zero NumPy storage spelled `T[()]`; caller-supplied integer +raw addresses spelled `Addr(T)`; and visible or hidden scalar `in`, `out`, and +`inout` behavior. For a mixed native fixture whose declarations cannot be +safely sliced, add a small distinctly named scalar-only native test routine +that preserves the policy decision under test. + +The boundary contract remains: + +- `T` accepts a Python/NumPy scalar value. When native code only reads it, the + wrapper converts into call-local storage. When native code writes through an + address projection and the contract projects `Returns["name", T]`, the + wrapper performs copy-in, native mutation, and copy-out to a replacement + Python scalar; the caller's immutable scalar object is not mutated. +- `T[()]` accepts a rank-zero NumPy array with exactly the declared dtype. The + wrapper validates caller storage and passes its data address; native `out` or + `inout` mutation remains visible in that same array and the Python call + returns `None` unless the contract declares another result. +- `Addr(T)` accepts an integer address such as `array.ctypes.data`. The wrapper + converts it to a raw pointer and forwards that same address without copying + or owning the pointee. Mutation is therefore observed through caller-owned + storage. +- `@native_call(...)` controls only native slot order and value/address/result + projection. It does not change which Python representation (`T`, `T[()]`, or + `Addr(T)`) the declared argument accepts. +- Use one necessary-copy rule. For interoperable scalar replacement, the + binding's converted C scalar is the copy-in storage and the bridge passes + that same storage directly to the native routine; after mutation the binding + converts it once to the Python replacement. `c_f_pointer` association for + `T[()]` or `Addr(T)` is not a data copy. A bridge-local data copy is allowed + only when the native representation actually changes, such as descriptor, + string-buffer, or ownership-snapshot construction. +- Enforce that rule with a completed `BridgeDataAction` on every argument, + result, and native-call output slot. `DIRECT_TRANSFER` reuses boundary + storage, `ASSOCIATE_VIEW` may create only a non-owning native view, + `COPY_REPRESENTATION` is the sole bridge data-copy permission and requires a + non-empty policy reason, and `BLOCKED` keeps the whole generation unit off + the plan route. A non-copying action carrying a copy reason is also invalid. + New array, string, or object support must complete this fact before route + eligibility is widened. + +Excluded: simultaneous multiple-result tuple assembly; rank-positive arrays; +strings including fixed status buffers except for already completed Phase 2D +status projection; derived types; callbacks; and any compatibility fallback to +the legacy generator. + +- [x] Record scalar-only tests separately from mixed array/string/derived + generation units in the route ledger; use copied minimal native routines + when fixture declarations are coupled. +- [x] Cover every primitive scalar kind exercised by the scalar runtime suite + through the direct registry and both binding/bridge generators. +- [x] Add direct named binding and bridge lowering for rank-zero numeric/logical + storage using the completed `SCALAR_STORAGE` and `PASS_STORAGE_ADDRESS` + decisions; validate dtype, rank zero, and writability before the native call. +- [x] Add direct named binding and bridge lowering for primitive raw addresses + using the completed `RAW_ADDRESS` and `PASS_RAW_ADDRESS` decisions; accept an + integer address and forward it without copy or ownership inference. +- [x] Prove isolated scalar input, hidden output, copy-in/copy-out `inout`, + caller-storage `out`/`inout`, and raw-address `out`/`inout` behavior through + compiled legacy/direct-plan parity where applicable. +- [x] Prove scalar copy-in/copy-out reuses one binding local and does not add a + redundant bridge-local value copy. +- [x] Prove plan validation rejects an unexplained bridge copy, a copy reason + on a non-copying path, and any still-blocked bridge data action. +- [x] Prove isolated `@native_call` argument mapping, including `Addr(Arg(i))` + and hidden `Return(...)` slots, without arrays determining route selection. +- [x] Move only proven scalar-only nodes to `wrapper-plan`, update collected + route counts, and leave the original mixed integration nodes on their real + datatype blockers. + +## Phase 2F — Multiple Scalar Result Assembly — Pending + +This is result aggregation, not another scalar boundary representation. The +first isolated oracle is the `with_scalar` policy from +`test_output_arguments.py`: one direct primitive scalar function return plus +one hidden primitive scalar output, assembled into a Python tuple in declared +result order. Keep it separate from arrays, strings, derived types, and native +handles before widening the plan route. + +- [ ] Add a scalar-only copied native routine and contract for a direct return + plus hidden scalar `Return(...)` slot. +- [ ] Represent every Python result as an explicit binding consumer while + preserving the bridge's direct-return and output-address ABI roles. +- [ ] Validate contiguous result positions and reject unclaimed outputs before + either backend emits source. +- [ ] Prove compiled legacy/direct-plan parity, then update the route counts. + ## Phase 5 — Strings Scope: scalar character values, fixed-length strings, deferred-length strings, diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 3de9f996f..99f8dcd57 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -622,7 +622,32 @@ Python immutable scalars cannot expose native in-place mutation. Scalar `intent(out)` values are hidden and returned as new Python values, while mutable semantics for strings use replacement projection as described below. -Runtime tests: [`test_verified_baseline.py`](../../../tests/wrapper/fortran/scalars/test_verified_baseline.py). +Editable semantic contracts distinguish three numeric scalar boundaries: + +- `Float64` accepts a scalar value. If a writable native reference is projected + back with `Returns["value", Float64]`, x2py copies into call-local storage and + returns the mutated replacement; the original Python scalar is unchanged. +- `Float64[()]` accepts caller-owned rank-zero NumPy storage. x2py validates its + exact dtype, native byte order, alignment, rank, and writeability, then passes + its data address so native `out` or `inout` mutation remains visible in the + same array. +- `Addr(Float64)` accepts an integer raw address and forwards it without copying + or owning the pointee. For a NumPy buffer, pass `value.ctypes.data`. + +```python +storage = np.array(3.5, dtype=np.float64) +update_storage(storage) + +raw_storage = np.array(4.5, dtype=np.float64) +update_raw(raw_storage.ctypes.data) +``` + +`Addr(Arg(i))` inside `@native_call(...)` is different from `Addr(T)`: it tells +the wrapper to take the address of its converted call-local scalar. It does not +make the Python caller pass an address. + +Runtime tests: [`test_verified_baseline.py`](../../../tests/wrapper/fortran/scalars/test_verified_baseline.py) +and [`test_scalar_boundary_plan.py`](../../../tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py). ## Generic Procedure Interfaces diff --git a/tests/codegen/printers/_support.py b/tests/codegen/printers/_support.py index 1fc49c12e..e2a847fe6 100644 --- a/tests/codegen/printers/_support.py +++ b/tests/codegen/printers/_support.py @@ -110,6 +110,7 @@ def normalize(text: str) -> str: "SemanticType", "SemanticVariable", "_parse_pyi_text", + "complete_semantic_policies", "emit_module", "emit_module_stubs", "fortran_module_to_semantic_module", diff --git a/tests/codegen/printers/test_calls_and_policy_metadata.py b/tests/codegen/printers/test_calls_and_policy_metadata.py index 0f119d2c9..0b64c3646 100644 --- a/tests/codegen/printers/test_calls_and_policy_metadata.py +++ b/tests/codegen/printers/test_calls_and_policy_metadata.py @@ -19,6 +19,7 @@ SemanticStorageContract, SemanticType, SemanticVariable, + complete_semantic_policies, emit_module, fortran_module_to_semantic_module, generate_pyi, @@ -319,7 +320,7 @@ def test_runtime_status_policy_rejects_invalid_output_contracts(source: str, mes loaded = parse_pyi_text(source, module_name="invalid_runtime_policy") with pytest.raises(ValueError, match=message): - semantic_ir_to_codegen_ast(loaded, Scope(name=loaded.name, scope_type="module")) + complete_semantic_policies(loaded) @pytest.mark.parametrize( diff --git a/tests/pipeline/test_rendered_wrapper_artifact_build.py b/tests/pipeline/test_rendered_wrapper_artifact_build.py index a522cc8bf..d3d795c1d 100644 --- a/tests/pipeline/test_rendered_wrapper_artifact_build.py +++ b/tests/pipeline/test_rendered_wrapper_artifact_build.py @@ -79,6 +79,7 @@ def scale(x: Float64) -> Float64: ... produced_objects=(native_obj.module_target,), link_items=(NativeLinkItem("object", native_obj.module_target),), module_dirs=(native_dir,), + library_dirs=(native_dir,), ) compiler = RecordingCompiler() @@ -116,6 +117,7 @@ def scale(x: Float64) -> Float64: ... assert native_dir in bridge_obj.include assert tuple(binding_obj.dependencies) == (native_obj, bridge_obj, runtime_obj) assert binding_obj.link_args == (str(native_obj.module_target),) + assert native_dir in binding_obj.libdir assert binding_obj.extra_compilation_tools == {"python"} assert compiler.linked == (binding_obj, tmp_path / "extension", "fortran", False, "plan_scalar_build") diff --git a/tests/pipeline/test_wrapper_plan_route_selection.py b/tests/pipeline/test_wrapper_plan_route_selection.py index 0c85dc26e..07485d2e9 100644 --- a/tests/pipeline/test_wrapper_plan_route_selection.py +++ b/tests/pipeline/test_wrapper_plan_route_selection.py @@ -43,7 +43,11 @@ def scale(x: Float64) -> Float64: ... assert decision.owner_path == "fmath" assert decision.selected_route == "wrapper-plan" assert decision.uses_wrapper_plan is True - assert decision.covered_lanes == ("scalar-inputs", "scalar-direct-results") + assert decision.covered_lanes == ( + "scalar-inputs", + "scalar-direct-results", + "native-call-runtime", + ) assert decision.blockers == () assert decision.rollout_eligible is False assert decision.rollout_evidence == ( @@ -59,6 +63,15 @@ def scale(x: Float64) -> Float64: ... "test_whole_scalar_module_variable_behavior_matches_legacy_route", "tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::" "test_complete_general_source_preserves_namespaces_through_both_routes", + "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" + "test_compiled_runtime_policies_release_gil_and_project_native_errors", + "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" + "test_pyi_runtime_policies_release_gil_and_project_native_errors", + "tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", ) assert decision.selection_reason == "wrapper-plan route forced for internal migration verification" @@ -67,7 +80,7 @@ def scale(x: Float64) -> Float64: ... assert rendered.extension_init_name == "PyInit_fmath" -def test_route_selector_keeps_core_scalar_module_legacy_while_production_rollout_is_deferred(): +def test_route_selector_selects_core_scalar_module_for_production_plan_rollout(): module = _completed_module( """ def scale(x: Float64) -> Float64: ... @@ -81,11 +94,15 @@ def scale(x: Float64) -> Float64: ... strict_wrapper_names=False, ) - assert decision.selected_route == "legacy" - assert decision.covered_lanes == ("scalar-inputs", "scalar-direct-results") + assert decision.selected_route == "wrapper-plan" + assert decision.covered_lanes == ( + "scalar-inputs", + "scalar-direct-results", + "native-call-runtime", + ) assert decision.blockers == () - assert decision.rollout_eligible is False - assert decision.selection_reason == "GIL runtime parity is deferred to Phase 2D" + assert decision.rollout_eligible is True + assert decision.selection_reason == "whole generation unit is covered by completed wrapper-plan lanes" @pytest.mark.parametrize( @@ -98,7 +115,12 @@ def scale(x: Float64) -> Float64: ... def optional_value(base: Int32, value: Int32 = ...) -> Int32: ... """, "optional_value", - ("scalar-inputs", "scalar-optional-inputs", "scalar-direct-results"), + ( + "scalar-inputs", + "scalar-optional-inputs", + "scalar-direct-results", + "native-call-runtime", + ), ), ( """ @@ -111,12 +133,13 @@ def descriptor(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... "scalar-optional-inputs", "scalar-descriptor-inputs", "scalar-direct-results", + "native-call-runtime", ), ), ( 'def bump(value: Annotated[Int32, Immutable]) -> Returns["value", Int32]: ...', "scalar_writeback", - ("scalar-inputs", "scalar-writebacks"), + ("scalar-inputs", "scalar-writebacks", "native-call-runtime"), ), ), ) @@ -139,6 +162,65 @@ def test_route_selector_accepts_completed_phase3_scalar_lanes_for_forced_whole_m assert decision.blockers == () +@pytest.mark.parametrize( + ("source", "module_name", "covered_lanes"), + ( + ( + "def update(value: Float64[()]) -> None: ...", + "scalar_storage_route", + ("scalar-storage-inputs", "void-calls", "native-call-runtime"), + ), + ( + "def update(value: Addr(Float64)) -> None: ...", + "scalar_raw_address_route", + ("scalar-raw-address-inputs", "void-calls", "native-call-runtime"), + ), + ), +) +def test_route_selector_accepts_isolated_scalar_address_boundaries( + source: str, + module_name: str, + covered_lanes: tuple[str, ...], +): + module = _completed_module(source, module_name=module_name) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + force_wrapper_plan=True, + ) + + assert decision.selected_route == "wrapper-plan" + assert decision.covered_lanes == covered_lanes + assert decision.blockers == () + + +def test_route_selector_selects_completed_scalar_address_boundaries_in_production(): + module = _completed_module( + """ +def update_storage(value: Float64[()]) -> None: ... +def update_raw(value: Addr(Float64)) -> None: ... +""", + module_name="scalar_address_boundaries", + ) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "wrapper-plan" + assert decision.rollout_eligible is True + assert decision.covered_lanes == ( + "scalar-storage-inputs", + "void-calls", + "native-call-runtime", + "scalar-raw-address-inputs", + ) + + def test_route_selector_accepts_completed_scalar_module_variable_lane(): module = _completed_module( """ @@ -161,6 +243,7 @@ def summarize() -> Int32: ... assert decision.selected_route == "wrapper-plan" assert decision.covered_lanes == ( "scalar-direct-results", + "native-call-runtime", "scalar-module-variables", ) assert decision.blockers == () @@ -187,12 +270,39 @@ def value(x: Int32) -> Int32: ... assert decision.selected_route == "wrapper-plan" assert decision.covered_lanes == ( "void-calls", + "native-call-runtime", "scalar-inputs", "scalar-direct-results", "python-namespaces", ) +def test_route_selector_records_completed_native_status_error_lane(): + module = _completed_module( + """ +@raises(status="status", message="message", success=0) +@native_call([Addr(Arg(0)), Return("status", 0), Return("message", 1)]) +def solve(value: Int32) -> tuple[Int32, String[32]]: ... +""", + module_name="runtime_status", + ) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "wrapper-plan" + assert decision.rollout_eligible is True + assert decision.covered_lanes == ( + "scalar-inputs", + "void-calls", + "native-call-runtime", + "native-status-errors", + ) + + def test_route_selector_keeps_a_module_with_any_unsupported_member_entirely_legacy(): module = _completed_module( """ @@ -214,6 +324,46 @@ def sum_values(values: Float64[:]) -> Float64: ... assert decision.selection_reason == "generation unit has unsupported wrapper-plan owners" +def test_route_selector_keeps_unimplemented_scalar_kinds_on_legacy_route(): + module = _completed_module( + "def identity(value: Float128) -> Float128: ...", + module_name="wide_scalar", + ) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "legacy" + assert [blocker.reason for blocker in decision.blockers] == [ + "argument 'value' is not a first-lane primitive scalar", + "result is not a first-lane primitive scalar", + ] + + +def test_route_selector_never_silently_falls_back_when_plan_route_is_forced(): + module = _completed_module( + """ +def scale(x: Float64) -> Float64: ... +def sum_values(values: Float64[:]) -> Float64: ... +""", + module_name="fmath", + ) + + with pytest.raises( + ValueError, + match=r"cannot force wrapper-plan route.*fmath\.sum_values", + ): + build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + force_wrapper_plan=True, + ) + + def test_route_selector_keeps_an_explicitly_forced_legacy_module_entirely_legacy(): module = _completed_module( """ @@ -302,7 +452,7 @@ def fail_legacy_lowering(*args, **kwargs): ) -def test_source_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): +def test_default_source_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): class FailingWrapperPlanner: def __init__(self, **_kwargs): pass @@ -320,7 +470,6 @@ def fail_legacy_lowering(*args, **kwargs): build_pipeline.build_fortran_extension( wrapper_source("fmath.f"), output_dir=tmp_path, - _force_wrapper_plan_route=True, ) @@ -350,6 +499,11 @@ def test_source_and_pyi_forced_plan_routes_build_complete_extensions(tmp_path: P assert pyi_result.module_name == "fmath" assert source_result.shared_library.exists() assert pyi_result.shared_library.exists() + assert pyi_result.manifest is not None + assert pyi_result.manifest["native_build_plan"] == build_pipeline._manifest_native_plan( + pyi_result.native_build_plan, + base=pyi_result.output_dir, + ) def test_pyi_plan_build_failure_does_not_pre_run_or_retry_legacy_lowering(monkeypatch, tmp_path: Path): @@ -374,7 +528,7 @@ def fail_legacy_lowering(*args, **kwargs): ) -def test_pyi_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): +def test_default_pyi_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): class FailingWrapperPlanner: def __init__(self, **_kwargs): pass @@ -396,7 +550,6 @@ def fail_legacy_lowering(*args, **kwargs): contract, native_objects=[native_object], output_dir=tmp_path / "build", - _force_wrapper_plan_route=True, ) diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index d3435c703..2013f63de 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -9,8 +9,9 @@ from x2py.pipeline.pyi import pyi_file_to_semantic_module from x2py.semantics.fortran2ir import fortran_project_to_semantic_modules from x2py.semantics.models import ( - RESOLVED_MODULE_VARIABLE_POLICY_METADATA, RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA, + RESOLVED_MODULE_VARIABLE_POLICY_METADATA, + RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA, SemanticFunction, SemanticType, ) @@ -25,10 +26,13 @@ ) from x2py.semantics.policy_completion import complete_semantic_policies from x2py.semantics.wrapper_policy import ( + BridgeDataAction, + FunctionWrapperPolicy, ModuleGetterAction, ModuleVariablePolicy, + NativeStatusErrorPolicy, OptionalMode, - FunctionWrapperPolicy, + PythonExceptionKind, WritebackPhase, completed_function_wrapper_policy, ) @@ -97,6 +101,8 @@ def alloc_state(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... assert policy.supported is True assert value.optional_mode is OptionalMode.DESCRIPTOR + assert value.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert value.bridge_copy_reason == "materialize owned Fortran allocatable scalar storage from the binding value" assert value.nullable is True assert value.descriptor_boundary is True assert policy.native_module == "scalar_optional_descriptors" @@ -119,6 +125,27 @@ def test_scalar_copy_in_out_policy_completes_writeback_before_planning(): assert {action.result_position for action in policy.writeback_actions} == {0} +def test_native_call_policy_maps_visible_positions_when_hidden_output_precedes_input(): + module = parse_pyi_text( + """ +@native_call([Return("status", 0), Addr(Arg(0))]) +def mapped_status(base: Int32) -> Int32: ... +""", + module_name="scalar_native_order", + ) + complete_semantic_policies(module) + + policy = completed_function_wrapper_policy(module.functions[0]) + + assert [(argument.name, argument.python_position, argument.native_position) for argument in policy.arguments] == [ + ("base", 0, 1) + ] + assert [(slot.owner_path, slot.source_kind, slot.native_position) for slot in policy.native_call_slots] == [ + ("scalar_native_order.mapped_status.status", "result", 0), + ("scalar_native_order.mapped_status.base", "projection", 1), + ] + + def test_source_fmath_scalar_policy_accepts_storage_address_native_action(): module = _source_semantic_module("fmath.f", module_name="fmath") function = next(item for item in module.functions if item.name == "ADD_R8") @@ -265,6 +292,38 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... ] +def test_runtime_status_policy_is_completed_before_wrapper_planning(): + module = parse_pyi_text( + """ +@raises(status="status", message="message", success=0) +@native_call([Addr(Arg(0)), Return("status", 0), Return("message", 1)]) +def solve(value: Int32) -> tuple[Int32, String[32]]: ... +""", + module_name="runtime_status", + ) + + complete_semantic_policies(module) + + function = module.functions[0] + status_error = function.metadata[RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA] + policy = completed_function_wrapper_policy(function) + assert isinstance(status_error, NativeStatusErrorPolicy) + assert policy.status_error is status_error + assert status_error.success == 0 + assert status_error.exception_kind is PythonExceptionKind.RUNTIME_ERROR + assert status_error.status.owner_path == "runtime_status.solve.status" + assert status_error.status.native_position == 1 + assert status_error.status.semantic_type_name == "Int32" + assert status_error.message is not None + assert status_error.message.owner_path == "runtime_status.solve.message" + assert status_error.message.native_position == 2 + assert status_error.message.semantic_type_name == "String" + assert status_error.message.character_length == 32 + assert policy.result is None + assert [slot.semantic_type_name for slot in policy.native_call_slots] == ["Int32", "Int32", "String"] + assert [slot.character_length for slot in policy.native_call_slots] == [None, None, 32] + + def test_wrapper_policy_records_implicit_native_order(): module = parse_pyi_text( """ @@ -427,11 +486,26 @@ def sum_values(values: Float64[:]) -> Float64: ... assert isinstance(policy, FunctionWrapperPolicy) assert policy.supported is False assert "argument 'values' is not a first-lane primitive scalar" in policy.blockers + assert "argument 'values' has no completed bridge data action" in policy.blockers + assert policy.arguments[0].bridge_data_action is BridgeDataAction.BLOCKED with pytest.raises(ValueError, match="blocked wrapper policy"): completed_function_wrapper_policy(function) +def test_wrapper_policy_keeps_string_arguments_blocked_until_bridge_data_action_is_completed(): + module = parse_pyi_text( + "def consume(value: String) -> None: ...", + module_name="string_argument", + ) + complete_semantic_policies(module) + policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert policy.supported is False + assert "argument 'value' has no completed bridge data action" in policy.blockers + assert policy.arguments[0].bridge_data_action is BridgeDataAction.BLOCKED + + def test_missing_wrapper_policy_fails_before_planning(): function = SemanticFunction( name="add", diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index c0154e4ad..8429e9c69 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -53,7 +53,7 @@ records the zero-legacy completion target. | Roadmap item | Evidence | | --- | --- | -| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | +| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators; isolated scalar-only parity covers primitive kinds, value/address projection, hidden output, copy-in/copy-out, rank-zero storage, and raw addresses without array/string route blockers | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_boundary_plan.py`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | | Function-call contracts rebuild from generated `.pyi` fixtures with the same optional-argument handling, scalar replacement writeback, hidden output projection, multiple-result ordering, allocatable nullable outputs, native-call projection metadata, native shared-library link inputs, and validation failures as source builds | `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes`, `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules`, `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection`, `function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | | Array contracts rebuild from generated `.pyi` fixtures with the same dtype, rank, shape, order, stride, lower-bound, writeability, byte-order, alignment, zero-extent, assumed-rank dispatch, ordinary Python-owned result behavior, and allocatable result-handle behavior as source builds | `arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument`, `arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides`, `arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views` | | Character contracts rebuild from generated `.pyi` fixtures with the same fixed-length buffers, assumed-length strings, deferred character results, copy-in/copy-out behavior, optional strings, Unicode handling, and embedded-NUL validation as source builds | `strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results`, `strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_projected_replacement_without_native_call_keeps_writable_argument_storage`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_native_call_projected_output_keeps_visible_storage_writable` | @@ -143,6 +143,7 @@ records the zero-legacy completion target. ## Scalars - `scalars/test_fortran_enums.py` +- `scalars/test_scalar_boundary_plan.py` - `scalars/test_scalar_generated_pyi_contracts.py` - `scalars/test_scalar_kinds.py` - `scalars/test_value_and_bind_c.py` diff --git a/tests/wrapper/fortran/external_routines/test_external_procedures.py b/tests/wrapper/fortran/external_routines/test_external_procedures.py index 0734f53fd..dc02dfd19 100644 --- a/tests/wrapper/fortran/external_routines/test_external_procedures.py +++ b/tests/wrapper/fortran/external_routines/test_external_procedures.py @@ -222,7 +222,6 @@ def test_external_bridge_uses_explicit_interface_and_no_module_use(tmp_path: Pat assert "function free_square(" in bridge assert "end function free_square" in bridge assert "private\n" not in bridge - assert "private :: c_malloc" in bridge assert "public :: bind_c_free_square" not in bridge assert "use free_external" not in bridge diff --git a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py index aed46186b..5b7adf288 100644 --- a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py +++ b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py @@ -58,6 +58,7 @@ ), "scalars": ( "test_fortran_enums.py", + "test_scalar_boundary_plan.py", "test_scalar_generated_pyi_contracts.py", "test_scalar_kinds.py", "test_value_and_bind_c.py", diff --git a/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py b/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py index 2f030172b..9727b5c46 100644 --- a/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py +++ b/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py @@ -22,9 +22,41 @@ MODIFIED_POLICY_CONTRACT = ( Path(__file__).parent / "modified_contracts" / "fruntime_policy_f90" / "fruntime_policy_f90.pyi" ) +ROUTES = ( + pytest.param("legacy", {"_force_legacy_wrapper_route": True}, id="legacy"), + pytest.param("wrapper-plan", {"_force_wrapper_plan_route": True}, id="wrapper-plan"), +) + + +def _wrapper_start(source: str, route: str, function_name: str) -> int: + marker = ( + f"static PyObject* bind_c_{function_name}_wrapper" + if route == "legacy" + else f"static PyObject * wrap_{function_name}" + ) + return source.index(marker) + + +def _assert_runtime_policy_source(source: str, route: str) -> None: + released_start = _wrapper_start(source, route, "pause_for_one_second") + held_start = _wrapper_start(source, route, "pause_with_gil") + solve_start = _wrapper_start(source, route, "solve") + released_wrapper = source[released_start:held_start] + held_wrapper = source[held_start:solve_start] + assert "Py_BEGIN_ALLOW_THREADS" in released_wrapper + assert "Py_END_ALLOW_THREADS" in released_wrapper + assert "Py_BEGIN_ALLOW_THREADS" not in held_wrapper + assert "Py_END_ALLOW_THREADS" not in held_wrapper + assert "PyErr_SetObject(PyExc_RuntimeError" in source -def test_compiled_runtime_policies_release_gil_and_project_native_errors(tmp_path: Path, monkeypatch): +@pytest.mark.parametrize(("route", "route_kwargs"), ROUTES) +def test_compiled_runtime_policies_release_gil_and_project_native_errors( + tmp_path: Path, + monkeypatch, + route: str, + route_kwargs: dict[str, bool], +): from x2py.pipeline import build from x2py.semantics.models import RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA @@ -45,7 +77,7 @@ def convert_with_runtime_policy(*args, **kwargs): return modules monkeypatch.setattr(build, "fortran_project_to_semantic_modules", convert_with_runtime_policy) - result = build.build_fortran_extension(source, output_dir=tmp_path) + result = build.build_fortran_extension(source, output_dir=tmp_path, **route_kwargs) sys.modules.pop(result.module_name, None) sys.path.insert(0, str(tmp_path)) @@ -80,25 +112,22 @@ def native_pause(): sys.path.remove(str(tmp_path)) wrapper_source = (tmp_path / "fruntime_policy_f90_wrapper.c").read_text(encoding="utf-8") - released_start = wrapper_source.index("static PyObject* bind_c_pause_for_one_second_wrapper") - held_start = wrapper_source.index("static PyObject* bind_c_pause_with_gil_wrapper") - solve_start = wrapper_source.index("static PyObject* bind_c_solve_wrapper") - released_wrapper = wrapper_source[released_start:held_start] - held_wrapper = wrapper_source[held_start:solve_start] - assert "Py_BEGIN_ALLOW_THREADS" in released_wrapper - assert "Py_END_ALLOW_THREADS" in released_wrapper - assert "Py_BEGIN_ALLOW_THREADS" not in held_wrapper - assert "Py_END_ALLOW_THREADS" not in held_wrapper - assert "PyErr_SetObject(PyExc_RuntimeError" in wrapper_source + _assert_runtime_policy_source(wrapper_source, route) -def test_pyi_runtime_policies_release_gil_and_project_native_errors(tmp_path: Path): +@pytest.mark.parametrize(("route", "route_kwargs"), ROUTES) +def test_pyi_runtime_policies_release_gil_and_project_native_errors( + tmp_path: Path, + route: str, + route_kwargs: dict[str, bool], +): native_object = _compile_native_object(RUNTIME_POLICY_SOURCE, tmp_path / "native") result = build_pyi_extension( MODIFIED_POLICY_CONTRACT, native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "pyi_build", + **route_kwargs, ) module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) @@ -129,13 +158,4 @@ def native_pause(): assert failures == [] wrapper_source = (result.output_dir / "fruntime_policy_f90_wrapper.c").read_text(encoding="utf-8") - released_start = wrapper_source.index("static PyObject* bind_c_pause_for_one_second_wrapper") - held_start = wrapper_source.index("static PyObject* bind_c_pause_with_gil_wrapper") - solve_start = wrapper_source.index("static PyObject* bind_c_solve_wrapper") - released_wrapper = wrapper_source[released_start:held_start] - held_wrapper = wrapper_source[held_start:solve_start] - assert "Py_BEGIN_ALLOW_THREADS" in released_wrapper - assert "Py_END_ALLOW_THREADS" in released_wrapper - assert "Py_BEGIN_ALLOW_THREADS" not in held_wrapper - assert "Py_END_ALLOW_THREADS" not in held_wrapper - assert "PyErr_SetObject(PyExc_RuntimeError" in wrapper_source + _assert_runtime_policy_source(wrapper_source, route) diff --git a/tests/wrapper/fortran/scalars/README.md b/tests/wrapper/fortran/scalars/README.md index 32c4e7e9a..121087ab3 100644 --- a/tests/wrapper/fortran/scalars/README.md +++ b/tests/wrapper/fortran/scalars/README.md @@ -1,7 +1,8 @@ # Scalars Scope: scalar calls, scalar kind coverage, `value` and scalar `bind(C)` -behavior, enum-like values, and the basic compiled-wrapper baseline. +behavior, value/storage/raw-address boundaries, scalar output and inout +projection, enum-like values, and the basic compiled-wrapper baseline. Focused pytest command: `python3 -m pytest -q tests/wrapper/fortran/scalars` @@ -13,6 +14,6 @@ Contract fixtures: generated scalar packages live under Roadmap items: Stage 5 generated-contract runtime parity for scalar ABI types, kinds, intents, and Python-visible values. -Tests: `test_fortran_enums.py`, `test_scalar_generated_pyi_contracts.py`, -`test_scalar_kinds.py`, `test_value_and_bind_c.py`, +Tests: `test_fortran_enums.py`, `test_scalar_boundary_plan.py`, +`test_scalar_generated_pyi_contracts.py`, `test_scalar_kinds.py`, `test_value_and_bind_c.py`, `test_verified_baseline.py`. diff --git a/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py new file mode 100644 index 000000000..8ceccffd8 --- /dev/null +++ b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py @@ -0,0 +1,301 @@ +"""Isolated compiled parity for primitive scalar boundary representations.""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from tests.wrapper.fortran._support import ( + _compile_native_object, + _import_from_build_dir, + _sole_native_module, +) +from x2py import build_pyi_extension + + +def _build_contract_routes( + tmp_path: Path, + *, + module_name: str, + source_text: str, + contract_text: str, +): + source = tmp_path / f"{module_name}.f90" + source.write_text(source_text, encoding="utf-8") + contract = tmp_path / f"{module_name}.pyi" + contract.write_text(contract_text, encoding="utf-8") + native_object = _compile_native_object(source, tmp_path / "native") + + modules = [] + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {}), + ): + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + generated_c = (result.output_dir / f"{module_name}_wrapper.c").read_text(encoding="utf-8") + if route == "wrapper_plan": + assert "static PyObject * wrap_" in generated_c + modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) + return tuple(modules) + + +def _build_scalar_boundary_modules(tmp_path: Path): + return _build_contract_routes( + tmp_path, + module_name="scalar_boundary_plan", + source_text=""" +module scalar_boundary_plan + use iso_c_binding, only: c_double, c_int32_t +contains + function value_input(value) result(output) + integer(c_int32_t), intent(in) :: value + integer(c_int32_t) :: output + output = value + 2 + end function value_input + + subroutine bump_value(value) + integer(c_int32_t), intent(inout) :: value + value = value + 1 + end subroutine bump_value + + subroutine bump_storage(value) + integer(c_int32_t), intent(inout) :: value + value = value + 1 + end subroutine bump_storage + + subroutine bump_raw(value) + integer(c_int32_t), intent(inout) :: value + value = value + 1 + end subroutine bump_raw + + subroutine bump_storage_float(value) + real(c_double), intent(inout) :: value + value = value * 2.0_c_double + end subroutine bump_storage_float + + subroutine bump_raw_float(value) + real(c_double), intent(inout) :: value + value = value * 2.0_c_double + end subroutine bump_raw_float + + subroutine make_value(value) + integer(c_int32_t), intent(out) :: value + value = 41 + end subroutine make_value + + subroutine make_storage(value) + integer(c_int32_t), intent(out) :: value + value = 42 + end subroutine make_storage + + subroutine make_raw(value) + integer(c_int32_t), intent(out) :: value + value = 43 + end subroutine make_raw + + subroutine mapped_status(status, base) + integer(c_int32_t), intent(out) :: status + integer(c_int32_t), intent(in) :: base + status = base + 11 + end subroutine mapped_status +end module scalar_boundary_plan +""", + contract_text=""" +from x2py.contracts import Addr, Annotated, Arg, Float64, Immutable, Int32, Return, Returns, native_call + +def value_input(value: Int32) -> Int32: ... + +def bump_value( + value: Annotated[Int32, Immutable] +) -> Returns["value", Int32]: ... + +def bump_storage(value: Int32[()]) -> None: ... + +def bump_raw(value: Addr(Int32)) -> None: ... + +def bump_storage_float(value: Float64[()]) -> None: ... + +def bump_raw_float(value: Addr(Float64)) -> None: ... + +@native_call([Return("value", 0)]) +def make_value() -> Int32: ... + +def make_storage(value: Int32[()]) -> None: ... + +def make_raw(value: Addr(Int32)) -> None: ... + +@native_call([Return("status", 0), Addr(Arg(0))]) +def mapped_status(base: Int32) -> Int32: ... +""", + ) + + +def _build_scalar_kind_modules(tmp_path: Path): + return _build_contract_routes( + tmp_path, + module_name="scalar_kind_plan", + source_text=""" +module scalar_kind_plan + use iso_c_binding, only: c_bool, c_double, c_double_complex, c_float, & + c_float_complex, c_int8_t, c_int16_t, c_int32_t, c_int64_t +contains + function id_i8(value) result(output) + integer(c_int8_t), intent(in) :: value + integer(c_int8_t) :: output + output = value + end function id_i8 + + function id_i16(value) result(output) + integer(c_int16_t), intent(in) :: value + integer(c_int16_t) :: output + output = value + end function id_i16 + + function id_i32(value) result(output) + integer(c_int32_t), intent(in) :: value + integer(c_int32_t) :: output + output = value + end function id_i32 + + function id_i64(value) result(output) + integer(c_int64_t), intent(in) :: value + integer(c_int64_t) :: output + output = value + end function id_i64 + + function id_bool(value) result(output) + logical(c_bool), intent(in) :: value + logical(c_bool) :: output + output = value + end function id_bool + + function id_r32(value) result(output) + real(c_float), intent(in) :: value + real(c_float) :: output + output = value + end function id_r32 + + function id_r64(value) result(output) + real(c_double), intent(in) :: value + real(c_double) :: output + output = value + end function id_r64 + + function conj_c64(value) result(output) + complex(c_float_complex), intent(in) :: value + complex(c_float_complex) :: output + output = conjg(value) + end function conj_c64 + + function conj_c128(value) result(output) + complex(c_double_complex), intent(in) :: value + complex(c_double_complex) :: output + output = conjg(value) + end function conj_c128 +end module scalar_kind_plan +""", + contract_text=""" +from x2py.contracts import Addr, Arg, Bool, Complex128, Complex64, Float32, Float64, Int16, Int32, Int64, Int8, native_call + +@native_call([Addr(Arg(0))]) +def id_i8(value: Int8) -> Int8: ... + +@native_call([Addr(Arg(0))]) +def id_i16(value: Int16) -> Int16: ... + +@native_call([Addr(Arg(0))]) +def id_i32(value: Int32) -> Int32: ... + +@native_call([Addr(Arg(0))]) +def id_i64(value: Int64) -> Int64: ... + +@native_call([Addr(Arg(0))]) +def id_bool(value: Bool) -> Bool: ... + +@native_call([Addr(Arg(0))]) +def id_r32(value: Float32) -> Float32: ... + +@native_call([Addr(Arg(0))]) +def id_r64(value: Float64) -> Float64: ... + +@native_call([Addr(Arg(0))]) +def conj_c64(value: Complex64) -> Complex64: ... + +@native_call([Addr(Arg(0))]) +def conj_c128(value: Complex128) -> Complex128: ... +""", + ) + + +def test_scalar_value_storage_raw_address_out_and_inout_match_both_routes(tmp_path: Path): + modules = _build_scalar_boundary_modules(tmp_path) + + for module in modules: + assert module.value_input(np.int32(5)) == np.int32(7) + + original = np.int32(4) + replacement = module.bump_value(original) + assert original == np.int32(4) + assert replacement == np.int32(5) + + storage = np.array(6, dtype=np.int32) + assert module.bump_storage(storage) is None + assert storage[()] == np.int32(7) + + raw = np.array(8, dtype=np.int32) + assert module.bump_raw(raw.ctypes.data) is None + assert raw[()] == np.int32(9) + + float_storage = np.array(3.5, dtype=np.float64) + assert module.bump_storage_float(float_storage) is None + assert float_storage[()] == np.float64(7.0) + + float_raw = np.array(4.5, dtype=np.float64) + assert module.bump_raw_float(float_raw.ctypes.data) is None + assert float_raw[()] == np.float64(9.0) + + assert module.make_value() == np.int32(41) + + output_storage = np.empty((), dtype=np.int32) + assert module.make_storage(output_storage) is None + assert output_storage[()] == np.int32(42) + + output_raw = np.empty((), dtype=np.int32) + assert module.make_raw(output_raw.ctypes.data) is None + assert output_raw[()] == np.int32(43) + + assert module.mapped_status(np.int32(4)) == np.int32(15) + + with pytest.raises(TypeError): + module.bump_storage(np.int32(6)) + with pytest.raises(TypeError): + module.bump_storage(np.array(6, dtype=np.int64)) + read_only = np.array(6, dtype=np.int32) + read_only.flags.writeable = False + with pytest.raises(TypeError, match="writeable"): + module.bump_storage(read_only) + with pytest.raises(TypeError): + module.bump_raw(raw) + + +def test_scalar_primitive_kinds_match_both_routes_without_array_blockers(tmp_path: Path): + modules = _build_scalar_kind_modules(tmp_path) + + for module in modules: + assert module.id_i8(np.int8(np.iinfo(np.int8).min)) == np.int8(np.iinfo(np.int8).min) + assert module.id_i16(np.int16(np.iinfo(np.int16).max)) == np.int16(np.iinfo(np.int16).max) + assert module.id_i32(np.int32(np.iinfo(np.int32).min)) == np.int32(np.iinfo(np.int32).min) + assert module.id_i64(np.int64(2**40)) == np.int64(2**40) + assert bool(module.id_bool(True)) is True + assert module.id_r32(np.float32(1.25)) == np.float32(1.25) + assert module.id_r64(np.float64(-2.5)) == np.float64(-2.5) + assert module.conj_c64(np.complex64(1 + 2j)) == np.complex64(1 - 2j) + assert module.conj_c128(np.complex128(2 + 3j)) == np.complex128(2 - 3j) diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index c2509f3b4..6c9b82638 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -66,7 +66,8 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... assert "static PyObject * wrap_swap_args" in c_header assert "module bind_c_render_demo_wrapper" in fortran_source assert 'function bind_c_swap_args(y, x) result(result) bind(c, name="bind_c_swap_args")' in fortran_source - assert "real(c_double), external :: SWAP_ARGS" in fortran_source + assert "function SWAP_ARGS(y, x) result(native_result)" in fortran_source + assert "real(c_double) :: native_result" in fortran_source assert "result = SWAP_ARGS(y, x)" in fortran_source diff --git a/tests/wrapper_codegen/test_phase1b_scalar_input_conversion.py b/tests/wrapper_codegen/test_phase1b_scalar_input_conversion.py index 5ef6fbbd6..274e31432 100644 --- a/tests/wrapper_codegen/test_phase1b_scalar_input_conversion.py +++ b/tests/wrapper_codegen/test_phase1b_scalar_input_conversion.py @@ -12,8 +12,11 @@ @pytest.mark.parametrize( ("type_name", "c_type", "converter", "check"), [ - ("Bool", "bool", "PyBool_to_Bool", "PyArray_IsScalar(x_obj, Bool)"), + ("Bool", "bool", "PyBool_to_Bool", "PyIs_Bool(x_obj)"), + ("Int8", "int8_t", "PyInt8_to_Int8", "PyIs_Int8(x_obj)"), + ("Int16", "int16_t", "PyInt16_to_Int16", "PyIs_Int16(x_obj)"), ("Int32", "int32_t", "PyInt32_to_Int32", "PyArray_IsScalar(x_obj, Int)"), + ("Int64", "int64_t", "PyInt64_to_Int64", "PyIs_Int64(x_obj)"), ("Float32", "float", "PyFloat_to_Float", "PyArray_IsScalar(x_obj, Float)"), ("Float64", "double", "PyDouble_to_Double", "PyArray_IsScalar(x_obj, Double)"), ("Complex64", "float complex", "PyComplex_to_Complex64", "PyArray_IsScalar(x_obj, CFloat)"), diff --git a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py index ebc5b9558..60c8dea20 100644 --- a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py +++ b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py @@ -18,7 +18,14 @@ def scale(x: Float64) -> Float64: ... module_name="hidden_result", ) complete_semantic_policies(module) - artifacts = WrapperCodeGenerator().generate(WrapperPlanner().build(module)) + plan = WrapperPlanner().build(module) + function = plan.namespaces[0].functions[0] + result = function.result + + assert result is not None + assert result.native_call_slot is function.native_call_slots[result.bridge.abi_position] + + artifacts = WrapperCodeGenerator().generate(plan) c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") fortran_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") diff --git a/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py b/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py new file mode 100644 index 000000000..a8b8c9aab --- /dev/null +++ b/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py @@ -0,0 +1,163 @@ +"""Phase 2D native-call runtime envelope and status-error lowering tests.""" + +from __future__ import annotations + +from dataclasses import replace +from pathlib import Path + +import pytest + +from x2py.pipeline.pyi import pyi_file_to_semantic_module +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import BridgeDataAction, PythonExceptionKind +from x2py.wrapper_codegen import DatatypeFamily, WrapperCodeGenerator, WrapperPlanner + + +RUNTIME_POLICY_CONTRACT = ( + Path("tests/wrapper/fortran/runtime_behavior/modified_contracts") + / "fruntime_policy_f90" + / "fruntime_policy_f90.pyi" +) +RECURSION_CONTRACT = ( + Path("tests/wrapper/fortran/runtime_behavior/contracts") / "fruntime_recursion_f90" / "fruntime_recursion_f90.pyi" +) + + +def _runtime_plan(): + module = pyi_file_to_semantic_module(RUNTIME_POLICY_CONTRACT, module_name="fruntime_policy_f90") + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def _rendered_source(artifacts, suffix: str) -> str: + return next(source.text for source in artifacts.sources if source.path.name.endswith(suffix)) + + +def _function_source(source: str, function_name: str, next_name: str | None = None) -> str: + start = source.index(f"static PyObject * wrap_{function_name}") + if next_name is None: + return source[start : source.index("PyMODINIT_FUNC", start)] + return source[start : source.index(f"static PyObject * wrap_{next_name}", start)] + + +def _edit_function(plan, function_name: str, edit): + root = plan.namespaces[0] + functions = tuple( + edit(function) if function.binding.python_name == function_name else function for function in root.functions + ) + return replace(plan, namespaces=(replace(root, functions=functions), *plan.namespaces[1:])) + + +def test_planner_records_editable_native_runtime_and_status_error_facts(): + plan = _runtime_plan() + functions = {function.binding.python_name: function for function in plan.namespaces[0].functions} + solve = functions["solve"] + + assert functions["pause_for_one_second"].binding.hold_gil is False + assert functions["pause_with_gil"].binding.hold_gil is True + assert solve.binding.hold_gil is False + assert solve.binding.status_error is not None + assert solve.binding.status_error.success == 0 + assert solve.binding.status_error.exception_kind is PythonExceptionKind.RUNTIME_ERROR + assert solve.binding.status_error.status_role == solve.native_call_slots[1].symbolic_role + assert solve.binding.status_error.message_role == solve.native_call_slots[2].symbolic_role + assert solve.native_call_slots[1].semantic_type_name == "Int32" + assert solve.native_call_slots[1].datatype_family is DatatypeFamily.INTEGER + assert solve.native_call_slots[1].bridge_data_action is BridgeDataAction.DIRECT_TRANSFER + assert solve.native_call_slots[1].bridge_copy_reason is None + assert solve.native_call_slots[2].semantic_type_name == "String" + assert solve.native_call_slots[2].datatype_family is DatatypeFamily.STRING + assert solve.native_call_slots[2].character_length == 32 + assert solve.native_call_slots[2].bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert solve.native_call_slots[2].bridge_copy_reason == ( + "copy fixed-length Fortran character output into C-owned null-terminated storage" + ) + + +def test_direct_binding_lowering_places_only_native_call_outside_the_gil(): + artifacts = WrapperCodeGenerator().generate(_runtime_plan()) + c_source = _rendered_source(artifacts, ".c") + released = _function_source(c_source, "pause_for_one_second", "pause_with_gil") + held = _function_source(c_source, "pause_with_gil", "solve") + solve = _function_source(c_source, "solve") + + assert released.index("Py_BEGIN_ALLOW_THREADS") < released.index("bind_c_pause_for_one_second()") + assert released.index("bind_c_pause_for_one_second()") < released.index("Py_END_ALLOW_THREADS") + assert "Py_BEGIN_ALLOW_THREADS" not in held + assert "Py_END_ALLOW_THREADS" not in held + assert solve.index("Py_BEGIN_ALLOW_THREADS") < solve.index("bind_c_solve(&value, &status, &message)") + assert solve.index("bind_c_solve(&value, &status, &message)") < solve.index("Py_END_ALLOW_THREADS") + assert solve.index("Py_END_ALLOW_THREADS") < solve.index("PyUnicode_FromString") + assert solve.index("PyUnicode_FromString") < solve.index("status != 0") + assert "PyErr_SetObject(PyExc_RuntimeError, message_obj)" in solve + assert "free(message)" in solve + + +def test_direct_bridge_lowering_projects_status_and_copies_fixed_message(): + artifacts = WrapperCodeGenerator().generate(_runtime_plan()) + fortran_source = _rendered_source(artifacts, ".f90") + + assert "subroutine bind_c_solve(value, status, message)" in fortran_source + assert "integer(c_int32_t) :: status" in fortran_source + assert "type(c_ptr) :: message" in fortran_source + assert "character(kind=c_char, len=32) :: message_value" in fortran_source + assert "call native_solve(value, status, message_value)" in fortran_source + assert "message = c_malloc(33_c_size_t)" in fortran_source + assert "message_copy(33) = c_null_char" in fortran_source + + +def test_fixed_message_bridge_copy_requires_its_completed_reason(): + plan = _runtime_plan() + invalid = _edit_function( + plan, + "solve", + lambda function: replace( + function, + native_call_slots=tuple( + replace(slot, bridge_copy_reason=None) if slot.datatype_family is DatatypeFamily.STRING else slot + for slot in function.native_call_slots + ), + ), + ) + + with pytest.raises(ValueError, match="missing-bridge-copy-reason"): + WrapperCodeGenerator().generate(invalid) + + +def test_runtime_plan_edits_dispatch_to_named_lowering_and_validate_roles(): + plan = _runtime_plan() + held = _edit_function( + plan, + "pause_for_one_second", + lambda function: replace(function, binding=replace(function.binding, hold_gil=True)), + ) + c_source = _rendered_source(WrapperCodeGenerator().generate(held), ".c") + released = _function_source(c_source, "pause_for_one_second", "pause_with_gil") + assert "Py_BEGIN_ALLOW_THREADS" not in released + assert "Py_END_ALLOW_THREADS" not in released + + invalid = _edit_function( + plan, + "solve", + lambda function: replace( + function, + binding=replace( + function.binding, + status_error=replace(function.binding.status_error, status_role="missing:status"), + ), + ), + ) + with pytest.raises(ValueError, match="missing-status-result-role"): + WrapperCodeGenerator().generate(invalid) + + +def test_recursive_runtime_contract_keeps_release_policy_in_the_plan(): + module = pyi_file_to_semantic_module(RECURSION_CONTRACT, module_name="fruntime_recursion_f90") + complete_semantic_policies(module) + plan = WrapperPlanner().build(module) + + assert plan.namespaces[0].functions + assert all(function.binding.hold_gil is False for function in plan.namespaces[0].functions) + c_source = _rendered_source(WrapperCodeGenerator().generate(plan), ".c") + assert c_source.count("Py_BEGIN_ALLOW_THREADS") == len(plan.namespaces[0].functions) + assert c_source.count("Py_END_ALLOW_THREADS") == len(plan.namespaces[0].functions) diff --git a/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py b/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py new file mode 100644 index 000000000..6c9a20d21 --- /dev/null +++ b/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py @@ -0,0 +1,128 @@ +"""Direct-plan scalar storage and raw-address boundary lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import NativeBarrierAction, PythonBarrierAction +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import ArgumentHandoffMode, BridgeDataAction +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + +def _scalar_boundary_plan(): + module = parse_pyi_text( + """ +def storage(x: Float64[()]) -> None: ... +def raw(x: Addr(Float64)) -> None: ... +""", + module_name="scalar_boundaries", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_scalar_storage_and_raw_address_plans_keep_explicit_boundary_facts(): + plan = _scalar_boundary_plan() + functions = {function.binding.python_name: function for function in plan.namespaces[0].functions} + storage_function = functions["storage"] + raw_function = functions["raw"] + storage = storage_function.arguments[0] + raw = raw_function.arguments[0] + + assert storage.native_call_slot is storage_function.native_call_slots[storage.native_position] + assert storage.binding.python_action is PythonBarrierAction.SCALAR_STORAGE + assert storage.binding.writable is True + assert storage.bridge.native_action is NativeBarrierAction.PASS_STORAGE_ADDRESS + assert storage.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + assert storage.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + assert storage.bridge.copy_reason is None + assert raw.native_call_slot is raw_function.native_call_slots[raw.native_position] + assert raw.binding.python_action is PythonBarrierAction.RAW_ADDRESS + assert raw.bridge.native_action is NativeBarrierAction.PASS_RAW_ADDRESS + assert raw.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + assert raw.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + assert raw.bridge.copy_reason is None + + +def test_scalar_storage_and_raw_address_lower_to_direct_named_paths(): + artifacts = WrapperCodeGenerator().generate(_scalar_boundary_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void bind_c_storage(void * x);" in c_source + assert "PyArray_TYPE((PyArrayObject *)x_obj) != NPY_FLOAT64" in c_source + assert "PyArray_NDIM((PyArrayObject *)x_obj) != 0" in c_source + assert "PyArray_ISNOTSWAPPED((PyArrayObject *)x_obj)" in c_source + assert "PyArray_ISALIGNED((PyArrayObject *)x_obj)" in c_source + assert "PyArray_ISWRITEABLE((PyArrayObject *)x_obj)" in c_source + assert "x = PyArray_DATA((PyArrayObject *)x_obj);" in c_source + assert "bind_c_storage(x);" in c_source + assert "void bind_c_raw(void * x);" in c_source + assert "if (!PyLong_Check(x_obj))" in c_source + assert "x = PyLong_AsVoidPtr(x_obj);" in c_source + assert "bind_c_raw(x);" in c_source + + assert 'subroutine bind_c_storage(bound_x) bind(c, name="bind_c_storage")' in bridge_source + assert 'subroutine bind_c_raw(bound_x) bind(c, name="bind_c_raw")' in bridge_source + assert bridge_source.count("type(c_ptr), value :: bound_x") == 2 + assert bridge_source.count("call c_f_pointer(bound_x, x)") == 2 + assert "call native_storage(x)" in bridge_source + assert "call native_raw(x)" in bridge_source + + +def test_scalar_address_handoff_plan_edits_fail_before_lowering(): + plan = _scalar_boundary_plan() + storage = plan.namespaces[0].functions[0].arguments[0] + storage.bridge.handoff_mode = ArgumentHandoffMode.VALUE + + with pytest.raises(ValueError, match="invalid-scalar-address-handoff"): + WrapperCodeGenerator().generate(plan) + + +@pytest.mark.parametrize( + ("action", "reason", "diagnostic"), + [ + (BridgeDataAction.COPY_REPRESENTATION, None, "missing-bridge-copy-reason"), + (BridgeDataAction.ASSOCIATE_VIEW, "unnecessary second copy", "unexpected-bridge-copy-reason"), + (BridgeDataAction.BLOCKED, None, "blocked-bridge-data-action"), + ], +) +def test_bridge_data_action_invariant_rejects_unjustified_or_blocked_plans(action, reason, diagnostic): + plan = _scalar_boundary_plan() + function = plan.namespaces[0].functions[0] + storage = function.arguments[0] + storage.bridge.data_action = action + storage.bridge.copy_reason = reason + storage.native_call_slot.bridge_data_action = action + storage.native_call_slot.bridge_copy_reason = reason + assert function.native_call_slots[storage.native_position] is storage.native_call_slot + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) + + +def test_scalar_copy_in_out_reuses_one_binding_local_without_bridge_copy(): + module = parse_pyi_text( + 'def bump(value: Annotated[Int32, Immutable]) -> Returns["value", Int32]: ...', + module_name="one_copy", + ) + complete_semantic_policies(module) + plan = WrapperPlanner().build(module) + value = plan.namespaces[0].functions[0].arguments[0] + assert value.bridge.data_action is BridgeDataAction.DIRECT_TRANSFER + assert value.bridge.copy_reason is None + + artifacts = WrapperCodeGenerator().generate(plan) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert c_source.count("int32_t value;") == 1 + assert "value = PyInt32_to_Int32(value_obj);" in c_source + assert "bind_c_bump(&value);" in c_source + assert "PyObject * result_obj = Int32_to_PyLong(&value);" in c_source + assert "integer(c_int32_t) :: value" in bridge_source + assert "call native_bump(value)" in bridge_source + assert "value =" not in bridge_source + assert "value_input" not in bridge_source diff --git a/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py b/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py index 11367a1a7..5f1e5cb50 100644 --- a/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py +++ b/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py @@ -10,7 +10,7 @@ from tests._shared.ownership_policy_support import parse_pyi_text from x2py.pipeline.pyi import pyi_file_to_semantic_module from x2py.semantics.policy_completion import complete_semantic_policies -from x2py.semantics.wrapper_policy import OptionalMode, WritebackPhase +from x2py.semantics.wrapper_policy import BridgeDataAction, OptionalMode, WritebackPhase from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner @@ -67,6 +67,8 @@ def alloc_state(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... assert value.binding.optional_mode is OptionalMode.DESCRIPTOR assert value.bridge.presence_role == "scalar_optional_descriptors.alloc_state.value:present" + assert value.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert value.bridge.copy_reason == "materialize owned Fortran allocatable scalar storage from the binding value" artifacts = WrapperCodeGenerator().generate(plan) c_source = _source(artifacts, ".c") fortran_source = _source(artifacts, ".f90") diff --git a/x2py/codegen/bindings/c_to_python.py b/x2py/codegen/bindings/c_to_python.py index 23620f899..2d473f51f 100644 --- a/x2py/codegen/bindings/c_to_python.py +++ b/x2py/codegen/bindings/c_to_python.py @@ -39,6 +39,7 @@ NativeArrayHandlePolicyDispatcher, NativeArrayOutputProjectionDispatcher, ) +from x2py.semantics.wrapper_policy import NativeStatusErrorPolicy, PythonExceptionKind from ..bind_c import ( BindCArrayVariable, @@ -7702,13 +7703,12 @@ def _native_call_nodes(self, func, original_func, args, results, wrapped_args, * def _status_error_output_names(original_func): """Handle status error output names for the current generation context.""" policy = getattr(original_func, "decorators", {}).get(RUNTIME_STATUS_ERROR_METADATA) - if not isinstance(policy, dict): + if not isinstance(policy, NativeStatusErrorPolicy): return set() - names = {policy.get("status")} - message = policy.get("message") - if message is not None: - names.add(message) - return {name for name in names if isinstance(name, str)} + names = {policy.status.name} + if policy.message is not None: + names.add(policy.message.name) + return names @staticmethod def _result_bindings_by_name(wrapped_results): @@ -7721,32 +7721,25 @@ def _result_bindings_by_name(wrapped_results): return bindings @staticmethod - def _validate_status_error_binding(policy, bindings): - """Validate status error binding.""" - status_name = policy.get("status") - if not isinstance(status_name, str): - raise ValueError("raises metadata requires a status output name") - status = bindings.get(status_name) + def _status_error_bindings(policy, bindings): + """Resolve already-completed status outputs in the lowered binding graph.""" + status = bindings.get(policy.status.name) if status is None: - raise ValueError(f"raises status target {status_name!r} is not a native output") - status_var = status.get("c_result") - status_dtype = getattr(status_var, "dtype", None) - if not isinstance(getattr(status_dtype, "primitive_type", None), PrimitiveIntegerType): - raise ValueError(f"raises status target {status_name!r} must be a scalar integer output") - - message_name = policy.get("message") + raise ValueError(f"completed raises status target {policy.status.name!r} is missing after lowering") message = None - if message_name is not None: - if not isinstance(message_name, str): - raise ValueError("raises message target must be an output name") - message = bindings.get(message_name) + if policy.message is not None: + message = bindings.get(policy.message.name) if message is None: - raise ValueError(f"raises message target {message_name!r} is not a native output") - original = message.get("original") - if not isinstance(getattr(original, "class_type", None), StringType): - raise ValueError(f"raises message target {message_name!r} must be a string output") + raise ValueError(f"completed raises message target {policy.message.name!r} is missing after lowering") return status, message + @staticmethod + def _status_error_exception(policy): + """Lower one completed Python exception kind without datatype inference.""" + if policy.exception_kind is PythonExceptionKind.RUNTIME_ERROR: + return PyRuntimeError + raise ValueError(f"Unsupported completed Python exception kind: {policy.exception_kind!r}") + def _status_error_check( self, original_func, @@ -7757,18 +7750,19 @@ def _status_error_check( ): """Handle status error check for the current generation context.""" policy = getattr(original_func, "decorators", {}).get(RUNTIME_STATUS_ERROR_METADATA) - if not isinstance(policy, dict): + if not isinstance(policy, NativeStatusErrorPolicy): return [] bindings = self._result_bindings_by_name(wrapped_results) - status, message = self._validate_status_error_binding(policy, bindings) + status, message = self._status_error_bindings(policy, bindings) status_var = status["c_result"] - success = int(policy.get("success", 0)) + success = policy.success + exception = self._status_error_exception(policy) if message is not None: - set_error = PyErr_SetObject(PyRuntimeError, message["py_result"]) + set_error = PyErr_SetObject(exception, message["py_result"]) else: set_error = PyErr_SetString( - PyRuntimeError, + exception, CStrStr(convert_to_literal(f"native call failed with status {status['name']} != {success}")), ) error_body = [ diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index f1fa85237..9da457014 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -3,11 +3,12 @@ from __future__ import annotations from collections.abc import Iterable -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace import json import os from pathlib import Path import shlex +import time from filelock import FileLock @@ -15,7 +16,7 @@ from x2py.codegen.scope import Scope from x2py.compiling.basic import CompileObj from x2py.compiling.compilers import Compiler, get_condaless_search_path -from x2py.compiling.python_wrapper import create_shared_library +from x2py.compiling.python_wrapper import _print_verbose_timing, create_shared_library from x2py.compiling.runtime_support import install_runtime_support from x2py.fortran_parser.parser import parse_fortran_project from x2py.probes.fortran_types import evaluate_fortran_type_facts, evaluate_fortran_type_requirements @@ -78,6 +79,8 @@ _WRAPPER_PLAN_COMPLETED_LANES = frozenset( { "scalar-inputs", + "scalar-storage-inputs", + "scalar-raw-address-inputs", "scalar-direct-results", "scalar-hidden-outputs", "scalar-optional-inputs", @@ -86,6 +89,8 @@ "scalar-module-variables", "void-calls", "python-namespaces", + "native-call-runtime", + "native-status-errors", } ) _WRAPPER_PLAN_EVIDENCE = ( @@ -101,8 +106,16 @@ "test_whole_scalar_module_variable_behavior_matches_legacy_route", "tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::" "test_complete_general_source_preserves_namespaces_through_both_routes", + "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" + "test_compiled_runtime_policies_release_gil_and_project_native_errors", + "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" + "test_pyi_runtime_policies_release_gil_and_project_native_errors", + "tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", ) -_WRAPPER_PLAN_PRODUCTION_DEFERRED_REASON = "GIL runtime parity is deferred to Phase 2D" @dataclass(frozen=True) @@ -394,6 +407,7 @@ def _rendered_wrapper_compile_obj( link_args: tuple[str, ...], flags: tuple[str, ...], include_dirs: tuple[Path, ...], + library_dirs: tuple[Path, ...], language: str, ) -> CompileObj: """Return one compile object for a rendered wrapper-plan source.""" @@ -402,6 +416,7 @@ def _rendered_wrapper_compile_obj( _rendered_artifact_output_path(output_dir, source_path).parent, flags=flags, include=include_dirs, + libdir=library_dirs, link_args=link_args, dependencies=dependencies, extra_compilation_tools=("python",) if language == "c" else (), @@ -417,6 +432,7 @@ def _rendered_wrapper_compile_objects( wrapper_fortran_flags: tuple[str, ...], wrapper_c_flags: tuple[str, ...], native_module_dirs: tuple[Path, ...], + native_library_dirs: tuple[Path, ...], ) -> tuple[tuple[CompileObj, str, Path], ...]: """Return compile objects for rendered wrapper-plan sources.""" compiled: list[CompileObj] = [] @@ -433,6 +449,7 @@ def _rendered_wrapper_compile_objects( link_args=native_link_args if source_path == final_source else (), flags=flags, include_dirs=native_module_dirs if language == "fortran" else (), + library_dirs=native_library_dirs if source_path == final_source else (), language=language, ) compiled.append(obj) @@ -467,9 +484,12 @@ def _build_rendered_wrapper_extension( output_path.mkdir(parents=True, exist_ok=True) shared_output_path = Path(shared_library_output_dir) if shared_library_output_dir is not None else output_path shared_output_path.mkdir(parents=True, exist_ok=True) + printing_started = time.perf_counter() _write_rendered_wrapper_sources(rendered, output_path) + _print_verbose_timing(verbose, "Wrapper printing", time.perf_counter() - printing_started) compiler = compiler or _new_gnu_compiler() + resolved_native_build_plan = native_build_plan or NativeBuildPlan() compile_items = _rendered_wrapper_compile_objects( rendered, output_path, @@ -477,8 +497,15 @@ def _build_rendered_wrapper_extension( native_link_args=tuple(native_link_args), wrapper_fortran_flags=_compiler_flags(wrapper_fortran_flags), wrapper_c_flags=_compiler_flags(wrapper_c_flags), - native_module_dirs=(native_build_plan or NativeBuildPlan()).module_dirs, + native_module_dirs=_unique_paths( + ( + *resolved_native_build_plan.module_dirs, + *resolved_native_build_plan.include_dirs, + ) + ), + native_library_dirs=resolved_native_build_plan.library_dirs, ) + compilation_started = time.perf_counter() runtime_imports = _rendered_wrapper_runtime_imports(rendered.artifacts.runtime_support_keys) for compile_obj, language, source_path in compile_items: imports = runtime_imports if source_path in rendered.artifacts.binding_sources else () @@ -507,6 +534,7 @@ def _build_rendered_wrapper_extension( verbose=verbose, ) ) + _print_verbose_timing(verbose, "Wrapper compilation", time.perf_counter() - compilation_started) generated_sources = tuple( path for path in rendered.artifacts.generated_files @@ -527,7 +555,7 @@ def _build_rendered_wrapper_extension( module_name=rendered.artifacts.module_name, shared_library=shared_library, ), - native_build_plan=native_build_plan or NativeBuildPlan(), + native_build_plan=resolved_native_build_plan, ) @@ -555,6 +583,11 @@ def _select_wrapper_plan_route( selection_reason="legacy route forced for migration rollback or comparison", ) if not support_report.supported: + if force_wrapper_plan: + details = "; ".join(f"{item.owner_path}: {item.reason}" for item in support_report.blockers) + raise ValueError( + f"cannot force wrapper-plan route for unsupported generation unit {module.name!r}: {details}" + ) return WrapperPlanRouteDecision( owner_path=module.name, selected_route="legacy", @@ -573,31 +606,31 @@ def _select_wrapper_plan_route( rollout_evidence=_WRAPPER_PLAN_EVIDENCE, selection_reason=f"{mode} mode remains on the legacy route", ) - if not frozenset(support_report.covered_lanes) <= _WRAPPER_PLAN_COMPLETED_LANES: + if force_wrapper_plan: return WrapperPlanRouteDecision( owner_path=module.name, - selected_route="legacy", + selected_route="wrapper-plan", support_report=support_report, rollout_eligible=False, rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="covered lanes exceed the recorded wrapper-plan parity evidence", + selection_reason="wrapper-plan route forced for internal migration verification", ) - if force_wrapper_plan: + if not frozenset(support_report.covered_lanes) <= _WRAPPER_PLAN_COMPLETED_LANES: return WrapperPlanRouteDecision( owner_path=module.name, - selected_route="wrapper-plan", + selected_route="legacy", support_report=support_report, rollout_eligible=False, rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="wrapper-plan route forced for internal migration verification", + selection_reason="covered lanes exceed the recorded wrapper-plan parity evidence", ) return WrapperPlanRouteDecision( owner_path=module.name, - selected_route="legacy", + selected_route="wrapper-plan", support_report=support_report, - rollout_eligible=False, + rollout_eligible=True, rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason=_WRAPPER_PLAN_PRODUCTION_DEFERRED_REASON, + selection_reason="whole generation unit is covered by completed wrapper-plan lanes", ) @@ -614,8 +647,10 @@ def _generated_wrapper_plan_artifacts( strict_wrapper_names: bool, force_legacy: bool, force_wrapper_plan: bool, + verbose: bool | int = False, ) -> RenderedGeneratedWrapperArtifacts | None: """Complete policy and generate selected wrapper-plan artifacts, if chosen.""" + creation_started = time.perf_counter() complete_semantic_policies(module) decision = _select_wrapper_plan_route( module, @@ -624,7 +659,11 @@ def _generated_wrapper_plan_artifacts( force_legacy=force_legacy, force_wrapper_plan=force_wrapper_plan, ) - return _render_selected_wrapper_plan(module) if decision.uses_wrapper_plan else None + if not decision.uses_wrapper_plan: + return None + rendered = _render_selected_wrapper_plan(module) + _print_verbose_timing(verbose, "Wrapper creation", time.perf_counter() - creation_started) + return rendered def _source_compile_object( @@ -1374,6 +1413,37 @@ def _write_build_manifest(path: Path, manifest: dict[str, object]) -> Path: return path +def _with_pyi_manifest( + result: WrapperBuildResult, + *, + bundle: _PyiContractBundle, + strict_wrapper_names: bool, + requested_output_name: str | None, + native_fortran_flags: tuple[str, ...], + wrapper_compiler_debug: bool, + wrapper_fortran_flags: tuple[str, ...], + wrapper_c_flags: tuple[str, ...], + native_array_build_requirements: NativeArrayBuildRequirements, +) -> WrapperBuildResult: + """Attach the standard in-memory `.pyi` build manifest to a plan result.""" + manifest = _pyi_build_manifest( + bundle=bundle, + module_name=result.module_name, + output_dir=result.output_dir, + shared_library=result.shared_library, + strict_wrapper_names=strict_wrapper_names, + requested_output_name=requested_output_name, + native_fortran_flags=native_fortran_flags, + wrapper_compiler_debug=wrapper_compiler_debug, + wrapper_fortran_flags=wrapper_fortran_flags, + wrapper_c_flags=wrapper_c_flags, + native_build_plan=result.native_build_plan, + native_array_build_requirements=native_array_build_requirements, + manifest_dir=result.output_dir, + ) + return replace(result, manifest=manifest) + + def _load_build_manifest(path: str | Path) -> tuple[Path, dict[str, object]]: manifest_path = Path(path) if not manifest_path.is_file(): @@ -1788,6 +1858,7 @@ def build_fortran_extension( strict_wrapper_names=strict_wrapper_names, force_legacy=_force_legacy_wrapper_route, force_wrapper_plan=_force_wrapper_plan_route, + verbose=verbose, ) wrapper_fortran_flags = _compiler_flags(wrapper_fortran_flags) @@ -1957,6 +2028,7 @@ def build_pyi_extension( strict_wrapper_names=strict_wrapper_names, force_legacy=_force_legacy_wrapper_route, force_wrapper_plan=_force_wrapper_plan_route, + verbose=verbose, ) include_dirs = _pyi_native_include_dirs(native_inputs, output_path=output_path) @@ -1987,8 +2059,9 @@ def build_pyi_extension( verbose=verbose, ) + native_array_build_requirements = native_array_handle_build_requirements(module) if rendered_wrapper_plan is not None: - return _build_rendered_wrapper_extension( + result = _build_rendered_wrapper_extension( rendered_wrapper_plan, output_dir=output_path, shared_library_output_dir=shared_library_output_path, @@ -2001,8 +2074,18 @@ def build_pyi_extension( compiler=compiler, verbose=verbose, ) + return _with_pyi_manifest( + result, + bundle=bundle, + strict_wrapper_names=strict_wrapper_names, + requested_output_name=output_name, + native_fortran_flags=native_inputs.source_flags, + wrapper_compiler_debug=wrapper_compiler_debug, + wrapper_fortran_flags=wrapper_fortran_flags, + wrapper_c_flags=wrapper_c_flags, + native_array_build_requirements=native_array_build_requirements, + ) - native_array_build_requirements = native_array_handle_build_requirements(module) scope = Scope( name=module.name, scope_type="module", diff --git a/x2py/semantics/ir2ast.py b/x2py/semantics/ir2ast.py index 894331509..93f11424e 100644 --- a/x2py/semantics/ir2ast.py +++ b/x2py/semantics/ir2ast.py @@ -55,6 +55,7 @@ from x2py.semantics.native_array_handles import array_interop_policy, native_array_descriptor_kind from x2py.semantics.native_contract import NATIVE_CONTRACT_PREPARED_METADATA from x2py.semantics.pyi_metadata import PYI_LOADED_METADATA +from x2py.semantics.wrapper_policy import NativeStatusErrorPolicy from x2py.utilities.visitor import ClassVisitor @@ -752,54 +753,6 @@ def _raise_for_unsupported_polymorphic_contracts( ) -def _is_scalar_integer_runtime_status(semantic_type: models.SemanticType) -> bool: - if semantic_type.rank != 0: - return False - try: - numpy_dtype = SEMANTIC_DTYPE_TO_NUMPY_DTYPE[semantic_type.dtype] - return bool(np.issubdtype(np.dtype(_numpy_type(numpy_dtype)), np.integer)) - except (AttributeError, KeyError, TypeError): - return False - - -def _raise_for_invalid_runtime_policy(node: models.SemanticFunction) -> None: - policy = node.metadata.get(models.RUNTIME_STATUS_ERROR_METADATA) - if policy is None: - return - if not isinstance(policy, dict): - raise ValueError(f"Function {node.name!r} has invalid raises metadata") - - success = policy.get("success", 0) - if not isinstance(success, int) or isinstance(success, bool): - raise ValueError(f"Function {node.name!r} raises success value must be an integer") - - hidden_names = { - mapping.native_name or mapping.python_name - for mapping in node.projection - if mapping.result_position is not None and mapping.python_position is None - } - hidden_outputs = {argument.name: argument for argument in node.arguments if argument.name in hidden_names} - status_name = policy.get("status") - status = hidden_outputs.get(status_name) if isinstance(status_name, str) else None - if status is None: - raise ValueError(f"Function {node.name!r} raises status target must name a hidden output") - if not _is_scalar_integer_runtime_status(status.semantic_type): - raise ValueError( - f"Function {node.name!r} raises status target {status.name!r} must be a scalar integer hidden output" - ) - - message_name = policy.get("message") - if message_name is None: - return - message = hidden_outputs.get(message_name) if isinstance(message_name, str) else None - if message is None: - raise ValueError(f"Function {node.name!r} raises message target must name a hidden output") - if message.semantic_type.rank != 0 or message.semantic_type.name != "String": - raise ValueError( - f"Function {node.name!r} raises message target {message.name!r} must be a scalar string hidden output" - ) - - def _is_bind_c_derived_type( semantic_type: models.SemanticType, class_lookup: dict[str, models.SemanticClass], @@ -1065,8 +1018,15 @@ def _semantic_function_decorators(node): decorators[NATIVE_PROJECTION_METADATA] = True if node.metadata.get(models.RUNTIME_HOLD_GIL_METADATA): decorators[models.RUNTIME_HOLD_GIL_METADATA] = True - if isinstance(status_policy := node.metadata.get(models.RUNTIME_STATUS_ERROR_METADATA), dict): - decorators[models.RUNTIME_STATUS_ERROR_METADATA] = dict(status_policy) + raw_status_policy = node.metadata.get(models.RUNTIME_STATUS_ERROR_METADATA) + if raw_status_policy is not None: + status_policy = node.metadata.get(models.RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA) + if not isinstance(status_policy, NativeStatusErrorPolicy): + raise ValueError( + f"Function {node.name!r} is missing completed native status error policy; " + "run complete_semantic_policies before ir2ast lowering" + ) + decorators[models.RUNTIME_STATUS_ERROR_METADATA] = status_policy return decorators @@ -1510,7 +1470,6 @@ def _visit_ProcedureOverloadSet(self, node): return overload_set def _visit_SemanticFunction(self, node): - _raise_for_invalid_runtime_policy(node) _raise_for_unsupported_bind_c_abi(node, self.class_lookup or {}) _raise_for_unsupported_pointer_outputs(node) _raise_for_blocked_ownership_contracts_in_function(node) diff --git a/x2py/semantics/models.py b/x2py/semantics/models.py index 3a03cd8e6..c83b4b3ae 100644 --- a/x2py/semantics/models.py +++ b/x2py/semantics/models.py @@ -16,6 +16,7 @@ RUNTIME_HOLD_GIL_METADATA = "runtime_hold_gil" RUNTIME_RETAIN_RESULT_OWNER_METADATA = "runtime_retain_result_owner" RUNTIME_STATUS_ERROR_METADATA = "runtime_status_error" +RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA = "resolved_runtime_status_error_policy" # ============================================================ diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index 2730e7803..bbf22b06c 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -8,6 +8,7 @@ from x2py.types.numpy import SEMANTIC_SCALAR_TYPE_NAMES from x2py.semantics.ownership import ( + CodegenAction, DestructionPolicy, OWNERSHIP_POLICY_METADATA, POINTER_POLICY_METADATA, @@ -31,6 +32,9 @@ from x2py.semantics import models from x2py.semantics.native_array_handles import NativeArrayHandlePolicy, native_array_descriptor_kind from x2py.semantics.wrapper_policy import ( + NativeStatusErrorPolicy, + NativeStatusOutputPolicy, + PythonExceptionKind, build_module_variable_policy, build_function_wrapper_policy, ) @@ -203,12 +207,132 @@ def _complete_function(function: models.SemanticFunction, owner_path: str) -> No else: function.metadata.pop(models.RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA, None) function.metadata.pop(models.RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA, None) + _complete_native_status_error_policy(function, owner_path) function.metadata[models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] = build_function_wrapper_policy( function, owner_path=owner_path, ) +def _complete_native_status_error_policy(function: models.SemanticFunction, owner_path: str) -> None: + """Validate and complete one native-status-to-Python-exception decision.""" + raw_policy = function.metadata.get(models.RUNTIME_STATUS_ERROR_METADATA) + if raw_policy is None: + function.metadata.pop(models.RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA, None) + return + if not isinstance(raw_policy, dict): + raise ValueError(f"Function {function.name!r} has invalid raises metadata") + + success = raw_policy.get("success", 0) + if not isinstance(success, int) or isinstance(success, bool): + raise ValueError(f"Function {function.name!r} raises success value must be an integer") + status = _native_status_output(function, owner_path, raw_policy.get("status"), subject="status") + if status.rank != 0 or not _is_scalar_integer_status(status.semantic_type_name): + raise ValueError( + f"Function {function.name!r} raises status target {status.name!r} must be a scalar integer hidden output" + ) + + message_name = raw_policy.get("message") + message = None + if message_name is not None: + message = _native_status_output(function, owner_path, message_name, subject="message") + if message.rank != 0 or message.semantic_type_name != "String": + raise ValueError( + f"Function {function.name!r} raises message target {message.name!r} " + "must be a scalar string hidden output" + ) + if message.owner_path == status.owner_path: + raise ValueError(f"Function {function.name!r} raises status and message targets must be distinct") + + function.metadata[models.RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA] = NativeStatusErrorPolicy( + status=status, + message=message, + success=success, + exception_kind=PythonExceptionKind.RUNTIME_ERROR, + ) + + +def _native_status_output( + function: models.SemanticFunction, + owner_path: str, + output_name: object, + *, + subject: str, +) -> NativeStatusOutputPolicy: + """Return one completed hidden output selected by a runtime policy.""" + if not isinstance(output_name, str) or not output_name: + raise ValueError(f"Function {function.name!r} raises {subject} target must name a hidden output") + mappings = tuple( + mapping + for mapping in function.projection + if ( + mapping.python_position is None + and isinstance(mapping.result_position, int) + and output_name in {mapping.python_name, mapping.native_name} + ) + ) + if len(mappings) != 1: + raise ValueError(f"Function {function.name!r} raises {subject} target must name a hidden output") + mapping = mappings[0] + argument = next((item for item in function.arguments if item.name == mapping.python_name), None) + if argument is None or not isinstance(mapping.native_position, int): + raise ValueError(f"Function {function.name!r} raises {subject} target must name a hidden output") + decision = argument.metadata.get(models.RESOLVED_OWNERSHIP_POLICY_METADATA) + if not isinstance(decision, OwnershipDecision) or not _is_compatible_status_handoff(decision): + raise ValueError( + f"Function {function.name!r} raises {subject} target {output_name!r} " + "has no compatible completed hidden-output handoff" + ) + semantic_type = argument.semantic_type + return NativeStatusOutputPolicy( + owner_path=f"{owner_path}.{argument.name}", + name=argument.name, + native_name=mapping.native_name or argument.name, + native_position=mapping.native_position, + result_position=mapping.result_position, + semantic_type_name=semantic_type.name, + rank=int(semantic_type.rank or 0), + character_length=_fixed_character_length(semantic_type), + ) + + +def _is_compatible_status_handoff(decision: OwnershipDecision) -> bool: + return bool( + decision.projects_result + and not decision.python_visible + and decision.codegen_action is CodegenAction.HIDDEN_OUTPUT + ) + + +def _is_scalar_integer_status(semantic_type_name: str) -> bool: + return semantic_type_name in { + "Byte", + "CEnum", + "Int", + "Int8", + "Int16", + "Int32", + "Int64", + "SizeT", + "UInt", + "UInt8", + "UInt16", + "UInt32", + "UInt64", + } + + +def _fixed_character_length(semantic_type: models.SemanticType) -> int | None: + if semantic_type.rank != 0: + return None + value = semantic_type.metadata.get("fortran_character_length") + if isinstance(value, int) and not isinstance(value, bool) and value > 0: + return value + if isinstance(value, str) and value.strip().isdigit() and int(value.strip()) > 0: + return int(value.strip()) + return None + + def _complete_native_array_handle_result_policy( function: models.SemanticFunction, decision: OwnershipDecision, diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index 40db9feb7..3ec1addd8 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -7,7 +7,6 @@ from enum import Enum from typing import Any -from x2py.types.numpy import SEMANTIC_SCALAR_TYPE_NAMES from x2py.semantics import models from x2py.semantics.wrapper_exports import PythonExportPolicy, completed_python_exports from x2py.semantics.ownership import ( @@ -22,6 +21,21 @@ ) +_PLAN_PRIMITIVE_SCALAR_TYPES = frozenset( + { + "Bool", + "Int8", + "Int16", + "Int32", + "Int64", + "Float32", + "Float64", + "Complex64", + "Complex128", + } +) + + class OptionalMode(str, Enum): """Completed ABI behavior for one argument's presence states.""" @@ -30,6 +44,23 @@ class OptionalMode(str, Enum): DESCRIPTOR = "descriptor" +class ArgumentHandoffMode(str, Enum): + """Completed binding-to-bridge ABI shape for one scalar argument.""" + + VALUE = "value" + TYPED_REFERENCE = "typed_reference" + OPAQUE_ADDRESS = "opaque_address" + + +class BridgeDataAction(str, Enum): + """Completed bridge-side data movement for one boundary value.""" + + DIRECT_TRANSFER = "direct_transfer" + ASSOCIATE_VIEW = "associate_view" + COPY_REPRESENTATION = "copy_representation" + BLOCKED = "blocked" + + class WritebackPhase(str, Enum): """Ordered phases of one completed replacement writeback.""" @@ -47,6 +78,36 @@ class ModuleGetterAction(str, Enum): NULLABLE_SNAPSHOT = "nullable_snapshot" +class PythonExceptionKind(str, Enum): + """Completed Python exception selected for one native failure policy.""" + + RUNTIME_ERROR = "RuntimeError" + + +@dataclass(frozen=True) +class NativeStatusOutputPolicy: + """One validated native output consumed by status-error handling.""" + + owner_path: str + name: str + native_name: str + native_position: int + result_position: int + semantic_type_name: str + rank: int + character_length: int | None = None + + +@dataclass(frozen=True) +class NativeStatusErrorPolicy: + """Completed native-status decision owned by post-IR policy completion.""" + + status: NativeStatusOutputPolicy + message: NativeStatusOutputPolicy | None + success: int + exception_kind: PythonExceptionKind + + @dataclass(frozen=True) class ModuleVariablePolicy: """Completed module-variable behavior before wrapper planning.""" @@ -96,7 +157,11 @@ class ArgumentPolicy: rank: int optional: bool optional_mode: OptionalMode + handoff_mode: ArgumentHandoffMode + bridge_data_action: BridgeDataAction + bridge_copy_reason: str | None nullable: bool + writable: bool descriptor_boundary: bool ownership: OwnershipDecision codegen_action: CodegenAction @@ -122,6 +187,8 @@ class ResultPolicy: native_barrier_action: NativeBarrierAction storage_mode: StorageMode boundary_storage_mode: StorageMode + bridge_data_action: BridgeDataAction + bridge_copy_reason: str | None source_kind: str = "direct_return" native_name: str | None = None native_position: int | None = None @@ -141,9 +208,13 @@ class NativeCallSlotPolicy: value_kind: str native_barrier_action: NativeBarrierAction codegen_action: CodegenAction + bridge_data_action: BridgeDataAction + bridge_copy_reason: str | None literal_type: str | None = None literal_value: Any = None result_position: int | None = None + semantic_type_name: str | None = None + character_length: int | None = None @dataclass(frozen=True) @@ -157,6 +228,7 @@ class FunctionWrapperPolicy: native_module: str | None native_is_subroutine: bool hold_gil: bool + status_error: NativeStatusErrorPolicy | None supported: bool arguments: tuple[ArgumentPolicy, ...] = () result: ResultPolicy | None = None @@ -243,11 +315,22 @@ def build_function_wrapper_policy( """Build typed function policy from completed post-IR decisions.""" argument_native_positions, native_call_slots, slot_blockers = _native_call_slot_policies(function, owner_path) - arguments, argument_blockers = _argument_policies(function, owner_path, argument_native_positions) + arguments, argument_blockers = _argument_policies( + function, + owner_path, + argument_native_positions, + native_call_slots, + ) result, result_blockers = _result_policy(function, owner_path) writeback_actions, lifecycle_blockers = _lifecycle_policies(arguments) + status_error = _completed_native_status_error_policy(function) blockers = ( - _function_shape_blockers(function) + argument_blockers + result_blockers + slot_blockers + lifecycle_blockers + _function_shape_blockers(function) + + argument_blockers + + result_blockers + + slot_blockers + + lifecycle_blockers + + _runtime_status_plan_blockers(status_error) ) return FunctionWrapperPolicy( owner_path=owner_path, @@ -257,6 +340,7 @@ def build_function_wrapper_policy( native_module=_native_module(function, owner_path), native_is_subroutine=_native_is_subroutine(function), hold_gil=bool(function.metadata.get(models.RUNTIME_HOLD_GIL_METADATA)), + status_error=status_error, supported=not blockers, arguments=tuple(arguments), result=result, @@ -270,34 +354,59 @@ def _argument_policies( function: models.SemanticFunction, owner_path: str, argument_native_positions: dict[int, int], + native_call_slots: tuple[NativeCallSlotPolicy, ...], ) -> tuple[list[ArgumentPolicy], tuple[str, ...]]: policies: list[ArgumentPolicy] = [] blockers: list[str] = [] - for python_position, argument in enumerate(function.arguments): + python_position = 0 + for argument in function.arguments: decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) if decision is None: blockers.append(f"argument {argument.name!r} is missing completed ownership policy") continue if decision.projects_result and not decision.python_visible: continue - blockers.extend(_argument_blockers(argument, decision)) - native_position = argument_native_positions.get(python_position) + current_python_position = python_position + python_position += 1 + native_position = argument_native_positions.get(current_python_position) + native_slot = next( + (slot for slot in native_call_slots if slot.python_position == current_python_position), + None, + ) if native_position is None: blockers.append(f"argument {argument.name!r} has no completed native-call slot") native_position = -1 + optional_mode = _optional_mode(argument, decision) + bridge_data_action, bridge_copy_reason = _argument_bridge_data_action( + decision, + optional_mode, + native_slot.value_kind if native_slot is not None else None, + ) + blockers.extend( + _argument_blockers( + argument, + decision, + bridge_data_action, + bridge_copy_reason, + ) + ) policies.append( ArgumentPolicy( owner_path=f"{owner_path}.{argument.name}", name=argument.name, python_name=argument.name, - native_name=_argument_native_name(function, python_position, argument), - python_position=python_position, + native_name=_argument_native_name(function, current_python_position, argument), + python_position=current_python_position, native_position=native_position, semantic_type_name=argument.semantic_type.name, rank=int(argument.semantic_type.rank or 0), optional=argument.optional, - optional_mode=_optional_mode(argument, decision), + optional_mode=optional_mode, + handoff_mode=_argument_handoff_mode(decision), + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, nullable=decision.nullable, + writable=decision.mutates_native, descriptor_boundary=decision.descriptor_boundary, ownership=decision, codegen_action=decision.codegen_action, @@ -307,7 +416,7 @@ def _argument_policies( boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, projects_result=decision.projects_result, python_visible=decision.python_visible, - result_position=_argument_result_position(function, python_position), + result_position=_argument_result_position(function, current_python_position), ) ) return policies, tuple(blockers) @@ -332,6 +441,9 @@ def _result_policy( if not isinstance(decision, OwnershipDecision): return None, ("function result is missing completed ownership policy",) blockers = list(_result_blockers(function.return_type, decision)) + bridge_data_action, bridge_copy_reason = _result_bridge_data_action(function.return_type) + if bridge_data_action is BridgeDataAction.BLOCKED and decision.kind is not ObjectKind.SCALAR: + blockers.append("result has no completed bridge data action") if hidden_results: blockers.append("direct scalar returns cannot share the first Phase 2B lane with hidden scalar outputs") return ( @@ -345,6 +457,8 @@ def _result_policy( native_barrier_action=decision.native_barrier_action, storage_mode=decision.storage_mode, boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, ), tuple(blockers), ) @@ -356,10 +470,13 @@ def _hidden_result_policies( ) -> tuple[tuple[ResultPolicy | None, tuple[str, ...]], ...]: """Return completed policy candidates for hidden scalar output projections.""" policies = [] + suppressed_outputs = _runtime_status_output_owner_paths(function) for argument in function.arguments: decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) if decision is None or not (decision.projects_result and not decision.python_visible): continue + if f"{owner_path}.{argument.name}" in suppressed_outputs: + continue mapping = next( ( item @@ -372,6 +489,9 @@ def _hidden_result_policies( policies.append((None, (f"hidden result {argument.name!r} has no completed return projection",))) continue blockers = _hidden_result_blockers(argument, decision, mapping) + bridge_data_action, bridge_copy_reason = _result_bridge_data_action(argument.semantic_type) + if bridge_data_action is BridgeDataAction.BLOCKED and decision.kind is not ObjectKind.SCALAR: + blockers = (*blockers, f"hidden result {argument.name!r} has no completed bridge data action") policies.append( ( ResultPolicy( @@ -384,6 +504,8 @@ def _hidden_result_policies( native_barrier_action=decision.native_barrier_action, storage_mode=decision.storage_mode, boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, source_kind="hidden_output", native_name=mapping.native_name or argument.name, native_position=mapping.native_position, @@ -411,6 +533,14 @@ def _projected_native_call_slot_policies( slots: list[NativeCallSlotPolicy] = [] blockers: list[str] = [] positions: dict[int, int] = {} + visible_arguments = tuple( + argument + for argument in function.arguments + if ( + (decision := _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA)) is not None + and decision.python_visible + ) + ) for mapping in sorted( function.projection, key=lambda item: item.native_position if item.native_position is not None else -1 ): @@ -437,10 +567,10 @@ def _projected_native_call_slot_policies( if python_position is None: blockers.append(f"native-call slot {native_position} is not a first-lane Python argument projection") continue - if not 0 <= python_position < len(function.arguments): + if not 0 <= python_position < len(visible_arguments): blockers.append(f"native-call slot {native_position} references argument position {python_position}") continue - argument = function.arguments[python_position] + argument = visible_arguments[python_position] decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) if decision is None: blockers.append(f"native-call slot {native_position} references argument without completed policy") @@ -451,6 +581,13 @@ def _projected_native_call_slot_policies( if python_position in positions: blockers.append(f"argument {argument.name!r} appears in more than one native-call slot") positions[python_position] = native_position + bridge_data_action, bridge_copy_reason = _argument_bridge_data_action( + decision, + _optional_mode(argument, decision), + value_kind, + ) + if bridge_data_action is BridgeDataAction.BLOCKED: + blockers.append(f"native-call slot {native_position} has no completed bridge data action") slots.append( NativeCallSlotPolicy( owner_path=f"{owner_path}.{argument.name}", @@ -462,7 +599,11 @@ def _projected_native_call_slot_policies( value_kind=value_kind, native_barrier_action=decision.native_barrier_action, codegen_action=decision.codegen_action, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, result_position=mapping.result_position, + semantic_type_name=argument.semantic_type.name, + character_length=_character_length(argument.semantic_type), ) ) blockers.extend(_native_position_blockers(slot.native_position for slot in slots)) @@ -489,6 +630,8 @@ def _hidden_result_native_call_slot_policy( value_kind=mapping.value_kind, native_barrier_action=NativeBarrierAction.BLOCKED, codegen_action=CodegenAction.BLOCKED, + bridge_data_action=BridgeDataAction.BLOCKED, + bridge_copy_reason=None, result_position=mapping.result_position, ), (f"native-call result slot {native_position} has no hidden argument {mapping.python_name!r}",), @@ -506,10 +649,20 @@ def _hidden_result_native_call_slot_policy( value_kind=mapping.value_kind, native_barrier_action=NativeBarrierAction.BLOCKED, codegen_action=CodegenAction.BLOCKED, + bridge_data_action=BridgeDataAction.BLOCKED, + bridge_copy_reason=None, result_position=mapping.result_position, + semantic_type_name=argument.semantic_type.name, + character_length=_character_length(argument.semantic_type), ), (f"native-call result slot {native_position} references argument without completed policy",), ) + bridge_data_action, bridge_copy_reason = _native_result_bridge_data_action(argument.semantic_type) + blockers = ( + (f"native-call result slot {native_position} has no completed bridge data action",) + if bridge_data_action is BridgeDataAction.BLOCKED + else () + ) return ( NativeCallSlotPolicy( owner_path=f"{owner_path}.{argument.name}", @@ -521,9 +674,13 @@ def _hidden_result_native_call_slot_policy( value_kind=mapping.value_kind, native_barrier_action=decision.native_barrier_action, codegen_action=decision.codegen_action, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, result_position=mapping.result_position, + semantic_type_name=argument.semantic_type.name, + character_length=_character_length(argument.semantic_type), ), - (), + blockers, ) @@ -545,8 +702,11 @@ def _literal_native_call_slot_policy( value_kind="literal", native_barrier_action=NativeBarrierAction.PASS_VALUE, codegen_action=CodegenAction.DIRECT_VALUE, + bridge_data_action=BridgeDataAction.DIRECT_TRANSFER, + bridge_copy_reason=None, literal_type=literal_type, literal_value=literal_value, + semantic_type_name=literal_type, ), tuple(blockers), ) @@ -584,6 +744,13 @@ def _implicit_native_call_slot_policies( if decision is None: blockers.append(f"implicit native-call slot {position} references argument without completed policy") continue + bridge_data_action, bridge_copy_reason = _argument_bridge_data_action( + decision, + _optional_mode(argument, decision), + "arg", + ) + if bridge_data_action is BridgeDataAction.BLOCKED: + blockers.append(f"implicit native-call slot {position} has no completed bridge data action") positions[position] = position slots.append( NativeCallSlotPolicy( @@ -596,12 +763,34 @@ def _implicit_native_call_slot_policies( value_kind="arg", native_barrier_action=decision.native_barrier_action, codegen_action=decision.codegen_action, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, + semantic_type_name=argument.semantic_type.name, ) ) return positions, tuple(slots), tuple(blockers) -def _argument_blockers(argument: models.SemanticArgument, decision: OwnershipDecision) -> tuple[str, ...]: +def _argument_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, + bridge_data_action: BridgeDataAction, + bridge_copy_reason: str | None, +) -> tuple[str, ...]: + """Return scalar-lane blockers without reconstructing policy in a backend.""" + return ( + *_argument_shape_blockers(argument, decision), + *_argument_boundary_blockers(argument, decision), + *_argument_bridge_data_blockers(argument, bridge_data_action, bridge_copy_reason), + *_argument_projection_blockers(argument, decision), + ) + + +def _argument_shape_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Return ownership, type, and visibility blockers for one argument.""" blockers: list[str] = [] if decision.is_blocked: blockers.append( @@ -613,27 +802,77 @@ def _argument_blockers(argument: models.SemanticArgument, decision: OwnershipDec blockers.append(f"argument {argument.name!r} is not Python-visible") if decision.kind is not ObjectKind.SCALAR: blockers.append(f"argument {argument.name!r} policy kind is {decision.kind.value}, not scalar") - if decision.python_barrier_action is not PythonBarrierAction.SCALAR_VALUE: + return tuple(blockers) + + +def _argument_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Return Python/native boundary-action blockers for one argument.""" + blockers: list[str] = [] + if decision.python_barrier_action not in { + PythonBarrierAction.SCALAR_VALUE, + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: blockers.append( - f"argument {argument.name!r} Python action is {decision.python_barrier_action.value}, not scalar_value" + f"argument {argument.name!r} has unsupported scalar Python action {decision.python_barrier_action.value}" ) if decision.native_barrier_action not in { NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + NativeBarrierAction.PASS_RAW_ADDRESS, NativeBarrierAction.PASS_STORAGE_ADDRESS, NativeBarrierAction.PASS_VALUE, }: blockers.append( f"argument {argument.name!r} native action is {decision.native_barrier_action.value}, " - "not pass_call_local_address, pass_storage_address, or pass_value" + "not a supported scalar handoff" ) + if ( + decision.python_barrier_action is PythonBarrierAction.SCALAR_STORAGE + and decision.native_barrier_action is not NativeBarrierAction.PASS_STORAGE_ADDRESS + ): + blockers.append(f"argument {argument.name!r} scalar storage does not use its storage address") + if ( + decision.python_barrier_action is PythonBarrierAction.RAW_ADDRESS + and decision.native_barrier_action is not NativeBarrierAction.PASS_RAW_ADDRESS + ): + blockers.append(f"argument {argument.name!r} raw address is not forwarded as a raw address") + if argument.optional and decision.python_barrier_action is not PythonBarrierAction.SCALAR_VALUE: + blockers.append(f"argument {argument.name!r} optional storage/address boundaries are not supported") + return tuple(blockers) + + +def _argument_bridge_data_blockers( + argument: models.SemanticArgument, + bridge_data_action: BridgeDataAction, + bridge_copy_reason: str | None, +) -> tuple[str, ...]: + """Return incomplete or contradictory bridge data-action blockers.""" + blockers: list[str] = [] + if bridge_data_action is BridgeDataAction.BLOCKED: + blockers.append(f"argument {argument.name!r} has no completed bridge data action") + if bridge_data_action is BridgeDataAction.COPY_REPRESENTATION and not bridge_copy_reason: + blockers.append(f"argument {argument.name!r} bridge representation copy has no completed reason") + if bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION and bridge_copy_reason is not None: + blockers.append(f"argument {argument.name!r} copy-free bridge action carries a copy reason") + return tuple(blockers) + + +def _argument_projection_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Return projected-result action blockers for one argument.""" if decision.projects_result and decision.codegen_action not in { CodegenAction.COPY_IN_OUT, CodegenAction.IN_PLACE_ARGUMENT, }: - blockers.append( - f"argument {argument.name!r} projects a result with unsupported action {decision.codegen_action.value}" + return ( + f"argument {argument.name!r} projects a result with unsupported action {decision.codegen_action.value}", ) - return tuple(blockers) + return () def _result_blockers(semantic_type: models.SemanticType, decision: OwnershipDecision) -> tuple[str, ...]: @@ -705,6 +944,44 @@ def _function_shape_blockers(function: models.SemanticFunction) -> tuple[str, .. return tuple(blockers) +def _completed_native_status_error_policy( + function: models.SemanticFunction, +) -> NativeStatusErrorPolicy | None: + policy = function.metadata.get(models.RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA) + return policy if isinstance(policy, NativeStatusErrorPolicy) else None + + +def _runtime_status_output_owner_paths(function: models.SemanticFunction) -> frozenset[str]: + policy = _completed_native_status_error_policy(function) + if policy is None: + return frozenset() + outputs = [policy.status.owner_path] + if policy.message is not None: + outputs.append(policy.message.owner_path) + return frozenset(outputs) + + +def _runtime_status_plan_blockers(policy: NativeStatusErrorPolicy | None) -> tuple[str, ...]: + """Return Phase 2D backend blockers after semantic validity is complete.""" + if policy is None: + return () + blockers = [] + if policy.status.semantic_type_name != "Int32": + blockers.append("native status error projection requires an Int32 status in the current plan lane") + if policy.message is not None and policy.message.character_length is None: + blockers.append("native status error message requires a fixed positive character length") + return tuple(blockers) + + +def _character_length(semantic_type: models.SemanticType) -> int | None: + value = semantic_type.metadata.get("fortran_character_length") + if isinstance(value, int) and not isinstance(value, bool) and value > 0: + return value + if isinstance(value, str) and value.strip().isdigit() and int(value.strip()) > 0: + return int(value.strip()) + return None + + def _lifecycle_policies( arguments: list[ArgumentPolicy], ) -> tuple[tuple[LifecyclePolicy, ...], tuple[str, ...]]: @@ -750,7 +1027,7 @@ def _is_first_lane_scalar_type(semantic_type: models.SemanticType) -> bool: return bool( int(semantic_type.rank or 0) == 0 and semantic_type.name != "String" - and scalar_name in SEMANTIC_SCALAR_TYPE_NAMES + and scalar_name in _PLAN_PRIMITIVE_SCALAR_TYPES ) @@ -890,6 +1167,72 @@ def _optional_mode( return OptionalMode.NULLABLE_VALUE +def _argument_bridge_data_action( + decision: OwnershipDecision, + optional_mode: OptionalMode, + value_kind: str | None, +) -> tuple[BridgeDataAction, str | None]: + """Complete whether the bridge reuses, views, or copies one input payload.""" + if optional_mode is OptionalMode.DESCRIPTOR: + if value_kind == "pointer": + return BridgeDataAction.ASSOCIATE_VIEW, None + if value_kind == "allocatable": + return ( + BridgeDataAction.COPY_REPRESENTATION, + "materialize owned Fortran allocatable scalar storage from the binding value", + ) + return BridgeDataAction.BLOCKED, None + if optional_mode is OptionalMode.NULLABLE_VALUE: + return BridgeDataAction.ASSOCIATE_VIEW, None + if decision.python_barrier_action in { + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: + return BridgeDataAction.ASSOCIATE_VIEW, None + if decision.python_barrier_action is PythonBarrierAction.SCALAR_VALUE and decision.native_barrier_action in { + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + NativeBarrierAction.PASS_STORAGE_ADDRESS, + NativeBarrierAction.PASS_VALUE, + }: + return BridgeDataAction.DIRECT_TRANSFER, None + return BridgeDataAction.BLOCKED, None + + +def _result_bridge_data_action( + semantic_type: models.SemanticType, +) -> tuple[BridgeDataAction, str | None]: + """Complete direct result transfer without widening unsupported lanes.""" + if _is_first_lane_scalar_type(semantic_type): + return BridgeDataAction.DIRECT_TRANSFER, None + return BridgeDataAction.BLOCKED, None + + +def _native_result_bridge_data_action( + semantic_type: models.SemanticType, +) -> tuple[BridgeDataAction, str | None]: + """Complete bridge data movement for one hidden native output slot.""" + if _is_first_lane_scalar_type(semantic_type): + return BridgeDataAction.DIRECT_TRANSFER, None + if semantic_type.name == "String" and _character_length(semantic_type) is not None: + return ( + BridgeDataAction.COPY_REPRESENTATION, + "copy fixed-length Fortran character output into C-owned null-terminated storage", + ) + return BridgeDataAction.BLOCKED, None + + +def _argument_handoff_mode(decision: OwnershipDecision) -> ArgumentHandoffMode: + """Return the completed scalar ABI shape consumed by both backends.""" + if decision.python_barrier_action in { + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: + return ArgumentHandoffMode.OPAQUE_ADDRESS + if decision.native_barrier_action is NativeBarrierAction.PASS_VALUE: + return ArgumentHandoffMode.VALUE + return ArgumentHandoffMode.TYPED_REFERENCE + + def _argument_result_position(function: models.SemanticFunction, python_position: int) -> int | None: """Return one visible argument's completed projected result position.""" for mapping in function.projection: diff --git a/x2py/wrapper_codegen/__init__.py b/x2py/wrapper_codegen/__init__.py index 4e9dcd3b3..2b313c897 100644 --- a/x2py/wrapper_codegen/__init__.py +++ b/x2py/wrapper_codegen/__init__.py @@ -7,6 +7,8 @@ from .generator import WrapperCodeGenerator from .nodes import ( BackendScalarType, + CAllowThreadsBegin, + CAllowThreadsEnd, CDeclaration, CExpressionStatement, CFunction, @@ -44,6 +46,7 @@ BindingModulePlan, BindingModuleVariablePlan, BindingResultPlan, + BindingStatusErrorPlan, BridgeArgumentPlan, BridgeFunctionPlan, BridgeLifecyclePlan, @@ -77,12 +80,15 @@ "BindingModulePlan", "BindingModuleVariablePlan", "BindingResultPlan", + "BindingStatusErrorPlan", "BridgeArgumentPlan", "BridgeFunctionPlan", "BridgeLifecyclePlan", "BridgeModulePlan", "BridgeModuleVariablePlan", "BridgeResultPlan", + "CAllowThreadsBegin", + "CAllowThreadsEnd", "CBindingGenerator", "CDeclaration", "CExpressionStatement", diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index a05f0d20f..e3ab63ee5 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -6,16 +6,20 @@ from x2py.semantics.ownership import ( CodegenAction, - NativeBarrierAction, PythonBarrierAction, SetterAction, ) from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + BridgeDataAction, ModuleGetterAction, OptionalMode, + PythonExceptionKind, WritebackPhase, ) from x2py.wrapper_codegen.nodes import ( + CAllowThreadsBegin, + CAllowThreadsEnd, CDeclaration, CExpressionStatement, CFunction, @@ -42,6 +46,7 @@ ModulePlan, ModuleVariablePlan, NamespacePlan, + NativeCallSlotPlan, ResultPlan, ) from x2py.wrapper_codegen.primitive_scalar_types import PrimitiveScalarTypeRegistry @@ -59,6 +64,7 @@ class _CArgumentNames: @dataclass class _CFunctionContext: arguments: dict[str, _CArgumentNames] + native_outputs: dict[str, str] result_name: str | None python_result_name: str | None @@ -69,26 +75,74 @@ class CBindingGenerator(ClassVisitor): def require_supported(self, plan: ModulePlan) -> None: """Reject unsupported C ABI actions and scalar types.""" for function in self._functions(plan): - for argument in function.arguments: - if argument.binding.python_action is not PythonBarrierAction.SCALAR_VALUE: - raise ValueError( - f"Unsupported C argument action for {argument.owner_path!r}: {argument.binding.python_action!r}" - ) - PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) - if function.result is not None: - PrimitiveScalarTypeRegistry.type_for(function.result.semantic_type_name) - for action in function.writeback_actions: - if action.phase is not WritebackPhase.COPY_OUT or action.binding is None: - continue - PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) + self._require_function_supported(function) for variable in self._variables(plan): PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) + def _require_function_supported(self, function: FunctionPlan) -> None: + """Reject unsupported actions and types for one binding function.""" + for argument in function.arguments: + self._require_argument_supported(argument) + if function.result is not None: + PrimitiveScalarTypeRegistry.type_for(function.result.semantic_type_name) + for slot in function.native_call_slots: + self._require_native_result_supported(function, slot) + for action in function.writeback_actions: + self._require_writeback_supported(action) + + def _require_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Reject one unsupported Python argument conversion.""" + if argument.binding.python_action not in { + PythonBarrierAction.SCALAR_VALUE, + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: + raise ValueError( + f"Unsupported C argument action for {argument.owner_path!r}: {argument.binding.python_action!r}" + ) + if ( + argument.binding.python_action in {PythonBarrierAction.SCALAR_STORAGE, PythonBarrierAction.RAW_ADDRESS} + and argument.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS + ): + raise ValueError(f"Unsupported C address handoff for {argument.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_native_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Reject one unsupported native result output.""" + if slot.source_kind != "result": + return + if slot.datatype_family is DatatypeFamily.STRING: + self._require_string_result_supported(function, slot) + return + if slot.semantic_type_name is None: + raise ValueError(f"Missing C result datatype for {slot.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + + def _require_string_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Require fixed-length status-message storage for one string result.""" + policy = function.binding.status_error + if policy is None: + raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") + if policy.message_role != slot.symbolic_role: + raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") + if slot.character_length is None: + raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") + if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported C string bridge data action for {slot.owner_path!r}") + + def _require_writeback_supported(self, action: LifecycleActionPlan) -> None: + """Require a scalar type for one binding-owned copy-out action.""" + if action.phase is not WritebackPhase.COPY_OUT: + return + if action.binding is None: + return + PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) + def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: """Return a complete C module and header from one shared plan.""" functions = tuple(function for namespace in plan.namespaces for function in self.visit(namespace)) needs_runtime = self.requires_runtime_support(plan) - needs_free = self._module_needs_snapshot_allocator(plan) + needs_free = self._module_needs_allocator(plan) c_module = CModule( name=f"{plan.binding.owner_path}_wrapper", defines=self._module_defines(needs_runtime), @@ -120,10 +174,15 @@ def requires_runtime_support(self, plan: ModulePlan) -> bool: function.arguments or function.result is not None for function in self._functions(plan) ) - def _module_needs_snapshot_allocator(self, plan: ModulePlan) -> bool: - """Return whether nullable module snapshots need the shared allocator.""" + def _module_needs_allocator(self, plan: ModulePlan) -> bool: + """Return whether emitted bridge-owned copies need the shared allocator.""" return any( variable.binding.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT for variable in self._variables(plan) + ) or any( + slot.datatype_family is DatatypeFamily.STRING + for function in self._functions(plan) + for slot in function.native_call_slots + if slot.source_kind == "result" ) def _module_defines(self, needs_runtime: bool) -> tuple[CMacroDefinition, ...]: @@ -364,7 +423,8 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> CFunction: body=( self._keyword_declaration(plan), *(node for node in argument_nodes if isinstance(node, CDeclaration)), - *(self._result_declaration(plan, context)), + *self._direct_result_declaration(plan, context), + *self._native_output_declarations(plan, context), self._parse_statement(plan, context), *(node for node in argument_nodes if not isinstance(node, CDeclaration)), *output_nodes, @@ -401,7 +461,23 @@ def _lower_argument_required( plan: ArgumentTransferPlan, context: _CFunctionContext, ) -> tuple[CDeclaration | CExpressionStatement, ...]: - """Return declarations and conversion statements for one scalar input.""" + """Dispatch one required argument from its completed Python action.""" + action = plan.binding.python_action + match action: + case PythonBarrierAction.SCALAR_VALUE: + return self._lower_argument_required_scalar_value(plan, context) + case PythonBarrierAction.SCALAR_STORAGE: + return self._lower_argument_required_scalar_storage(plan, context) + case PythonBarrierAction.RAW_ADDRESS: + return self._lower_argument_required_raw_address(plan, context) + raise ValueError(f"Unsupported required C argument action for {plan.owner_path!r}: {action!r}") + + def _lower_argument_required_scalar_value( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Return declarations and conversion statements for one scalar value.""" scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) if scalar_type.python_input_converter is None or scalar_type.python_input_check is None: raise ValueError(f"Unsupported scalar input type {plan.semantic_type_name!r}") @@ -423,6 +499,80 @@ def _lower_argument_required( CExpressionStatement(CodeExpression("if (PyErr_Occurred()) return NULL")), ) + def _lower_argument_required_scalar_storage( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Validate and borrow one rank-zero NumPy scalar data address.""" + scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) + if scalar_type.numpy_type_macro is None: + raise ValueError(f"Unsupported scalar storage type {plan.semantic_type_name!r}") + names = context.arguments[plan.owner_path] + array = f"(PyArrayObject *){names.object_name}" + expected = scalar_type.python_type_name + nodes = [ + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(names.value_name, "void *", CodeExpression("NULL")), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_Check({names.object_name}) || PyArray_TYPE({array}) != " + f"{scalar_type.numpy_type_macro} || PyArray_NDIM({array}) != 0) {{ " + f'PyErr_Format(PyExc_TypeError, "Expected a rank-zero numpy.ndarray of type ' + f"{expected} for argument {plan.binding.python_name}. Received \", " + f"Py_TYPE({names.object_name})->tp_name); return NULL; }}" + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISNOTSWAPPED({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must use native ' + 'byte order"); return NULL; }' + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISALIGNED({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must be aligned"); ' + "return NULL; }" + ) + ), + ] + if plan.binding.writable: + nodes.append( + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISWRITEABLE({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must be writeable"); ' + "return NULL; }" + ) + ) + ) + nodes.append(CExpressionStatement(CodeExpression(f"{names.value_name} = PyArray_DATA({array})"))) + return tuple(nodes) + + def _lower_argument_required_raw_address( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Convert one Python integer into a caller-owned raw address.""" + names = context.arguments[plan.owner_path] + return ( + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(names.value_name, "void *", CodeExpression("NULL")), + CExpressionStatement( + CodeExpression( + f"if (!PyLong_Check({names.object_name})) {{ " + f'PyErr_Format(PyExc_TypeError, "Expected an integer raw address for argument ' + f"{plan.binding.python_name}. Received \", " + f"Py_TYPE({names.object_name})->tp_name); return NULL; }}" + ) + ), + CExpressionStatement(CodeExpression(f"{names.value_name} = PyLong_AsVoidPtr({names.object_name})")), + CExpressionStatement(CodeExpression(f"if ({names.value_name} == NULL && PyErr_Occurred()) return NULL")), + ) + def _lower_argument_nullable_value( self, plan: ArgumentTransferPlan, @@ -495,50 +645,45 @@ def _visit_ResultPlan( self, plan: ResultPlan, *, - function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: """Lower one result through its completed binding action.""" - return self._lower_result(plan, function, context) + return self._lower_result(plan, context) def _lower_result( self, plan: ResultPlan, - function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: """Dispatch one completed binding result action explicitly.""" action = plan.binding.codegen_action match action: case CodegenAction.DIRECT_VALUE: - return self._lower_result_direct_value(plan, function, context) + return self._lower_result_direct_value(plan, context) case CodegenAction.HIDDEN_OUTPUT: - return self._lower_result_hidden_output(plan, function, context) + return self._lower_result_hidden_output(plan, context) raise ValueError(f"Unsupported C result action for {plan.owner_path!r}: {action!r}") def _lower_result_direct_value( self, plan: ResultPlan, - function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - return self._lower_result_value(plan, function, context) + return self._lower_result_value(plan, context) def _lower_result_hidden_output( self, plan: ResultPlan, - function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - return self._lower_result_value(plan, function, context) + return self._lower_result_value(plan, context) def _lower_result_value( self, plan: ResultPlan, - function: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return the bridge call and Python scalar result projection.""" + """Return Python scalar projection after the completed native envelope.""" scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) if ( scalar_type.python_result_converter is None @@ -546,14 +691,7 @@ def _lower_result_value( or context.python_result_name is None ): raise ValueError(f"Unsupported scalar result type {plan.semantic_type_name!r}") - call = self._bridge_call(function, context) - call_statement = ( - CExpressionStatement(CodeExpression(f"{context.result_name} = {call}")) - if plan.source_kind == "direct_return" - else CExpressionStatement(CodeExpression(call)) - ) return ( - call_statement, CDeclaration( context.python_result_name, "PyObject *", @@ -568,14 +706,116 @@ def _output_nodes( plan: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return the native call and completed Python output projection.""" + """Return the native envelope, status projection, and Python result.""" + nodes = [ + *self._lower_native_call(plan, self._bridge_call_statement(plan, context)), + *self._lower_status_error(plan, context), + ] if plan.result is not None: - return self.visit(plan.result, function=plan, context=context) - if plan.writeback_actions: - return self._writeback_nodes(plan, context) + nodes.extend(self.visit(plan.result, context=context)) + elif plan.writeback_actions: + nodes.extend(self._writeback_nodes(plan, context)) + else: + nodes.append(CExpressionStatement(CodeExpression("Py_RETURN_NONE"))) + return tuple(nodes) + + def _bridge_call_statement(self, plan: FunctionPlan, context: _CFunctionContext) -> CExpressionStatement: + """Return the mechanical bridge call selected by result storage.""" + call = self._bridge_call(plan, context) + expression = ( + f"{context.result_name} = {call}" + if plan.result is not None and plan.result.source_kind == "direct_return" + else call + ) + return CExpressionStatement(CodeExpression(expression)) + + def _lower_native_call( + self, + plan: FunctionPlan, + call: CExpressionStatement, + ) -> tuple[CAllowThreadsBegin | CAllowThreadsEnd | CExpressionStatement, ...]: + """Dispatch the completed GIL envelope to directly named methods.""" + if plan.binding.hold_gil: + return self._lower_native_call_held(call) + return self._lower_native_call_released(call) + + def _lower_native_call_held(self, call: CExpressionStatement) -> tuple[CExpressionStatement, ...]: + """Emit one native bridge call while retaining the GIL.""" + return (call,) + + def _lower_native_call_released( + self, + call: CExpressionStatement, + ) -> tuple[CAllowThreadsBegin | CExpressionStatement | CAllowThreadsEnd, ...]: + """Release the GIL only for the native bridge call.""" + return (CAllowThreadsBegin(), call, CAllowThreadsEnd()) + + def _lower_status_error( + self, + plan: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: + """Dispatch one completed post-call Python exception action.""" + policy = plan.binding.status_error + if policy is None: + return () + if policy.exception_kind is PythonExceptionKind.RUNTIME_ERROR: + return self._lower_status_error_runtime_error(plan, context) + raise ValueError(f"Unsupported C status exception for {plan.owner_path!r}: {policy.exception_kind!r}") + + def _lower_status_error_runtime_error( + self, + plan: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: + """Raise RuntimeError from completed status/message roles with the GIL held.""" + policy = plan.binding.status_error + status_name = context.native_outputs[policy.status_role] + condition = CodeExpression(f"{status_name} != {policy.success}") + if policy.message_role is None: + return ( + CIf( + condition, + body=( + CExpressionStatement( + CodeExpression( + f'PyErr_Format(PyExc_RuntimeError, "native call failed with status %d != {policy.success}", ' + f"(int){status_name})" + ) + ), + CReturn(CodeExpression("NULL")), + ), + ), + ) + message_name = context.native_outputs[policy.message_role] + message_object = f"{message_name}_obj" return ( - CExpressionStatement(CodeExpression(self._bridge_call(plan, context))), - CExpressionStatement(CodeExpression("Py_RETURN_NONE")), + CIf( + CodeExpression(f"{message_name} == NULL"), + body=( + CExpressionStatement(CodeExpression("PyErr_NoMemory()")), + CReturn(CodeExpression("NULL")), + ), + ), + CDeclaration( + message_object, + "PyObject *", + CodeExpression(f"PyUnicode_FromString((const char *){message_name})"), + ), + CExpressionStatement(CodeExpression(f"free({message_name})")), + CIf( + CodeExpression(f"{message_object} == NULL"), + body=(CReturn(CodeExpression("NULL")),), + ), + CIf( + condition, + body=( + CExpressionStatement(CodeExpression(f"PyErr_SetObject(PyExc_RuntimeError, {message_object})")), + CExpressionStatement(CodeExpression(f"Py_DECREF({message_object})")), + CReturn(CodeExpression("NULL")), + ), + ), + CExpressionStatement(CodeExpression(f"Py_DECREF({message_object})")), ) def _writeback_nodes( @@ -640,7 +880,6 @@ def _lower_writeback_value( scalar_type = PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) python_result_name = context.python_result_name or "result_obj" return ( - CExpressionStatement(CodeExpression(self._bridge_call(plan, context))), CDeclaration( python_result_name, "PyObject *", @@ -667,11 +906,20 @@ def _function_context(self, plan: FunctionPlan) -> _CFunctionContext: ) for argument in plan.arguments } + native_outputs = { + slot.symbolic_role: slot.native_name.lower() + for slot in plan.native_call_slots + if slot.source_kind == "result" + } if plan.result is None: python_result = "result_obj" if plan.writeback_actions else None - return _CFunctionContext(arguments, None, python_result) - base = plan.result.bridge.native_name.lower() if plan.result.source_kind == "hidden_output" else "result" - return _CFunctionContext(arguments, base, "result_obj") + return _CFunctionContext(arguments, native_outputs, None, python_result) + base = ( + native_outputs[plan.result.bridge.native_result_role] + if plan.result.source_kind == "hidden_output" + else "result" + ) + return _CFunctionContext(arguments, native_outputs, base, "result_obj") def _keyword_declaration(self, plan: FunctionPlan) -> CDeclaration: keywords = ", ".join( @@ -692,16 +940,36 @@ def _parse_statement(self, plan: FunctionPlan, context: _CFunctionContext) -> CE CodeExpression(f'if (!PyArg_ParseTupleAndKeywords(args, kwargs, "{units}", kwlist{suffix})) return NULL') ) - def _result_declaration( + def _direct_result_declaration( self, plan: FunctionPlan, context: _CFunctionContext, ) -> tuple[CDeclaration, ...]: - if plan.result is None or context.result_name is None: + if plan.result is None or plan.result.source_kind != "direct_return" or context.result_name is None: return () scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name) return (CDeclaration(context.result_name, scalar_type.c_spelling),) + def _native_output_declarations( + self, + plan: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration, ...]: + """Declare bridge output storage from typed native result slots.""" + declarations = [] + for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position): + if slot.source_kind != "result": + continue + name = context.native_outputs[slot.symbolic_role] + if slot.datatype_family is DatatypeFamily.STRING: + declarations.append(CDeclaration(name, "void *", CodeExpression("NULL"))) + continue + if slot.semantic_type_name is None: + raise ValueError(f"Missing native result datatype for {slot.owner_path!r}") + scalar_type = PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + declarations.append(CDeclaration(name, scalar_type.c_spelling)) + return tuple(declarations) + def _bridge_call(self, plan: FunctionPlan, context: _CFunctionContext) -> str: arguments = [] for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position): @@ -710,37 +978,77 @@ def _bridge_call(self, plan: FunctionPlan, context: _CFunctionContext) -> str: arguments.append(value) if argument.bridge.optional_mode is OptionalMode.DESCRIPTOR: arguments.append(names.present_name) - if plan.result is not None and plan.result.source_kind == "hidden_output": - arguments.append(f"&{context.result_name}") + arguments.extend( + f"&{context.native_outputs[slot.symbolic_role]}" + for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position) + if slot.source_kind == "result" + ) return f"{self._bridge_function_name(plan)}({', '.join(arguments)})" def _bridge_call_argument(self, plan: ArgumentTransferPlan, names: _CArgumentNames) -> str: """Return one binding-to-bridge C argument expression.""" if plan.bridge.optional_mode is not OptionalMode.REQUIRED: return names.nullable_name - if plan.bridge.native_action is not NativeBarrierAction.PASS_VALUE: + if plan.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS: + return names.value_name + if plan.bridge.handoff_mode is ArgumentHandoffMode.TYPED_REFERENCE: return f"&{names.value_name}" return names.value_name def _bridge_prototype(self, plan: FunctionPlan) -> CFunctionPrototype: - return_type = "void" - if plan.result is not None and plan.result.source_kind == "direct_return": - return_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).c_spelling - parameters = [] - for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position): - if argument.bridge.optional_mode is OptionalMode.REQUIRED: - scalar_type = PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).c_spelling - if argument.bridge.native_action is not NativeBarrierAction.PASS_VALUE: - scalar_type = f"{scalar_type} *" - else: - scalar_type = "void *" - parameters.append(CParameter(argument.bridge.native_name.lower(), scalar_type)) - if argument.bridge.optional_mode is OptionalMode.DESCRIPTOR: - parameters.append(CParameter(f"{argument.bridge.native_name.lower()}_present", "void *")) - if plan.result is not None and plan.result.source_kind == "hidden_output": - scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).c_spelling - parameters.append(CParameter(plan.result.bridge.native_name.lower(), f"{scalar_type} *")) - return CFunctionPrototype(self._bridge_function_name(plan), return_type, tuple(parameters)) + argument_parameters = tuple( + parameter + for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) + for parameter in self._bridge_argument_parameters(argument) + ) + result_parameters = tuple( + parameter + for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position) + for parameter in self._bridge_result_parameters(slot) + ) + return CFunctionPrototype( + self._bridge_function_name(plan), + self._bridge_return_type(plan), + (*argument_parameters, *result_parameters), + ) + + def _bridge_return_type(self, plan: FunctionPlan) -> str: + """Return the direct bridge result type, or void for subroutines.""" + if plan.result is None: + return "void" + if plan.result.source_kind != "direct_return": + return "void" + return PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).c_spelling + + def _bridge_argument_parameters(self, argument: ArgumentTransferPlan) -> tuple[CParameter, ...]: + """Return the bridge ABI parameters for one Python argument.""" + name = argument.bridge.native_name.lower() + scalar_type = self._bridge_argument_type(argument) + if argument.bridge.optional_mode is OptionalMode.DESCRIPTOR: + return (CParameter(name, scalar_type), CParameter(f"{name}_present", "void *")) + return (CParameter(name, scalar_type),) + + def _bridge_argument_type(self, argument: ArgumentTransferPlan) -> str: + """Return the C ABI type for one bridge input.""" + if argument.bridge.optional_mode is not OptionalMode.REQUIRED: + return "void *" + if argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS: + return "void *" + scalar_type = PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).c_spelling + if argument.bridge.handoff_mode is ArgumentHandoffMode.TYPED_REFERENCE: + return f"{scalar_type} *" + return scalar_type + + def _bridge_result_parameters(self, slot: NativeCallSlotPlan) -> tuple[CParameter, ...]: + """Return the C ABI parameter for one native result slot.""" + if slot.source_kind != "result": + return () + if slot.datatype_family is DatatypeFamily.STRING: + return (CParameter(slot.native_name.lower(), "void **"),) + if slot.semantic_type_name is None: + raise ValueError(f"Missing bridge result datatype for {slot.owner_path!r}") + scalar_type = PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name).c_spelling + return (CParameter(slot.native_name.lower(), f"{scalar_type} *"),) def _module_variable_bridge_prototypes( self, diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index 70b42c8af..3d9f9bfb7 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -3,7 +3,12 @@ from __future__ import annotations from x2py.semantics.ownership import AssignmentMode, CodegenAction, NativeBarrierAction -from x2py.semantics.wrapper_policy import ModuleGetterAction, OptionalMode +from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + BridgeDataAction, + ModuleGetterAction, + OptionalMode, +) from x2py.wrapper_codegen.nodes import ( CodeExpression, FortranAssignment, @@ -20,10 +25,12 @@ ) from x2py.wrapper_codegen.plan import ( ArgumentTransferPlan, + DatatypeFamily, FunctionPlan, ModulePlan, ModuleVariablePlan, NamespacePlan, + NativeCallSlotPlan, ) from x2py.wrapper_codegen.primitive_scalar_types import PrimitiveScalarTypeRegistry from x2py.wrapper_codegen.visitor import ClassVisitor @@ -41,22 +48,58 @@ def require_supported(self, plan: ModulePlan) -> None: def _require_function_supported(self, function: FunctionPlan) -> None: """Reject unsupported actions in one planned bridge procedure.""" - supported_native_actions = { - NativeBarrierAction.PASS_VALUE, - NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, - NativeBarrierAction.PASS_STORAGE_ADDRESS, - } if self._has_optional_arguments(function) and any( slot.source_kind == "literal" for slot in function.native_call_slots ): raise ValueError(f"{function.owner_path!r} mixes optional scalar arguments with hidden literals") for argument in function.arguments: - if argument.bridge.native_action not in supported_native_actions: - raise ValueError( - f"Unsupported Fortran argument action for {argument.owner_path!r}: " - f"{argument.bridge.native_action!r}" - ) - PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + self._require_argument_supported(argument) + for slot in function.native_call_slots: + self._require_native_result_supported(function, slot) + + def _require_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Reject one unsupported native argument action.""" + supported = { + NativeBarrierAction.PASS_VALUE, + NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, + NativeBarrierAction.PASS_RAW_ADDRESS, + NativeBarrierAction.PASS_STORAGE_ADDRESS, + } + if argument.bridge.native_action not in supported: + raise ValueError( + f"Unsupported Fortran argument action for {argument.owner_path!r}: {argument.bridge.native_action!r}" + ) + if ( + argument.bridge.native_action is NativeBarrierAction.PASS_RAW_ADDRESS + and argument.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS + ): + raise ValueError(f"Unsupported Fortran raw-address handoff for {argument.owner_path!r}") + if argument.bridge.data_action is BridgeDataAction.BLOCKED: + raise ValueError(f"Blocked Fortran bridge data action for {argument.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_native_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Reject one unsupported native result output.""" + if slot.source_kind != "result": + return + if slot.datatype_family is DatatypeFamily.STRING: + self._require_string_result_supported(function, slot) + return + if slot.semantic_type_name is None: + raise ValueError(f"Missing Fortran result datatype for {slot.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + + def _require_string_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Require one fixed-length status-message result.""" + policy = function.binding.status_error + if policy is None: + raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") + if policy.message_role != slot.symbolic_role: + raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") + if slot.character_length is None: + raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") + if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported Fortran string bridge data action for {slot.owner_path!r}") def _require_variable_supported(self, variable: ModuleVariablePlan) -> None: """Reject unsupported actions in one planned module variable.""" @@ -86,7 +129,7 @@ def _visit_ModulePlan(self, plan: ModulePlan) -> FortranModule: FortranUse("iso_c_binding", self._iso_c_symbols()), *self._native_module_uses(plan), ), - interfaces=(*self._external_interfaces(plan), *self._module_snapshot_interfaces(plan)), + interfaces=(*self._external_interfaces(plan), *self._allocator_interfaces(plan)), procedures=tuple(procedure for namespace in plan.namespaces for procedure in self.visit(namespace)), ) @@ -99,13 +142,13 @@ def _visit_NamespacePlan(self, plan: NamespacePlan) -> tuple[FortranFunction, .. def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: """Recursively assemble one complete bridge procedure.""" - result_parameters, result_name, result_type = self._lower_result(plan) + result_name, result_type = self._lower_result(plan) parameters = tuple( parameter for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) for parameter in self.visit(argument) ) - parameters = (*parameters, *result_parameters) + parameters = (*parameters, *self._native_output_parameters(plan)) bridge_name = self._bridge_function_name(plan) is_subroutine = plan.bridge.native_is_subroutine return FortranFunction( @@ -114,15 +157,24 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: result_name=result_name, result_type=result_type, bind_name=bridge_name, - declarations=(*self._external_declarations(plan), *self._optional_declarations(plan)), - body=(*self._descriptor_initializers(plan), *self._function_body(plan, result_name)), + declarations=( + *self._optional_declarations(plan), + *self._opaque_address_declarations(plan), + *self._native_output_declarations(plan), + ), + body=( + *self._descriptor_initializers(plan), + *self._opaque_address_initializers(plan), + *self._function_body(plan, result_name), + *self._native_output_finalizers(plan), + ), is_subroutine=is_subroutine, ) def _lower_result( self, plan: FunctionPlan, - ) -> tuple[tuple[FortranParameter, ...], str | None, str | None]: + ) -> tuple[str | None, str | None]: """Dispatch one completed bridge result action explicitly.""" if plan.result is None: return self._lower_result_none(plan) @@ -137,23 +189,23 @@ def _lower_result( def _lower_result_none( self, _plan: FunctionPlan, - ) -> tuple[tuple[FortranParameter, ...], str | None, str | None]: + ) -> tuple[str | None, str | None]: """Return the procedure shape of a native subroutine with no projection.""" - return (), None, None + return None, None def _lower_result_direct_value( self, plan: FunctionPlan, - ) -> tuple[tuple[FortranParameter, ...], str | None, str | None]: + ) -> tuple[str | None, str | None]: """Return the procedure shape of a direct native function result.""" - return (), "result", self._bridge_result_type(plan) + return "result", self._bridge_result_type(plan) def _lower_result_hidden_output( self, plan: FunctionPlan, - ) -> tuple[tuple[FortranParameter, ...], str | None, str | None]: + ) -> tuple[str | None, str | None]: """Return the procedure shape of a hidden native output parameter.""" - return self._hidden_result_parameters(plan), None, None + return None, None def _visit_ModuleVariablePlan(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: """Lower bridge-owned getter and setter actions into procedures.""" @@ -288,8 +340,35 @@ def _lower_argument(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, raise ValueError(f"Unsupported Fortran argument optional mode for {plan.owner_path!r}: {mode!r}") def _lower_argument_required(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: - attributes = ("value",) if plan.bridge.native_action is NativeBarrierAction.PASS_VALUE else () - return (self._parameter(plan, attributes),) + """Dispatch one required bridge parameter from its completed ABI shape.""" + mode = plan.bridge.handoff_mode + match mode: + case ArgumentHandoffMode.VALUE: + return self._lower_argument_required_value(plan) + case ArgumentHandoffMode.TYPED_REFERENCE: + return self._lower_argument_required_typed_reference(plan) + case ArgumentHandoffMode.OPAQUE_ADDRESS: + return self._lower_argument_required_opaque_address(plan) + raise ValueError(f"Unsupported Fortran argument handoff for {plan.owner_path!r}: {mode!r}") + + def _lower_argument_required_value(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: + """Return one interoperable scalar value parameter.""" + return (self._parameter(plan, ("value",)),) + + def _lower_argument_required_typed_reference( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranParameter, ...]: + """Return one ordinary interoperable scalar reference parameter.""" + return (self._parameter(plan, ()),) + + def _lower_argument_required_opaque_address( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranParameter, ...]: + """Return one C pointer value for caller-owned scalar storage.""" + name = plan.bridge.native_name.lower() + return (FortranParameter(f"bound_{name}", "type(c_ptr)", ("value",)),) def _lower_argument_nullable_value(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Return one nullable C pointer parameter.""" @@ -370,9 +449,7 @@ def _native_arguments( for slot in plan.native_call_slots if slot.source_kind == "literal" ) - hidden_result = self._hidden_native_result_entry(plan) - if hidden_result is not None: - expressions[hidden_result[0]] = hidden_result[1] + expressions.update(self._hidden_native_result_entries(plan)) return tuple( expressions[slot.native_position] for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position) @@ -396,17 +473,20 @@ def _visible_native_argument_entries( entries.append((argument.native_call_slot.native_position, CodeExpression(expression))) return tuple(entries) - def _hidden_native_result_entry( + def _hidden_native_result_entries( self, plan: FunctionPlan, - ) -> tuple[int, CodeExpression] | None: - """Return one hidden-result native-position entry when present.""" - if plan.result is None or plan.result.source_kind != "hidden_output": - return None - expression = plan.result.bridge.native_name.lower() - if self._has_optional_arguments(plan): - expression = f"{plan.result.bridge.native_name}={expression}" - return plan.result.bridge.abi_position, CodeExpression(expression) + ) -> tuple[tuple[int, CodeExpression], ...]: + """Return all mechanically lowered hidden-result native entries.""" + entries = [] + for slot in plan.native_call_slots: + if slot.source_kind != "result": + continue + expression = self._native_output_value_name(slot) + if self._has_optional_arguments(plan): + expression = f"{slot.native_name}={expression}" + entries.append((slot.native_position, CodeExpression(expression))) + return tuple(entries) def _native_argument_expression(self, plan: ArgumentTransferPlan) -> str: name = plan.bridge.native_name.lower() @@ -423,11 +503,21 @@ def _present_preparation( self, plan: ArgumentTransferPlan, ) -> tuple[FortranAssignment | FortranPointerAssignment | FortranCall | FortranIf, ...]: + """Dispatch only the bridge data action completed before lowering.""" + action = plan.bridge.data_action + match action: + case BridgeDataAction.ASSOCIATE_VIEW: + return self._prepare_present_associated_view(plan) + case BridgeDataAction.COPY_REPRESENTATION: + return self._prepare_present_representation_copy(plan) + raise ValueError(f"Unsupported present bridge data action for {plan.owner_path!r}: {action!r}") + + def _prepare_present_associated_view( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranPointerAssignment | FortranCall | FortranIf, ...]: + """Associate a non-owning native view without copying payload data.""" name = plan.bridge.native_name.lower() - pointer_call = FortranCall( - "c_f_pointer", - (CodeExpression(f"bound_{name}"), CodeExpression(f"{name}_input")), - ) if plan.bridge.optional_mode is OptionalMode.NULLABLE_VALUE: return ( FortranCall( @@ -435,20 +525,42 @@ def _present_preparation( (CodeExpression(f"bound_{name}"), CodeExpression(name)), ), ) - assignment = self._descriptor_assignment(plan, name) + if plan.bridge.optional_mode is not OptionalMode.DESCRIPTOR: + raise ValueError(f"Associated-view preparation requires an optional argument: {plan.owner_path!r}") + if plan.native_call_slot.value_kind != "pointer": + raise ValueError(f"Associated descriptor view requires pointer policy: {plan.owner_path!r}") return ( - pointer_call, - FortranIf(CodeExpression(f"associated({name}_input)"), body=(assignment,)), + self._descriptor_input_pointer_call(name), + FortranIf( + CodeExpression(f"associated({name}_input)"), + body=(FortranPointerAssignment(f"{name}_descriptor", CodeExpression(f"{name}_input")),), + ), ) - def _descriptor_assignment( + def _prepare_present_representation_copy( self, plan: ArgumentTransferPlan, - name: str, - ) -> FortranAssignment | FortranPointerAssignment: - if plan.native_call_slot.value_kind == "pointer": - return FortranPointerAssignment(f"{name}_descriptor", CodeExpression(f"{name}_input")) - return FortranAssignment(f"{name}_descriptor", CodeExpression(f"{name}_input")) + ) -> tuple[FortranCall | FortranIf, ...]: + """Copy only when completed policy requires a different native representation.""" + if plan.bridge.optional_mode is not OptionalMode.DESCRIPTOR: + raise ValueError(f"Representation copy requires descriptor policy: {plan.owner_path!r}") + if plan.native_call_slot.value_kind != "allocatable": + raise ValueError(f"Representation copy requires allocatable policy: {plan.owner_path!r}") + name = plan.bridge.native_name.lower() + return ( + self._descriptor_input_pointer_call(name), + FortranIf( + CodeExpression(f"associated({name}_input)"), + body=(FortranAssignment(f"{name}_descriptor", CodeExpression(f"{name}_input")),), + ), + ) + + def _descriptor_input_pointer_call(self, name: str) -> FortranCall: + """Associate one binding value with a typed descriptor-input view.""" + return FortranCall( + "c_f_pointer", + (CodeExpression(f"bound_{name}"), CodeExpression(f"{name}_input")), + ) def _optional_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: declarations = [] @@ -468,6 +580,40 @@ def _optional_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration ) return tuple(declarations) + def _opaque_address_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Return typed pointer locals for required opaque scalar addresses.""" + return tuple( + FortranDeclaration( + argument.bridge.native_name.lower(), + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).fortran_spelling, + ("pointer",), + ) + for argument in plan.arguments + if ( + argument.bridge.optional_mode is OptionalMode.REQUIRED + and argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + and argument.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + ) + ) + + def _opaque_address_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, ...]: + """Associate typed scalar locals with caller-provided C addresses.""" + return tuple( + FortranCall( + "c_f_pointer", + ( + CodeExpression(f"bound_{argument.bridge.native_name.lower()}"), + CodeExpression(argument.bridge.native_name.lower()), + ), + ) + for argument in plan.arguments + if ( + argument.bridge.optional_mode is OptionalMode.REQUIRED + and argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + and argument.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + ) + ) + def _descriptor_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, ...]: return tuple( FortranCall("nullify", (CodeExpression(f"{argument.bridge.native_name.lower()}_descriptor"),)) @@ -478,23 +624,117 @@ def _descriptor_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, ... ) ) - def _hidden_result_parameters(self, plan: FunctionPlan) -> tuple[FortranParameter, ...]: - if plan.result is None or plan.result.source_kind != "hidden_output": - return () - scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name) - return (FortranParameter(plan.result.bridge.native_name.lower(), scalar_type.fortran_spelling),) + def _native_output_parameters(self, plan: FunctionPlan) -> tuple[FortranParameter, ...]: + """Return bridge ABI parameters for every typed native result slot.""" + parameters = [] + for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position): + if slot.source_kind != "result": + continue + if slot.datatype_family is DatatypeFamily.STRING: + parameters.append(FortranParameter(slot.native_name.lower(), "type(c_ptr)")) + continue + if slot.semantic_type_name is None: + raise ValueError(f"Missing native output datatype for {slot.owner_path!r}") + scalar_type = PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + parameters.append(FortranParameter(slot.native_name.lower(), scalar_type.fortran_spelling)) + return tuple(parameters) + + def _native_output_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Dispatch helper-local output storage from completed bridge data actions.""" + declarations = [] + for slot in plan.native_call_slots: + if slot.source_kind != "result": + continue + if slot.bridge_data_action is BridgeDataAction.DIRECT_TRANSFER: + continue + if slot.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION: + declarations.extend(self._representation_copy_output_declarations(slot)) + continue + raise ValueError( + f"Unsupported native-output bridge data action for {slot.owner_path!r}: {slot.bridge_data_action!r}" + ) + return tuple(declarations) + + def _representation_copy_output_declarations( + self, + slot: NativeCallSlotPlan, + ) -> tuple[FortranDeclaration, ...]: + """Declare storage only for one justified representation-copy output.""" + if slot.datatype_family is not DatatypeFamily.STRING: + raise ValueError(f"Unsupported representation-copy output for {slot.owner_path!r}") + length = self._string_output_length(slot) + value_name = self._native_output_value_name(slot) + return ( + FortranDeclaration(value_name, f"character(kind=c_char, len={length})"), + FortranDeclaration( + f"{slot.native_name.lower()}_copy", + "character(kind=c_char)", + ("pointer", "dimension(:)"), + ), + ) + + def _native_output_finalizers( + self, + plan: FunctionPlan, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Dispatch output finalization from completed bridge data actions.""" + nodes = [] + for slot in plan.native_call_slots: + if slot.source_kind != "result" or slot.bridge_data_action is BridgeDataAction.DIRECT_TRANSFER: + continue + if slot.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION: + nodes.extend(self._lower_native_output_representation_copy(slot)) + continue + raise ValueError( + f"Unsupported native-output bridge data action for {slot.owner_path!r}: {slot.bridge_data_action!r}" + ) + return tuple(nodes) + + def _lower_native_output_representation_copy( + self, + slot: NativeCallSlotPlan, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Copy one native output only through the explicit policy permission.""" + if slot.datatype_family is not DatatypeFamily.STRING: + raise ValueError(f"Unsupported representation-copy output for {slot.owner_path!r}") + name = slot.native_name.lower() + value_name = self._native_output_value_name(slot) + copy_name = f"{name}_copy" + length = self._string_output_length(slot) + c_length = length + 1 + return ( + FortranAssignment(name, CodeExpression(f"c_malloc({c_length}_c_size_t)")), + FortranIf( + CodeExpression(f"c_associated({name})"), + body=( + FortranCall( + "c_f_pointer", + (CodeExpression(name), CodeExpression(copy_name), CodeExpression(f"[{c_length}]")), + ), + FortranAssignment( + f"{copy_name}(1:{length})", + CodeExpression(f"transfer({value_name}, {copy_name}(1:{length}))"), + ), + FortranAssignment(f"{copy_name}({c_length})", CodeExpression("c_null_char")), + ), + ), + ) + + def _native_output_value_name(self, slot: NativeCallSlotPlan) -> str: + """Return the native-call expression selected for one output slot.""" + name = slot.native_name.lower() + return f"{name}_value" if slot.datatype_family is DatatypeFamily.STRING else name + + def _string_output_length(self, slot: NativeCallSlotPlan) -> int: + if slot.character_length is None or slot.character_length <= 0: + raise ValueError(f"String output {slot.owner_path!r} is missing a fixed character length") + return slot.character_length def _bridge_result_type(self, plan: FunctionPlan) -> str: if plan.result is None: raise ValueError(f"{plan.owner_path!r} native function has no result plan") return PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).fortran_spelling - def _external_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: - if not plan.bridge.external or plan.result is None or self._has_optional_arguments(plan): - return () - result_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).fortran_spelling - return (FortranDeclaration(plan.bridge.native_name, result_type, ("external",)),) - def _native_module_uses(self, plan: ModulePlan) -> tuple[FortranUse, ...]: modules: dict[str, list[str]] = {} for function in self._functions(plan): @@ -513,15 +753,22 @@ def _external_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...] procedures = tuple( self._external_interface_procedure(function) for function in self._functions(plan) - if function.bridge.external and self._has_optional_arguments(function) + if function.bridge.external ) return (FortranInterface(procedures),) if procedures else () - def _module_snapshot_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...]: - """Return the allocator interface required by nullable module snapshots.""" - if not any( + def _allocator_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...]: + """Return the allocator interface required by detached bridge copies.""" + needs_snapshot = any( variable.bridge.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT for variable in self._variables(plan) - ): + ) + needs_string_output = any( + slot.datatype_family is DatatypeFamily.STRING + for function in self._functions(plan) + for slot in function.native_call_slots + if slot.source_kind == "result" + ) + if not needs_snapshot and not needs_string_output: return () procedure = FortranInterfaceProcedure( name="c_malloc", @@ -590,7 +837,10 @@ def _variables(self, plan: ModulePlan) -> tuple[ModuleVariablePlan, ...]: def _iso_symbol(self, semantic_type_name: str) -> str: symbols = { "Bool": "c_bool", + "Int8": "c_int8_t", + "Int16": "c_int16_t", "Int32": "c_int32_t", + "Int64": "c_int64_t", "Float32": "c_float", "Float64": "c_double", "Complex64": "c_float_complex", @@ -602,13 +852,18 @@ def _iso_c_symbols(self) -> tuple[str, ...]: return ( "c_associated", "c_bool", + "c_char", "c_double", "c_double_complex", "c_f_pointer", "c_float", "c_float_complex", + "c_int8_t", + "c_int16_t", "c_int", "c_int32_t", + "c_int64_t", + "c_null_char", "c_ptr", "c_null_ptr", "c_size_t", diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index ff818405e..709a5b90f 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -14,17 +14,22 @@ AssignmentMode, CodegenAction, NativeBarrierAction, + PythonBarrierAction, SetterAction, ) from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + BridgeDataAction, ModuleGetterAction, OptionalMode, + PythonExceptionKind, WritebackPhase, ) from x2py.wrapper_codegen.c.binding import CBindingGenerator from x2py.wrapper_codegen.fortran.bridge import FortranBridgeGenerator from x2py.wrapper_codegen.plan import ( ArgumentTransferPlan, + DatatypeFamily, FunctionPlan, LifecycleActionPlan, ModulePlan, @@ -278,6 +283,7 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost *self._duplicate_role_diagnostics(plan), *self._available_role_diagnostics(plan), *self._function_output_diagnostics(plan), + *self._status_error_diagnostics(plan), ] slots = {slot.native_position: slot for slot in plan.native_call_slots} for slot in plan.native_call_slots: @@ -297,6 +303,25 @@ def _argument_diagnostics( function_slots: dict[int, NativeCallSlotPlan], ) -> tuple[WrapperPlanDiagnostic, ...]: """Return binding-to-bridge handoff and slot diagnostics.""" + diagnostics = [ + *self._argument_policy_consistency_diagnostics(plan), + *self._argument_slot_consistency_diagnostics(plan, function_slots), + *self._optional_argument_diagnostics(plan), + *self._scalar_boundary_diagnostics(plan), + *self._argument_data_action_diagnostics(plan), + *self._bridge_data_diagnostics( + plan.owner_path, + plan.bridge.data_action, + plan.bridge.copy_reason, + ), + ] + return tuple(diagnostics) + + def _argument_policy_consistency_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return cross-view role and completed-action diagnostics.""" diagnostics = [] role = plan.binding.handoff_role if plan.bridge.handoff_role != role: @@ -307,6 +332,31 @@ def _argument_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-native-action", plan.bridge.native_action.value) ) + if plan.bridge.data_action is not plan.native_call_slot.bridge_data_action: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-bridge-data-action", + plan.native_call_slot.bridge_data_action.value, + ) + ) + if plan.bridge.copy_reason != plan.native_call_slot.bridge_copy_reason: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-bridge-copy-reason", + plan.native_call_slot.bridge_copy_reason, + ) + ) + return tuple(diagnostics) + + def _argument_slot_consistency_diagnostics( + self, + plan: ArgumentTransferPlan, + function_slots: dict[int, NativeCallSlotPlan], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return argument position and native-slot graph diagnostics.""" + diagnostics = [] if plan.bridge.abi_position != plan.native_position: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-bridge-position", plan.native_position)) if plan.native_call_slot.native_position != plan.native_position: @@ -321,7 +371,65 @@ def _argument_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "invalid-argument-native-slot", plan.native_call_slot.source_kind) ) - diagnostics.extend(self._optional_argument_diagnostics(plan)) + return tuple(diagnostics) + + def _argument_data_action_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require the completed data action to match the selected scalar path.""" + if plan.bridge.optional_mode is OptionalMode.DESCRIPTOR: + expected = ( + BridgeDataAction.COPY_REPRESENTATION + if plan.native_call_slot.value_kind == "allocatable" + else BridgeDataAction.ASSOCIATE_VIEW + ) + elif ( + plan.bridge.optional_mode is OptionalMode.NULLABLE_VALUE + or plan.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + ): + expected = BridgeDataAction.ASSOCIATE_VIEW + else: + expected = BridgeDataAction.DIRECT_TRANSFER + if plan.bridge.data_action is expected: + return () + return ( + self._diagnostic( + plan.owner_path, + "invalid-bridge-data-action", + f"{plan.bridge.data_action.value}:{expected.value}", + ), + ) + + def _scalar_boundary_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return completed Python/native scalar boundary consistency diagnostics.""" + action = plan.binding.python_action + expected = { + PythonBarrierAction.SCALAR_STORAGE: NativeBarrierAction.PASS_STORAGE_ADDRESS, + PythonBarrierAction.RAW_ADDRESS: NativeBarrierAction.PASS_RAW_ADDRESS, + }.get(action) + if expected is None: + if plan.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS: + return (self._diagnostic(plan.owner_path, "unexpected-opaque-address-handoff", action.value),) + return () + diagnostics = [] + if plan.bridge.native_action is not expected: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-scalar-address-action", plan.bridge.native_action.value) + ) + if plan.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-scalar-address-handoff", plan.bridge.handoff_mode.value) + ) + if plan.bridge.data_action is not BridgeDataAction.ASSOCIATE_VIEW: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-scalar-address-data-action", plan.bridge.data_action.value) + ) + if plan.binding.optional_mode is not OptionalMode.REQUIRED: + diagnostics.append(self._diagnostic(plan.owner_path, "optional-scalar-address-boundary", action.value)) return tuple(diagnostics) def _optional_argument_diagnostics( @@ -382,6 +490,13 @@ def _result_diagnostics( ) -> tuple[WrapperPlanDiagnostic, ...]: """Return direct or hidden result producer/consumer diagnostics.""" diagnostics = list(self._result_role_diagnostics(plan, available_roles)) + diagnostics.extend( + self._bridge_data_diagnostics( + plan.owner_path, + plan.bridge.data_action, + plan.bridge.copy_reason, + ) + ) if plan.bridge.codegen_action is not plan.binding.codegen_action: diagnostics.append( self._diagnostic( @@ -408,19 +523,40 @@ def _result_role_diagnostics( return () def _direct_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + diagnostics = [] if plan.native_call_slot is not None or plan.bridge.abi_position is not None: - return (self._diagnostic(plan.owner_path, "direct-result-has-native-slot", plan.source_kind),) - return () + diagnostics.append(self._diagnostic(plan.owner_path, "direct-result-has-native-slot", plan.source_kind)) + if plan.bridge.data_action is not BridgeDataAction.DIRECT_TRANSFER: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-direct-result-data-action", plan.bridge.data_action.value) + ) + return tuple(diagnostics) def _hidden_result_diagnostics( self, plan: ResultPlan, function_slots: dict[int, NativeCallSlotPlan], ) -> tuple[WrapperPlanDiagnostic, ...]: - diagnostics = [] if plan.native_call_slot is None or plan.bridge.abi_position is None: return (self._diagnostic(plan.owner_path, "missing-result-native-slot", plan.bridge.native_name),) slot = plan.native_call_slot + diagnostics = [ + *self._hidden_result_shape_diagnostics(plan, slot), + *self._hidden_result_policy_consistency_diagnostics(plan, slot), + ] + if function_slots.get(slot.native_position) != slot: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-function-result-slot", slot.native_position) + ) + return tuple(diagnostics) + + def _hidden_result_shape_diagnostics( + self, + plan: ResultPlan, + slot: NativeCallSlotPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return hidden-result native-slot shape diagnostics.""" + diagnostics = [] if slot.source_kind != "result": diagnostics.append(self._diagnostic(plan.owner_path, "invalid-result-native-slot", slot.source_kind)) if slot.native_position != plan.bridge.abi_position: @@ -431,6 +567,15 @@ def _hidden_result_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-result-position", slot.result_position)) if slot.symbolic_role != plan.bridge.native_result_role: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-result-role", slot.symbolic_role)) + return tuple(diagnostics) + + def _hidden_result_policy_consistency_diagnostics( + self, + plan: ResultPlan, + slot: NativeCallSlotPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return hidden-result completed-action consistency diagnostics.""" + diagnostics = [] if slot.native_action is not plan.bridge.native_action: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-result-native-action", slot.native_action.value) @@ -439,29 +584,91 @@ def _hidden_result_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-result-slot-codegen-action", slot.codegen_action.value) ) - if function_slots.get(slot.native_position) != slot: + if slot.bridge_data_action is not plan.bridge.data_action: diagnostics.append( - self._diagnostic(plan.owner_path, "inconsistent-function-result-slot", slot.native_position) + self._diagnostic(plan.owner_path, "inconsistent-result-data-action", slot.bridge_data_action.value) + ) + if slot.bridge_copy_reason != plan.bridge.copy_reason: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-result-copy-reason", slot.bridge_copy_reason) ) return tuple(diagnostics) def _native_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Return hidden literal and hidden result slot diagnostics.""" - diagnostics = [] + diagnostics = list( + self._bridge_data_diagnostics( + plan.owner_path, + plan.bridge_data_action, + plan.bridge_copy_reason, + ) + ) if plan.source_kind not in {"implicit", "projection", "literal", "result"}: diagnostics.append(self._diagnostic(plan.owner_path, "unknown-native-slot-source", plan.source_kind)) if plan.source_kind == "literal": - if plan.literal_type is None: - diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-type", plan.native_position)) - if plan.literal_value is None: - diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-value", plan.native_position)) - if plan.python_position is not None: - diagnostics.append(self._diagnostic(plan.owner_path, "literal-python-position", plan.python_position)) + diagnostics.extend(self._literal_slot_diagnostics(plan)) if plan.source_kind == "result": - if plan.result_position is None: - diagnostics.append(self._diagnostic(plan.owner_path, "missing-result-position", plan.native_position)) - if plan.python_position is not None: - diagnostics.append(self._diagnostic(plan.owner_path, "result-python-position", plan.python_position)) + diagnostics.extend(self._result_slot_diagnostics(plan)) + return tuple(diagnostics) + + def _bridge_data_diagnostics( + self, + owner_path: str, + action: BridgeDataAction, + copy_reason: str | None, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject uncompleted or unjustified bridge-side data movement.""" + if action is BridgeDataAction.BLOCKED: + return (self._diagnostic(owner_path, "blocked-bridge-data-action", action.value),) + if action is BridgeDataAction.COPY_REPRESENTATION: + if not copy_reason or not copy_reason.strip(): + return (self._diagnostic(owner_path, "missing-bridge-copy-reason", action.value),) + return () + if copy_reason is not None: + return (self._diagnostic(owner_path, "unexpected-bridge-copy-reason", action.value),) + return () + + def _literal_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one hidden literal slot.""" + diagnostics = [] + if plan.literal_type is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-type", plan.native_position)) + if plan.literal_value is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-value", plan.native_position)) + if plan.python_position is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "literal-python-position", plan.python_position)) + if plan.bridge_data_action is not BridgeDataAction.DIRECT_TRANSFER: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-literal-data-action", plan.bridge_data_action.value) + ) + return tuple(diagnostics) + + def _result_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return diagnostics for one native result slot.""" + diagnostics = [] + if plan.result_position is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-result-position", plan.native_position)) + if plan.python_position is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "result-python-position", plan.python_position)) + if plan.semantic_type_name is None or plan.datatype_family is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-result-datatype", plan.native_position)) + if plan.datatype_family is DatatypeFamily.STRING and plan.character_length is None: + diagnostics.append( + self._diagnostic(plan.owner_path, "missing-result-character-length", plan.native_position) + ) + expected = ( + BridgeDataAction.COPY_REPRESENTATION + if plan.datatype_family is DatatypeFamily.STRING + else BridgeDataAction.DIRECT_TRANSFER + ) + if plan.bridge_data_action is not expected: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-result-data-action", + f"{plan.bridge_data_action.value}:{expected.value}", + ) + ) return tuple(diagnostics) def _lifecycle_diagnostics( @@ -548,22 +755,112 @@ def _writeback_phase_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanD def _function_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Return output projection and native callable-kind diagnostics.""" - diagnostics = [] + diagnostics = [*self._mixed_output_diagnostics(plan)] + diagnostics.extend(self._writeback_result_diagnostics(plan)) + diagnostics.extend(self._native_callable_kind_diagnostics(plan)) + diagnostics.extend(self._unclaimed_result_diagnostics(plan)) + return tuple(diagnostics) + + def _mixed_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject simultaneous public result and writeback projections.""" if plan.result is not None and plan.writeback_actions: - diagnostics.append(self._diagnostic(plan.owner_path, "mixed-result-and-writeback", plan.owner_path)) + return (self._diagnostic(plan.owner_path, "mixed-result-and-writeback", plan.owner_path),) + return () + + def _writeback_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate contiguous Python result positions for copy-out actions.""" result_positions = tuple( action.binding.result_position for action in plan.writeback_actions if action.phase is WritebackPhase.COPY_OUT and action.binding is not None ) - diagnostics.extend( - self._sequence_diagnostics(plan.owner_path, "writeback-result", result_positions, len(result_positions)) + return self._sequence_diagnostics( + plan.owner_path, + "writeback-result", + result_positions, + len(result_positions), ) + + def _native_callable_kind_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require callable kind to agree with the public result representation.""" requires_subroutine = plan.result is None or plan.result.source_kind == "hidden_output" if plan.bridge.native_is_subroutine != requires_subroutine: - diagnostics.append( - self._diagnostic(plan.owner_path, "inconsistent-native-callable-kind", requires_subroutine) - ) + return (self._diagnostic(plan.owner_path, "inconsistent-native-callable-kind", requires_subroutine),) + return () + + def _unclaimed_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require every native result slot to have an explicit binding consumer.""" + claimed_roles = self._claimed_result_roles(plan) + return tuple( + self._diagnostic(plan.owner_path, "unclaimed-native-result", slot.symbolic_role) + for slot in plan.native_call_slots + if slot.source_kind == "result" and slot.symbolic_role not in claimed_roles + ) + + def _claimed_result_roles(self, plan: FunctionPlan) -> set[str]: + """Return public and status-policy consumers of native result slots.""" + roles = set() + if plan.result is not None and plan.result.source_kind == "hidden_output": + roles.add(plan.result.bridge.native_result_role) + if plan.binding.status_error is not None: + roles.add(plan.binding.status_error.status_role) + if plan.binding.status_error.message_role is not None: + roles.add(plan.binding.status_error.message_role) + return roles + + def _status_error_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate completed status/message roles before either backend emits.""" + policy = plan.binding.status_error + if policy is None: + return () + result_slots = {slot.symbolic_role: slot for slot in plan.native_call_slots if slot.source_kind == "result"} + diagnostics = [*self._status_role_diagnostics(plan, result_slots)] + diagnostics.extend(self._message_role_diagnostics(plan, result_slots)) + diagnostics.extend(self._status_policy_diagnostics(plan)) + return tuple(diagnostics) + + def _status_role_diagnostics( + self, + plan: FunctionPlan, + result_slots: dict[str, NativeCallSlotPlan], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the completed integer status role.""" + policy = plan.binding.status_error + status = result_slots.get(policy.status_role) + if status is None: + return (self._diagnostic(plan.owner_path, "missing-status-result-role", policy.status_role),) + if status.datatype_family is not DatatypeFamily.INTEGER: + return (self._diagnostic(plan.owner_path, "incompatible-status-result-role", policy.status_role),) + if status.semantic_type_name != "Int32": + return (self._diagnostic(plan.owner_path, "incompatible-status-result-role", policy.status_role),) + return () + + def _message_role_diagnostics( + self, + plan: FunctionPlan, + result_slots: dict[str, NativeCallSlotPlan], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the optional fixed-length status message role.""" + policy = plan.binding.status_error + if policy.message_role is None: + return () + message = result_slots.get(policy.message_role) + if message is None: + return (self._diagnostic(plan.owner_path, "missing-message-result-role", policy.message_role),) + if message.datatype_family is not DatatypeFamily.STRING: + return (self._diagnostic(plan.owner_path, "incompatible-message-result-role", policy.message_role),) + if message.character_length is None: + return (self._diagnostic(plan.owner_path, "incompatible-message-result-role", policy.message_role),) + return () + + def _status_policy_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate cross-role and exception facts for status handling.""" + policy = plan.binding.status_error + diagnostics = [] + if policy.message_role == policy.status_role: + diagnostics.append(self._diagnostic(plan.owner_path, "duplicate-status-message-role", policy.status_role)) + if policy.exception_kind is not PythonExceptionKind.RUNTIME_ERROR: + diagnostics.append(self._diagnostic(plan.owner_path, "unsupported-status-exception", policy.exception_kind)) return tuple(diagnostics) def _sequence_diagnostics( @@ -592,7 +889,8 @@ def _duplicate_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDi """Return duplicate symbolic producer/consumer role diagnostics.""" roles = [argument.binding.handoff_role for argument in plan.arguments] roles.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "literal") - if plan.result is not None: + roles.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "result") + if plan.result is not None and plan.result.source_kind == "direct_return": roles.append(plan.result.bridge.native_result_role) return tuple( self._diagnostic(plan.owner_path, "duplicate-symbolic-role", role) @@ -603,7 +901,8 @@ def _duplicate_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDi def _available_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Require the advertised roles to match argument and result producers.""" expected = [argument.binding.handoff_role for argument in plan.arguments] - if plan.result is not None: + expected.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "result") + if plan.result is not None and plan.result.source_kind == "direct_return": expected.append(plan.result.bridge.native_result_role) if Counter(plan.available_roles) != Counter(expected): return (self._diagnostic(plan.owner_path, "inconsistent-available-roles", plan.available_roles),) diff --git a/x2py/wrapper_codegen/nodes.py b/x2py/wrapper_codegen/nodes.py index 05f8cf10c..c8f9cae0c 100644 --- a/x2py/wrapper_codegen/nodes.py +++ b/x2py/wrapper_codegen/nodes.py @@ -128,6 +128,16 @@ class CExpressionStatement(StageRecord): expression: CodeExpression +@dataclass +class CAllowThreadsBegin(StageRecord): + """Release the CPython GIL immediately before one native call.""" + + +@dataclass +class CAllowThreadsEnd(StageRecord): + """Reacquire the CPython GIL immediately after one native call.""" + + @dataclass class CIf(StageRecord): """C conditional with recursively printable statement bodies.""" @@ -151,7 +161,7 @@ class CFunction(StageRecord): name: str return_type: str parameters: tuple[CParameter, ...] = () - body: tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...] = () + body: tuple[CDeclaration | CExpressionStatement | CAllowThreadsBegin | CAllowThreadsEnd | CIf | CReturn, ...] = () storage: str | None = None diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py index 684390105..830856631 100644 --- a/x2py/wrapper_codegen/plan.py +++ b/x2py/wrapper_codegen/plan.py @@ -14,8 +14,11 @@ SetterAction, ) from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + BridgeDataAction, ModuleGetterAction, OptionalMode, + PythonExceptionKind, WritebackPhase, ) from x2py.stage_values import StageRecord @@ -28,6 +31,17 @@ class DatatypeFamily(Enum): INTEGER = "integer" REAL = "real" COMPLEX = "complex" + STRING = "string" + + +@dataclass +class BindingStatusErrorPlan(StageRecord): + """Binding-owned post-call native status projection.""" + + status_role: str + message_role: str | None + success: int + exception_kind: PythonExceptionKind @dataclass @@ -86,6 +100,7 @@ class BindingFunctionPlan(StageRecord): python_name: str hold_gil: bool + status_error: BindingStatusErrorPlan | None @dataclass @@ -107,6 +122,7 @@ class BindingArgumentPlan(StageRecord): handoff_role: str optional_mode: OptionalMode nullable: bool + writable: bool descriptor_boundary: bool @@ -116,6 +132,9 @@ class BridgeArgumentPlan(StageRecord): native_name: str native_action: NativeBarrierAction + handoff_mode: ArgumentHandoffMode + data_action: BridgeDataAction + copy_reason: str | None abi_position: int handoff_role: str optional_mode: OptionalMode @@ -137,6 +156,8 @@ class BridgeResultPlan(StageRecord): codegen_action: CodegenAction native_action: NativeBarrierAction + data_action: BridgeDataAction + copy_reason: str | None native_result_role: str native_name: str | None abi_position: int | None @@ -175,9 +196,14 @@ class NativeCallSlotPlan(StageRecord): symbolic_role: str native_action: NativeBarrierAction codegen_action: CodegenAction + bridge_data_action: BridgeDataAction + bridge_copy_reason: str | None literal_type: str | None = None literal_value: Any = None result_position: int | None = None + semantic_type_name: str | None = None + datatype_family: DatatypeFamily | None = None + character_length: int | None = None @dataclass diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py index 11e5c025a..a80abfaee 100644 --- a/x2py/wrapper_codegen/planner.py +++ b/x2py/wrapper_codegen/planner.py @@ -12,6 +12,7 @@ FunctionWrapperPolicy, LifecyclePolicy, NativeCallSlotPolicy, + NativeStatusErrorPolicy, ResultPolicy, WritebackPhase, completed_module_variable_policy, @@ -27,6 +28,7 @@ BindingModulePlan, BindingModuleVariablePlan, BindingResultPlan, + BindingStatusErrorPlan, BridgeArgumentPlan, BridgeFunctionPlan, BridgeLifecyclePlan, @@ -48,11 +50,15 @@ _DATATYPE_FAMILIES = { "Bool": DatatypeFamily.BOOL, + "Int8": DatatypeFamily.INTEGER, + "Int16": DatatypeFamily.INTEGER, "Int32": DatatypeFamily.INTEGER, + "Int64": DatatypeFamily.INTEGER, "Float32": DatatypeFamily.REAL, "Float64": DatatypeFamily.REAL, "Complex64": DatatypeFamily.COMPLEX, "Complex128": DatatypeFamily.COMPLEX, + "String": DatatypeFamily.STRING, } @@ -179,12 +185,20 @@ def _function_plan( module_name: str, ) -> FunctionPlan: """Return one exported function plan from completed policy.""" - arguments = self._argument_plans(policy) - result = self._result_plan(policy) + native_call_slots = tuple( + self._native_slot_plan(slot, self._native_slot_role(slot, policy.result)) + for slot in policy.native_call_slots + ) + arguments = self._argument_plans(policy, native_call_slots) + result = self._result_plan(policy, native_call_slots) return FunctionPlan( owner_path=self._export_owner_path(module_name, export.namespace, export.name), symbol_name=export.name.casefold(), - binding=BindingFunctionPlan(export.name, policy.hold_gil), + binding=BindingFunctionPlan( + export.name, + policy.hold_gil, + self._status_error_plan(policy.status_error, native_call_slots), + ), bridge=BridgeFunctionPlan( policy.native_name, policy.external, @@ -193,32 +207,45 @@ def _function_plan( ), arguments=arguments, result=result, - native_call_slots=tuple( - self._native_slot_plan(slot, self._native_slot_role(slot, result)) for slot in policy.native_call_slots - ), - available_roles=self._available_roles(arguments, result), + native_call_slots=native_call_slots, + available_roles=self._available_roles(arguments, result, native_call_slots), writeback_actions=tuple(self.visit(action) for action in policy.writeback_actions), cleanup_actions=tuple(self.visit(action) for action in policy.cleanup_actions), release_actions=tuple(self.visit(action) for action in policy.release_actions), ) - def _argument_plans(self, policy: FunctionWrapperPolicy) -> tuple[ArgumentTransferPlan, ...]: - """Return declared transfers wired to completed native slots.""" + def _argument_plans( + self, + policy: FunctionWrapperPolicy, + native_call_slots: tuple[NativeCallSlotPlan, ...], + ) -> tuple[ArgumentTransferPlan, ...]: + """Return declared transfers sharing the function's native-slot records.""" return tuple( - self.visit(argument, native_slot=self._native_slot(policy, argument)) for argument in policy.arguments + self.visit( + argument, + native_slot=self._planned_native_slot(native_call_slots, argument.owner_path), + ) + for argument in policy.arguments ) - def _result_plan(self, policy: FunctionWrapperPolicy) -> ResultPlan | None: + def _result_plan( + self, + policy: FunctionWrapperPolicy, + native_call_slots: tuple[NativeCallSlotPlan, ...], + ) -> ResultPlan | None: """Return one completed result plan when the function has a result.""" if policy.result is None: return None - return self.visit(policy.result, native_slot=self._result_native_slot(policy)) + return self.visit( + policy.result, + native_slot=self._result_native_slot(policy, native_call_slots), + ) def _visit_ArgumentPolicy( self, policy: ArgumentPolicy, *, - native_slot: NativeCallSlotPolicy, + native_slot: NativeCallSlotPlan, ) -> ArgumentTransferPlan: """Return one transfer whose backend views share one handoff role.""" role = self._value_role(policy.owner_path) @@ -234,17 +261,21 @@ def _visit_ArgumentPolicy( role, policy.optional_mode, policy.nullable, + policy.writable, policy.descriptor_boundary, ), bridge=BridgeArgumentPlan( policy.native_name, policy.native_barrier_action, + policy.handoff_mode, + policy.bridge_data_action, + policy.bridge_copy_reason, native_slot.native_position, role, policy.optional_mode, f"{policy.owner_path}:present" if policy.descriptor_boundary else None, ), - native_call_slot=self._native_slot_plan(native_slot, role), + native_call_slot=native_slot, ) def _visit_LifecyclePolicy( @@ -280,7 +311,7 @@ def _visit_ResultPolicy( self, policy: ResultPolicy, *, - native_slot: NativeCallSlotPolicy | None, + native_slot: NativeCallSlotPlan | None, ) -> ResultPlan: """Return one result with binding consumer and bridge producer views.""" native_role = f"{policy.owner_path}:native-result" @@ -300,11 +331,13 @@ def _visit_ResultPolicy( bridge=BridgeResultPlan( policy.codegen_action, policy.native_barrier_action, + policy.bridge_data_action, + policy.bridge_copy_reason, native_role, policy.native_name, policy.native_position, ), - native_call_slot=self._native_slot_plan(native_slot, native_role) if native_slot is not None else None, + native_call_slot=native_slot, ) def _native_slot_plan(self, slot: NativeCallSlotPolicy, role: str) -> NativeCallSlotPlan: @@ -320,44 +353,81 @@ def _native_slot_plan(self, slot: NativeCallSlotPolicy, role: str) -> NativeCall symbolic_role=role, native_action=slot.native_barrier_action, codegen_action=slot.codegen_action, + bridge_data_action=slot.bridge_data_action, + bridge_copy_reason=slot.bridge_copy_reason, literal_type=slot.literal_type, literal_value=slot.literal_value, result_position=slot.result_position, + semantic_type_name=slot.semantic_type_name, + datatype_family=(self._datatype_family(slot.semantic_type_name) if slot.semantic_type_name else None), + character_length=slot.character_length, ) - def _native_slot( + def _status_error_plan( self, - function_policy: FunctionWrapperPolicy, - argument_policy: ArgumentPolicy, - ) -> NativeCallSlotPolicy: - """Return the completed native-call slot for one argument policy.""" - for slot in function_policy.native_call_slots: - if slot.owner_path == argument_policy.owner_path: + policy: NativeStatusErrorPolicy | None, + native_call_slots: tuple[NativeCallSlotPlan, ...], + ) -> BindingStatusErrorPlan | None: + """Project one completed native-status decision into binding roles.""" + if policy is None: + return None + roles = {slot.owner_path: slot.symbolic_role for slot in native_call_slots} + try: + status_role = roles[policy.status.owner_path] + message_role = roles[policy.message.owner_path] if policy.message is not None else None + except KeyError as error: + raise ValueError(f"Completed native status output {error.args[0]!r} has no native-call slot") from None + return BindingStatusErrorPlan( + status_role=status_role, + message_role=message_role, + success=policy.success, + exception_kind=policy.exception_kind, + ) + + def _planned_native_slot( + self, + native_call_slots: tuple[NativeCallSlotPlan, ...], + owner_path: str, + ) -> NativeCallSlotPlan: + """Return the one shared editable native-call slot for an owner.""" + for slot in native_call_slots: + if slot.owner_path == owner_path: return slot - raise ValueError(f"{argument_policy.owner_path!r} is missing a completed native-call slot") + raise ValueError(f"{owner_path!r} is missing a completed native-call slot") def _result_native_slot( self, function_policy: FunctionWrapperPolicy, - ) -> NativeCallSlotPolicy | None: + native_call_slots: tuple[NativeCallSlotPlan, ...], + ) -> NativeCallSlotPlan | None: """Return the completed slot for one hidden result, if any.""" if function_policy.result is None or function_policy.result.source_kind != "hidden_output": return None - for slot in function_policy.native_call_slots: - if slot.owner_path == function_policy.result.owner_path: - return slot - raise ValueError(f"{function_policy.result.owner_path!r} is missing a completed native-call result slot") + return self._planned_native_slot(native_call_slots, function_policy.result.owner_path) def _available_roles( self, arguments: tuple[ArgumentTransferPlan, ...], result: ResultPlan | None, + native_call_slots: tuple[NativeCallSlotPlan, ...], ) -> tuple[str, ...]: """Return symbolic roles available after the native call.""" roles = [argument.binding.handoff_role for argument in arguments] - if result is not None: - roles.append(result.bridge.native_result_role) - return tuple(roles) + roles.extend(self._native_result_roles(native_call_slots)) + roles.extend(self._direct_result_roles(result)) + return tuple(dict.fromkeys(roles)) + + def _native_result_roles(self, native_call_slots: tuple[NativeCallSlotPlan, ...]) -> tuple[str, ...]: + """Return every role produced through a native result slot.""" + return tuple(slot.symbolic_role for slot in native_call_slots if slot.source_kind == "result") + + def _direct_result_roles(self, result: ResultPlan | None) -> tuple[str, ...]: + """Return the direct-return role when the callable produces one.""" + if result is None: + return () + if result.source_kind != "direct_return": + return () + return (result.bridge.native_result_role,) def _datatype_family(self, semantic_type_name: str) -> DatatypeFamily: """Copy the backend-relevant family of one supported semantic type.""" @@ -400,13 +470,27 @@ def _value_role(self, owner_path: str) -> str: def _native_slot_role( self, native_slot: NativeCallSlotPolicy, - result: ResultPlan | None, + result: ResultPolicy | None, ) -> str: """Return the symbolic role for one native-call slot.""" if native_slot.source_kind == "literal": return f"{native_slot.owner_path}:literal" + if self._is_public_result_slot(native_slot, result): + return f"{result.owner_path}:native-result" if native_slot.source_kind == "result": - if result is None or result.native_call_slot is None: - raise ValueError(f"{native_slot.owner_path!r} result slot has no result plan") - return result.bridge.native_result_role + return f"{native_slot.owner_path}:native-result" return self._value_role(native_slot.owner_path) + + def _is_public_result_slot( + self, + native_slot: NativeCallSlotPolicy, + result: ResultPolicy | None, + ) -> bool: + """Return whether a native slot carries the Python-visible hidden result.""" + if native_slot.source_kind != "result": + return False + if result is None: + return False + if result.source_kind != "hidden_output": + return False + return result.owner_path == native_slot.owner_path diff --git a/x2py/wrapper_codegen/primitive_scalar_types.py b/x2py/wrapper_codegen/primitive_scalar_types.py index c6f463bc1..74e2479b4 100644 --- a/x2py/wrapper_codegen/primitive_scalar_types.py +++ b/x2py/wrapper_codegen/primitive_scalar_types.py @@ -20,10 +20,34 @@ class PrimitiveScalarTypeRegistry: "NPY_BOOL", "Bool_to_PyBool", "PyBool_to_Bool", - "PyArray_IsScalar({object_name}, Bool)", - "numpy.bool_", + "PyIs_Bool({object_name})", + "bool", "Bool_to_PyBool", ), + "Int8": BackendScalarType( + "Int8", + "int8_t", + "integer(c_int8_t)", + "O", + "NPY_INT8", + "Int8_to_NumpyLong", + "PyInt8_to_Int8", + "PyIs_Int8({object_name})", + "numpy.int8", + "Int8_to_NumpyLong", + ), + "Int16": BackendScalarType( + "Int16", + "int16_t", + "integer(c_int16_t)", + "O", + "NPY_INT16", + "Int16_to_NumpyLong", + "PyInt16_to_Int16", + "PyIs_Int16({object_name})", + "numpy.int16", + "Int16_to_NumpyLong", + ), "Int32": BackendScalarType( "Int32", "int32_t", @@ -36,6 +60,18 @@ class PrimitiveScalarTypeRegistry: "numpy.int32", "Int32_to_NumpyLong", ), + "Int64": BackendScalarType( + "Int64", + "int64_t", + "integer(c_int64_t)", + "O", + "NPY_INT64", + "Int64_to_PyLong", + "PyInt64_to_Int64", + "PyIs_Int64({object_name})", + "numpy.int64", + "Int64_to_NumpyLong", + ), "Float32": BackendScalarType( "Float32", "float", diff --git a/x2py/wrapper_codegen/source_printers.py b/x2py/wrapper_codegen/source_printers.py index 681b2ebf5..87ff882b5 100644 --- a/x2py/wrapper_codegen/source_printers.py +++ b/x2py/wrapper_codegen/source_printers.py @@ -3,6 +3,8 @@ from __future__ import annotations from x2py.wrapper_codegen.nodes import ( + CAllowThreadsBegin, + CAllowThreadsEnd, CDeclaration, CExpressionStatement, CFunction, @@ -218,6 +220,14 @@ def _visit_CExpressionStatement(self, node: CExpressionStatement) -> str: """Render one C expression statement.""" return f"{node.expression.text};" + def _visit_CAllowThreadsBegin(self, _node: CAllowThreadsBegin) -> str: + """Render the opening CPython thread-release macro without a semicolon.""" + return "Py_BEGIN_ALLOW_THREADS" + + def _visit_CAllowThreadsEnd(self, _node: CAllowThreadsEnd) -> str: + """Render the closing CPython thread-release macro without a semicolon.""" + return "Py_END_ALLOW_THREADS" + def _visit_CIf(self, node: CIf) -> str: """Render one C conditional statement.""" lines = [f"if ({node.condition.text}) {{"] diff --git a/x2py/wrapper_codegen/support.py b/x2py/wrapper_codegen/support.py index 317f0bae9..a6fbf7a93 100644 --- a/x2py/wrapper_codegen/support.py +++ b/x2py/wrapper_codegen/support.py @@ -7,6 +7,7 @@ ModuleVariablePolicy, FunctionWrapperPolicy, ) +from x2py.semantics.ownership import PythonBarrierAction from x2py.wrapper_codegen.plan import WrapperPlanSupportBlocker, WrapperPlanSupportReport from x2py.wrapper_codegen.visitor import ClassVisitor @@ -85,11 +86,21 @@ def _function_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: """Return completed first-lane coverage for one supported function.""" if policy.blockers: return () - return (*self._argument_lanes(policy), *self._output_lanes(policy)) + runtime_lanes = ["native-call-runtime"] + if policy.status_error is not None: + runtime_lanes.append("native-status-errors") + return (*self._argument_lanes(policy), *self._output_lanes(policy), *runtime_lanes) def _argument_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: """Return input-related lanes selected by completed arguments.""" - lanes = ["scalar-inputs"] if policy.arguments else [] + actions = {argument.python_barrier_action for argument in policy.arguments} + lanes = [] + if PythonBarrierAction.SCALAR_VALUE in actions: + lanes.append("scalar-inputs") + if PythonBarrierAction.SCALAR_STORAGE in actions: + lanes.append("scalar-storage-inputs") + if PythonBarrierAction.RAW_ADDRESS in actions: + lanes.append("scalar-raw-address-inputs") if any(argument.optional for argument in policy.arguments): lanes.append("scalar-optional-inputs") if any(argument.descriptor_boundary for argument in policy.arguments): From 54ef0086286803e2a953ec1a28e349c1ce093b88 Mon Sep 17 00:00:00 2001 From: said Date: Mon, 13 Jul 2026 14:46:47 +0100 Subject: [PATCH 08/30] multiple results --- .../wrapper-plan-migration-checklist.md | 46 ++++-- .../test_wrapper_plan_route_selection.py | 28 ++++ tests/semantics/policy/test_wrapper_policy.py | 70 +++++++-- tests/wrapper/CHECKLIST_COVERAGE.md | 2 +- tests/wrapper/fortran/scalars/README.md | 3 +- .../scalars/test_scalar_boundary_plan.py | 33 ++++ .../wrapper_codegen/test_phase0d_plan_core.py | 18 +-- .../test_phase2b_hidden_scalar_outputs.py | 3 +- .../test_phase2f_multiple_scalar_results.py | 92 +++++++++++ x2py/pipeline/build.py | 3 + x2py/semantics/policy_completion.py | 15 ++ x2py/semantics/wrapper_policy.py | 75 +++++---- x2py/wrapper_codegen/c/binding.py | 146 ++++++++++++------ x2py/wrapper_codegen/fortran/bridge.py | 38 ++--- x2py/wrapper_codegen/generator.py | 69 ++++++--- x2py/wrapper_codegen/plan.py | 2 +- x2py/wrapper_codegen/planner.py | 75 ++++----- x2py/wrapper_codegen/support.py | 14 +- 18 files changed, 536 insertions(+), 196 deletions(-) create mode 100644 tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index bf8baf559..98d4d012a 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -491,7 +491,7 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 65 | +| `wrapper-plan` | 66 | | `dual-route` | 0 | | `legacy` | 134 | | `not-applicable` | 95 | @@ -502,14 +502,17 @@ summary, the exhaustive matrix, and the test tree disagree. This history keeps phase movement visible instead of replacing the previous snapshot with only the latest totals. Phase 2D moved all 17 dual-route nodes and 44 legacy nodes to production plan routing, then added two parametrized -plan-route nodes. Phase 2E adds two scalar-only parity nodes; the original -mixed integration nodes retain their real array/string/object blockers. +plan-route nodes. Phase 2E adds two scalar-only parity nodes, and Phase 2F adds +one isolated direct-return plus hidden-output scalar aggregation node. The +original mixed integration nodes retain their real array/string/object +blockers. | Proven checkpoint | `wrapper-plan` | `dual-route` | `legacy` | `not-applicable` | `deferred-real-library` | Total | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Before Phase 2D | 0 | 17 | 178 | 95 | 2 | 292 | | Phase 2D complete | 63 | 0 | 134 | 95 | 2 | 294 | | Phase 2E scalar isolation | 65 | 0 | 134 | 95 | 2 | 296 | +| Phase 2F scalar result aggregation | 66 | 0 | 134 | 95 | 2 | 297 | Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating @@ -618,7 +621,7 @@ already covered by the new generator. | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | @@ -655,7 +658,7 @@ already covered by the new generator. | `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `legacy` | | `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | direct wrapper/build route | scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::*` | scalar-only copied native routines with deliberate legacy/direct-plan parity | primitive scalar kinds; value and `Addr(Arg(i))` inputs; hidden output; copy-in/copy-out; rank-zero storage; raw `Addr(T)`; native slot reordering | `wrapper-plan` | +| `tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::*` | scalar-only copied native routines with deliberate legacy/direct-plan parity | primitive scalar kinds; value and `Addr(Arg(i))` inputs; hidden output; copy-in/copy-out; rank-zero storage; raw `Addr(T)`; native slot reordering; direct-plus-hidden result tuple assembly | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | | `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `legacy` | @@ -1137,7 +1140,7 @@ the legacy generator. route counts, and leave the original mixed integration nodes on their real datatype blockers. -## Phase 2F — Multiple Scalar Result Assembly — Pending +## Phase 2F — Multiple Scalar Result Assembly — Complete This is result aggregation, not another scalar boundary representation. The first isolated oracle is the `with_scalar` policy from @@ -1146,13 +1149,36 @@ one hidden primitive scalar output, assembled into a Python tuple in declared result order. Keep it separate from arrays, strings, derived types, and native handles before widening the plan route. -- [ ] Add a scalar-only copied native routine and contract for a direct return +The completed representation is an ordered `FunctionWrapperPolicy.results` +tuple and an ordered `FunctionPlan.results` tuple. Each Python-visible result +has its own `ResultPolicy` and `ResultPlan`, including its binding consumer and +`result_position`. A direct native function return has +`source_kind="direct_return"` and no native-call slot. A hidden output has +`source_kind="hidden_output"` and references the exact same mutable +`NativeCallSlotPlan` stored in `FunctionPlan.native_call_slots`. The bridge +uses the sole direct result, when present, to select its function result and +passes every hidden result through its completed output-address slot. It does +not assemble Python results. + +After the native call, the binding converts each result from its completed +source role exactly once. One result is returned directly; two or more are +assembled into a Python tuple in ascending `result_position`. Tuple allocation, +reference transfer, and failure cleanup are binding-local emission details, +not semantic policy. Before either backend emits source, validation requires +result positions to cover `0..N-1` exactly once, at most one direct result, +every hidden result to share its function native-call slot, and every +non-status native output slot to have exactly one binding result consumer. +Phase 2F does not combine these consumers with projected argument writeback; +that broader aggregation remains blocked until it receives its own completed +policy. + +- [x] Add a scalar-only copied native routine and contract for a direct return plus hidden scalar `Return(...)` slot. -- [ ] Represent every Python result as an explicit binding consumer while +- [x] Represent every Python result as an explicit binding consumer while preserving the bridge's direct-return and output-address ABI roles. -- [ ] Validate contiguous result positions and reject unclaimed outputs before +- [x] Validate contiguous result positions and reject unclaimed outputs before either backend emits source. -- [ ] Prove compiled legacy/direct-plan parity, then update the route counts. +- [x] Prove compiled legacy/direct-plan parity, then update the route counts. ## Phase 5 — Strings diff --git a/tests/pipeline/test_wrapper_plan_route_selection.py b/tests/pipeline/test_wrapper_plan_route_selection.py index 07485d2e9..a52451bf9 100644 --- a/tests/pipeline/test_wrapper_plan_route_selection.py +++ b/tests/pipeline/test_wrapper_plan_route_selection.py @@ -72,6 +72,8 @@ def scale(x: Float64) -> Float64: ... "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_multiple_scalar_results_match_both_routes_without_array_blockers", ) assert decision.selection_reason == "wrapper-plan route forced for internal migration verification" @@ -221,6 +223,32 @@ def update_raw(value: Addr(Float64)) -> None: ... ) +def test_route_selector_selects_multiple_scalar_results_in_production(): + module = _completed_module( + """ +@native_call([Addr(Arg(0)), Return("status", 1)]) +def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... +""", + module_name="multiple_scalar_results", + ) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "wrapper-plan" + assert decision.rollout_eligible is True + assert decision.covered_lanes == ( + "scalar-inputs", + "scalar-direct-results", + "scalar-hidden-outputs", + "scalar-multiple-results", + "native-call-runtime", + ) + + def test_route_selector_accepts_completed_scalar_module_variable_lane(): module = _completed_module( """ diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index 2013f63de..d6fba726c 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -11,6 +11,7 @@ from x2py.semantics.models import ( RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA, RESOLVED_MODULE_VARIABLE_POLICY_METADATA, + RESOLVED_OWNERSHIP_POLICY_METADATA, RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA, SemanticFunction, SemanticType, @@ -118,7 +119,7 @@ def test_scalar_copy_in_out_policy_completes_writeback_before_planning(): policy = completed_function_wrapper_policy(module.functions[0]) assert policy.supported is True - assert policy.result is None + assert policy.results == () assert policy.native_is_subroutine is True assert tuple(action.phase for action in policy.writeback_actions) == tuple(WritebackPhase) assert {action.source_role for action in policy.writeback_actions} == {"scalar_writeback.bump.value:value"} @@ -146,6 +147,50 @@ def mapped_status(base: Int32) -> Int32: ... ] +def test_multiple_scalar_result_policy_completes_order_and_hidden_address_before_planning(): + module = parse_pyi_text( + """ +@native_call([Addr(Arg(0)), Return("status", 1)]) +def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... +""", + module_name="multiple_scalar_results", + ) + complete_semantic_policies(module) + + policy = completed_function_wrapper_policy(module.functions[0]) + + assert policy.supported is True + assert [(result.source_kind, result.result_position) for result in policy.results] == [ + ("direct_return", 0), + ("hidden_output", 1), + ] + hidden = policy.results[1] + assert hidden.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert policy.native_call_slots[1].owner_path == hidden.owner_path + assert policy.native_call_slots[1].result_position == hidden.result_position + assert policy.native_call_slots[1].native_barrier_action is hidden.native_barrier_action + + +def test_hidden_scalar_descriptor_result_keeps_descriptor_policy_instead_of_plain_address_storage(): + module = parse_pyi_text( + """ +@native_call([Allocatable(Return("value", 0))]) +def create_allocatable() -> Float64 | None: ... +""", + module_name="descriptor_result", + ) + function = module.functions[0] + result_argument = function.arguments[0] + + complete_semantic_policies(module) + + decision = result_argument.metadata[RESOLVED_OWNERSHIP_POLICY_METADATA] + assert function.projection[0].value_kind == "allocatable" + assert result_argument.semantic_type.storage is None + assert decision.descriptor_boundary is True + assert decision.native_barrier_action is NativeBarrierAction.PASS_VALUE + + def test_source_fmath_scalar_policy_accepts_storage_address_native_action(): module = _source_semantic_module("fmath.f", module_name="fmath") function = next(item for item in module.functions if item.name == "ADD_R8") @@ -255,16 +300,17 @@ def test_fmath_scalar_policy_records_address_projected_call_slots(): slot.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS for slot in policy.native_call_slots ) - assert policy.result is not None - assert policy.result.owner_path == "fmath.add_r8.return" - assert policy.result.semantic_type_name == "Float64" - assert policy.result.rank == 0 - assert policy.result.ownership.kind is ObjectKind.SCALAR - assert policy.result.codegen_action is CodegenAction.DIRECT_VALUE - assert policy.result.python_barrier_action is PythonBarrierAction.NONE - assert policy.result.native_barrier_action is NativeBarrierAction.NONE - assert policy.result.storage_mode is StorageMode.STACK - assert policy.result.boundary_storage_mode is StorageMode.STACK + assert len(policy.results) == 1 + result = policy.results[0] + assert result.owner_path == "fmath.add_r8.return" + assert result.semantic_type_name == "Float64" + assert result.rank == 0 + assert result.ownership.kind is ObjectKind.SCALAR + assert result.codegen_action is CodegenAction.DIRECT_VALUE + assert result.python_barrier_action is PythonBarrierAction.NONE + assert result.native_barrier_action is NativeBarrierAction.NONE + assert result.storage_mode is StorageMode.STACK + assert result.boundary_storage_mode is StorageMode.STACK def test_wrapper_policy_records_runtime_and_native_order_metadata(): @@ -319,7 +365,7 @@ def solve(value: Int32) -> tuple[Int32, String[32]]: ... assert status_error.message.native_position == 2 assert status_error.message.semantic_type_name == "String" assert status_error.message.character_length == 32 - assert policy.result is None + assert policy.results == () assert [slot.semantic_type_name for slot in policy.native_call_slots] == ["Int32", "Int32", "String"] assert [slot.character_length for slot in policy.native_call_slots] == [None, None, 32] diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index 8429e9c69..546a5e9a7 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -53,7 +53,7 @@ records the zero-legacy completion target. | Roadmap item | Evidence | | --- | --- | -| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators; isolated scalar-only parity covers primitive kinds, value/address projection, hidden output, copy-in/copy-out, rank-zero storage, and raw addresses without array/string route blockers | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_boundary_plan.py`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | +| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators; isolated scalar-only parity covers primitive kinds, value/address projection, hidden output, copy-in/copy-out, rank-zero storage, raw addresses, and direct-plus-hidden multiple-result tuple assembly without array/string route blockers | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_boundary_plan.py`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | | Function-call contracts rebuild from generated `.pyi` fixtures with the same optional-argument handling, scalar replacement writeback, hidden output projection, multiple-result ordering, allocatable nullable outputs, native-call projection metadata, native shared-library link inputs, and validation failures as source builds | `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes`, `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules`, `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection`, `function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | | Array contracts rebuild from generated `.pyi` fixtures with the same dtype, rank, shape, order, stride, lower-bound, writeability, byte-order, alignment, zero-extent, assumed-rank dispatch, ordinary Python-owned result behavior, and allocatable result-handle behavior as source builds | `arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument`, `arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides`, `arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views` | | Character contracts rebuild from generated `.pyi` fixtures with the same fixed-length buffers, assumed-length strings, deferred character results, copy-in/copy-out behavior, optional strings, Unicode handling, and embedded-NUL validation as source builds | `strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results`, `strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_projected_replacement_without_native_call_keeps_writable_argument_storage`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_native_call_projected_output_keeps_visible_storage_writable` | diff --git a/tests/wrapper/fortran/scalars/README.md b/tests/wrapper/fortran/scalars/README.md index 121087ab3..fa117d37a 100644 --- a/tests/wrapper/fortran/scalars/README.md +++ b/tests/wrapper/fortran/scalars/README.md @@ -2,7 +2,8 @@ Scope: scalar calls, scalar kind coverage, `value` and scalar `bind(C)` behavior, value/storage/raw-address boundaries, scalar output and inout -projection, enum-like values, and the basic compiled-wrapper baseline. +projection, direct-plus-hidden multiple-result assembly, enum-like values, and +the basic compiled-wrapper baseline. Focused pytest command: `python3 -m pytest -q tests/wrapper/fortran/scalars` diff --git a/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py index 8ceccffd8..2d1b754a6 100644 --- a/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py +++ b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py @@ -235,6 +235,32 @@ def conj_c128(value: Complex128) -> Complex128: ... ) +def _build_multiple_scalar_result_modules(tmp_path: Path): + return _build_contract_routes( + tmp_path, + module_name="multiple_scalar_results_plan", + source_text=""" +module multiple_scalar_results_plan + use iso_c_binding, only: c_int32_t +contains + function with_scalar(n, status) result(value) + integer(c_int32_t), intent(in) :: n + integer(c_int32_t), intent(out) :: status + integer(c_int32_t) :: value + value = n * 2 + status = n + 3 + end function with_scalar +end module multiple_scalar_results_plan +""", + contract_text=""" +from x2py.contracts import Addr, Arg, Int32, Return, native_call + +@native_call([Addr(Arg(0)), Return("status", 1)]) +def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... +""", + ) + + def test_scalar_value_storage_raw_address_out_and_inout_match_both_routes(tmp_path: Path): modules = _build_scalar_boundary_modules(tmp_path) @@ -286,6 +312,13 @@ def test_scalar_value_storage_raw_address_out_and_inout_match_both_routes(tmp_pa module.bump_raw(raw) +def test_multiple_scalar_results_match_both_routes_without_array_blockers(tmp_path: Path): + modules = _build_multiple_scalar_result_modules(tmp_path) + + for module in modules: + assert module.with_scalar(np.int32(4)) == (np.int32(8), np.int32(7)) + + def test_scalar_primitive_kinds_match_both_routes_without_array_blockers(tmp_path: Path): modules = _build_scalar_kind_modules(tmp_path) diff --git a/tests/wrapper_codegen/test_phase0d_plan_core.py b/tests/wrapper_codegen/test_phase0d_plan_core.py index bf8f084c2..a13b2c796 100644 --- a/tests/wrapper_codegen/test_phase0d_plan_core.py +++ b/tests/wrapper_codegen/test_phase0d_plan_core.py @@ -78,8 +78,8 @@ def test_planner_projects_one_shared_tree_with_explicit_backend_views(): assert first.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS assert first.binding.handoff_role == first.bridge.handoff_role == first.native_call_slot.symbolic_role assert first.native_call_slot.codegen_action is CodegenAction.CALL_LOCAL_INPUT - assert function.result.binding.codegen_action is CodegenAction.DIRECT_VALUE - assert function.result.bridge.native_result_role in function.available_roles + assert function.results[0].binding.codegen_action is CodegenAction.DIRECT_VALUE + assert function.results[0].bridge.native_result_role in function.available_roles def test_planner_records_hidden_literals_and_hidden_result_slots(): @@ -92,15 +92,15 @@ def test_planner_records_hidden_literals_and_hidden_result_slots(): ("literal", "Bool", False), ("result", None, None), ] - assert function.result.source_kind == "hidden_output" - assert function.result.bridge.abi_position == 3 - assert function.result.native_call_slot == function.native_call_slots[3] + assert function.results[0].source_kind == "hidden_output" + assert function.results[0].bridge.abi_position == 3 + assert function.results[0].native_call_slot is function.native_call_slots[3] def test_generator_rejects_hidden_result_native_action_disagreement(): plan = _hidden_result_plan() function = plan.namespaces[0].functions[0] - result = function.result + result = function.results[0] replacement = ( NativeBarrierAction.PASS_VALUE if result.bridge.native_action is not NativeBarrierAction.PASS_VALUE @@ -110,7 +110,7 @@ def test_generator_rejects_hidden_result_native_action_disagreement(): plan, lambda item: replace( item, - result=replace(result, bridge=replace(result.bridge, native_action=replacement)), + results=(replace(result, bridge=replace(result.bridge, native_action=replacement)),), ), ) @@ -121,13 +121,13 @@ def test_generator_rejects_hidden_result_native_action_disagreement(): def test_generator_rejects_hidden_result_slot_codegen_action_disagreement(): plan = _hidden_result_plan() function = plan.namespaces[0].functions[0] - result = function.result + result = function.results[0] edited_slot = replace(result.native_call_slot, codegen_action=CodegenAction.DIRECT_VALUE) invalid = _edit_first_function( plan, lambda item: replace( item, - result=replace(result, native_call_slot=edited_slot), + results=(replace(result, native_call_slot=edited_slot),), native_call_slots=tuple( edited_slot if slot.native_position == edited_slot.native_position else slot for slot in item.native_call_slots diff --git a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py index 60c8dea20..d1852fc59 100644 --- a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py +++ b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py @@ -20,9 +20,8 @@ def scale(x: Float64) -> Float64: ... complete_semantic_policies(module) plan = WrapperPlanner().build(module) function = plan.namespaces[0].functions[0] - result = function.result + result = function.results[0] - assert result is not None assert result.native_call_slot is function.native_call_slots[result.bridge.abi_position] artifacts = WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py b/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py new file mode 100644 index 000000000..ed7665c08 --- /dev/null +++ b/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py @@ -0,0 +1,92 @@ +"""Phase 2F direct-return plus hidden-output scalar aggregation.""" + +from __future__ import annotations + +from dataclasses import replace + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import NativeBarrierAction +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + +def _multiple_result_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Addr, Arg, Int32, Return, native_call + +@native_call([Addr(Arg(0)), Return("status", 1)]) +def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... +""", + module_name="multiple_scalar_results", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_multiple_scalar_result_plan_has_ordered_binding_consumers_and_shared_hidden_slot(): + function = _multiple_result_plan().namespaces[0].functions[0] + direct, hidden = function.results + + assert [(result.source_kind, result.result_position) for result in function.results] == [ + ("direct_return", 0), + ("hidden_output", 1), + ] + assert direct.native_call_slot is None + assert hidden.native_call_slot is function.native_call_slots[hidden.bridge.abi_position] + assert hidden.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert direct.bridge.native_result_role in function.available_roles + assert hidden.bridge.native_result_role in function.available_roles + + +def test_multiple_scalar_results_lower_to_binding_tuple_and_one_bridge_function_call(): + artifacts = WrapperCodeGenerator().generate(_multiple_result_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "int32_t bind_c_with_scalar(int32_t * n, int32_t * status);" in c_source + assert "result = bind_c_with_scalar(&n, &status);" in c_source + assert "PyObject * result_0_obj = Int32_to_PyLong(&result);" in c_source + assert "PyObject * result_1_obj = Int32_to_PyLong(&status);" in c_source + assert "PyObject * result_obj = PyTuple_New(2);" in c_source + assert "PyTuple_SET_ITEM(result_obj, 0, result_0_obj);" in c_source + assert "PyTuple_SET_ITEM(result_obj, 1, result_1_obj);" in c_source + assert "Py_DECREF(result_0_obj);" in c_source + + assert 'function bind_c_with_scalar(n, status) result(result) bind(c, name="bind_c_with_scalar")' in bridge_source + assert "result = native_with_scalar(n, status)" in bridge_source + assert "PyTuple" not in bridge_source + + +def test_multiple_scalar_result_validation_rejects_position_and_consumer_drift(): + plan = _multiple_result_plan() + function = plan.namespaces[0].functions[0] + _direct, hidden = function.results + + hidden.result_position = 0 + with pytest.raises(ValueError, match=r"duplicate-binding-result-position.*missing-binding-result-position"): + WrapperCodeGenerator().generate(plan) + + plan = _multiple_result_plan() + function = plan.namespaces[0].functions[0] + direct, _hidden = function.results + function.results = (direct,) + with pytest.raises(ValueError, match="unclaimed-native-result"): + WrapperCodeGenerator().generate(plan) + + plan = _multiple_result_plan() + function = plan.namespaces[0].functions[0] + direct, hidden = function.results + duplicate = replace(hidden, owner_path=f"{hidden.owner_path}.duplicate", result_position=2) + function.results = (direct, hidden, duplicate) + with pytest.raises(ValueError, match="multiple-native-result-consumers"): + WrapperCodeGenerator().generate(plan) + + plan = _multiple_result_plan() + function = plan.namespaces[0].functions[0] + _direct, hidden = function.results + hidden.native_call_slot = replace(hidden.native_call_slot) + with pytest.raises(ValueError, match="inconsistent-function-result-slot"): + WrapperCodeGenerator().generate(plan) diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 9da457014..646f2ad09 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -83,6 +83,7 @@ "scalar-raw-address-inputs", "scalar-direct-results", "scalar-hidden-outputs", + "scalar-multiple-results", "scalar-optional-inputs", "scalar-descriptor-inputs", "scalar-writebacks", @@ -115,6 +116,8 @@ "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", + "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" + "test_multiple_scalar_results_match_both_routes_without_array_blockers", ) diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index bbf22b06c..7af88a64f 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -673,6 +673,7 @@ def _native_array_handle_blocker( def _complete_callable_address_policy(function: models.SemanticFunction) -> None: """Validate Python/native address boundaries and complete scalar projections.""" + _complete_hidden_scalar_output_projections(function) visible_scalar_names = { argument.name for argument in function.arguments if _is_visible_extent_source(argument.semantic_type) } @@ -693,6 +694,20 @@ def _complete_callable_address_policy(function: models.SemanticFunction) -> None _complete_native_address_projections(function) +def _complete_hidden_scalar_output_projections(function: models.SemanticFunction) -> None: + """Complete every primitive hidden ``Return(...)`` as address storage.""" + arguments_by_name = {argument.name: argument for argument in function.arguments} + for mapping in function.projection: + if mapping.python_position is not None or mapping.result_position is None: + continue + if mapping.value_kind: + continue + argument = arguments_by_name.get(mapping.python_name) + if argument is None or not _is_primitive_scalar_value(argument.semantic_type, allow_completed_projection=True): + continue + _apply_scalar_address_projection(argument) + + def _complete_native_address_projections(function: models.SemanticFunction) -> None: arguments_by_name = {argument.name: argument for argument in function.arguments} for mapping in function.projection: diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index 3ec1addd8..b34269aca 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -231,7 +231,7 @@ class FunctionWrapperPolicy: status_error: NativeStatusErrorPolicy | None supported: bool arguments: tuple[ArgumentPolicy, ...] = () - result: ResultPolicy | None = None + results: tuple[ResultPolicy, ...] = () native_call_slots: tuple[NativeCallSlotPolicy, ...] = () blockers: tuple[str, ...] = () writeback_actions: tuple[LifecyclePolicy, ...] = () @@ -321,7 +321,7 @@ def build_function_wrapper_policy( argument_native_positions, native_call_slots, ) - result, result_blockers = _result_policy(function, owner_path) + results, result_blockers = _result_policies(function, owner_path) writeback_actions, lifecycle_blockers = _lifecycle_policies(arguments) status_error = _completed_native_status_error_policy(function) blockers = ( @@ -343,7 +343,7 @@ def build_function_wrapper_policy( status_error=status_error, supported=not blockers, arguments=tuple(arguments), - result=result, + results=results, native_call_slots=tuple(native_call_slots), blockers=tuple(blockers), writeback_actions=writeback_actions, @@ -422,45 +422,48 @@ def _argument_policies( return policies, tuple(blockers) -def _result_policy( +def _result_policies( function: models.SemanticFunction, owner_path: str, -) -> tuple[ResultPolicy | None, tuple[str, ...]]: - hidden_results = _hidden_result_policies(function, owner_path) +) -> tuple[tuple[ResultPolicy, ...], tuple[str, ...]]: + """Return every ordered binding result consumer for one function.""" + hidden_candidates = _hidden_result_policies(function, owner_path) + hidden_results = tuple(policy for policy, _blockers in hidden_candidates if policy is not None) + hidden_blockers = tuple(reason for _policy, blockers in hidden_candidates for reason in blockers) if function.return_type is None: projected_arguments = _visible_projected_arguments(function) - if len(hidden_results) == 1 and not projected_arguments: - return hidden_results[0] - if len(projected_arguments) == 1 and not hidden_results: - return None, () + if hidden_results and not projected_arguments: + return hidden_results, (*hidden_blockers, *_result_position_blockers(hidden_results)) + if projected_arguments and not hidden_results: + return (), hidden_blockers if not hidden_results and not projected_arguments: - return None, () - return None, ("scalar lane requires one direct, hidden, or writeback result",) + return (), hidden_blockers + return (), (*hidden_blockers, "scalar lane cannot combine binding results with argument writeback") decision = function.metadata.get(models.RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA) if not isinstance(decision, OwnershipDecision): - return None, ("function result is missing completed ownership policy",) + return (), (*hidden_blockers, "function result is missing completed ownership policy") blockers = list(_result_blockers(function.return_type, decision)) bridge_data_action, bridge_copy_reason = _result_bridge_data_action(function.return_type) if bridge_data_action is BridgeDataAction.BLOCKED and decision.kind is not ObjectKind.SCALAR: blockers.append("result has no completed bridge data action") - if hidden_results: - blockers.append("direct scalar returns cannot share the first Phase 2B lane with hidden scalar outputs") + direct_result = ResultPolicy( + owner_path=f"{owner_path}.return", + semantic_type_name=function.return_type.name, + rank=int(function.return_type.rank or 0), + ownership=decision, + codegen_action=decision.codegen_action, + python_barrier_action=decision.python_barrier_action, + native_barrier_action=decision.native_barrier_action, + storage_mode=decision.storage_mode, + boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, + bridge_data_action=bridge_data_action, + bridge_copy_reason=bridge_copy_reason, + ) + results = (direct_result, *hidden_results) return ( - ResultPolicy( - owner_path=f"{owner_path}.return", - semantic_type_name=function.return_type.name, - rank=int(function.return_type.rank or 0), - ownership=decision, - codegen_action=decision.codegen_action, - python_barrier_action=decision.python_barrier_action, - native_barrier_action=decision.native_barrier_action, - storage_mode=decision.storage_mode, - boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, - bridge_data_action=bridge_data_action, - bridge_copy_reason=bridge_copy_reason, - ), - tuple(blockers), + results, + (*blockers, *hidden_blockers, *_result_position_blockers(results)), ) @@ -926,11 +929,21 @@ def _hidden_result_blockers( blockers.append(f"hidden result {argument.name!r} does not project a Python result") if not isinstance(mapping.native_position, int): blockers.append(f"hidden result {argument.name!r} is missing a native position") - if mapping.result_position != 0: - blockers.append(f"hidden result {argument.name!r} has result position {mapping.result_position}, not zero") + if not isinstance(mapping.result_position, int) or isinstance(mapping.result_position, bool): + blockers.append(f"hidden result {argument.name!r} has no integer result position") + elif mapping.result_position < 0: + blockers.append(f"hidden result {argument.name!r} has negative result position {mapping.result_position}") return tuple(blockers) +def _result_position_blockers(results: tuple[ResultPolicy, ...]) -> tuple[str, ...]: + """Require completed Python results to cover one contiguous order.""" + positions = tuple(result.result_position for result in results) + if sorted(positions) == list(range(len(positions))) and len(set(positions)) == len(positions): + return () + return (f"binding result positions must cover 0..{len(positions) - 1} exactly once; received {positions}",) + + def _function_shape_blockers(function: models.SemanticFunction) -> tuple[str, ...]: blockers: list[str] = [] if function.visibility != "public": diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index e3ab63ee5..289c3474c 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -67,6 +67,7 @@ class _CFunctionContext: native_outputs: dict[str, str] result_name: str | None python_result_name: str | None + python_results: dict[str, str] class CBindingGenerator(ClassVisitor): @@ -83,8 +84,8 @@ def _require_function_supported(self, function: FunctionPlan) -> None: """Reject unsupported actions and types for one binding function.""" for argument in function.arguments: self._require_argument_supported(argument) - if function.result is not None: - PrimitiveScalarTypeRegistry.type_for(function.result.semantic_type_name) + for result in function.results: + PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) for slot in function.native_call_slots: self._require_native_result_supported(function, slot) for action in function.writeback_actions: @@ -171,7 +172,7 @@ def _visit_NamespacePlan(self, plan: NamespacePlan) -> tuple[CFunction, ...]: def requires_runtime_support(self, plan: ModulePlan) -> bool: """Return whether module lowering consumes NumPy/runtime helpers.""" return bool(tuple(self._variables(plan))) or any( - function.arguments or function.result is not None for function in self._functions(plan) + function.arguments or function.results for function in self._functions(plan) ) def _module_needs_allocator(self, plan: ModulePlan) -> bool: @@ -646,61 +647,80 @@ def _visit_ResultPlan( plan: ResultPlan, *, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: + failure_cleanup: tuple[str, ...] = (), + ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: """Lower one result through its completed binding action.""" - return self._lower_result(plan, context) + return self._lower_result(plan, context, failure_cleanup) def _lower_result( self, plan: ResultPlan, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: + failure_cleanup: tuple[str, ...], + ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: """Dispatch one completed binding result action explicitly.""" action = plan.binding.codegen_action match action: case CodegenAction.DIRECT_VALUE: - return self._lower_result_direct_value(plan, context) + return self._lower_result_direct_value(plan, context, failure_cleanup) case CodegenAction.HIDDEN_OUTPUT: - return self._lower_result_hidden_output(plan, context) + return self._lower_result_hidden_output(plan, context, failure_cleanup) raise ValueError(f"Unsupported C result action for {plan.owner_path!r}: {action!r}") def _lower_result_direct_value( self, plan: ResultPlan, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - return self._lower_result_value(plan, context) + failure_cleanup: tuple[str, ...], + ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: + return self._lower_result_value(plan, context, failure_cleanup) def _lower_result_hidden_output( self, plan: ResultPlan, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - return self._lower_result_value(plan, context) + failure_cleanup: tuple[str, ...], + ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: + return self._lower_result_value(plan, context, failure_cleanup) def _lower_result_value( self, plan: ResultPlan, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return Python scalar projection after the completed native envelope.""" + failure_cleanup: tuple[str, ...], + ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: + """Convert one native result into its binding-owned Python consumer.""" scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) - if ( - scalar_type.python_result_converter is None - or context.result_name is None - or context.python_result_name is None - ): + native_name = self._result_native_name(plan, context) + python_name = context.python_results.get(plan.owner_path) + if scalar_type.python_result_converter is None or python_name is None: raise ValueError(f"Unsupported scalar result type {plan.semantic_type_name!r}") return ( CDeclaration( - context.python_result_name, + python_name, "PyObject *", - CodeExpression(f"{scalar_type.python_result_converter}(&{context.result_name})"), + CodeExpression(f"{scalar_type.python_result_converter}(&{native_name})"), + ), + CIf( + CodeExpression(f"{python_name} == NULL"), + body=( + *(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in failure_cleanup), + CReturn(CodeExpression("NULL")), + ), ), - CExpressionStatement(CodeExpression(f"if ({context.python_result_name} == NULL) return NULL")), - CReturn(CodeExpression(context.python_result_name)), ) + def _result_native_name(self, plan: ResultPlan, context: _CFunctionContext) -> str: + """Return the validated C storage consumed by one result conversion.""" + if plan.source_kind == "direct_return": + if context.result_name is None: + raise ValueError(f"Direct result {plan.owner_path!r} has no C storage") + return context.result_name + try: + return context.native_outputs[plan.bridge.native_result_role] + except KeyError: + raise ValueError(f"Hidden result {plan.owner_path!r} has no C output storage") from None + def _output_nodes( self, plan: FunctionPlan, @@ -711,22 +731,55 @@ def _output_nodes( *self._lower_native_call(plan, self._bridge_call_statement(plan, context)), *self._lower_status_error(plan, context), ] - if plan.result is not None: - nodes.extend(self.visit(plan.result, context=context)) + if plan.results: + nodes.extend(self._binding_result_nodes(plan, context)) elif plan.writeback_actions: nodes.extend(self._writeback_nodes(plan, context)) else: nodes.append(CExpressionStatement(CodeExpression("Py_RETURN_NONE"))) return tuple(nodes) + def _binding_result_nodes( + self, + plan: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: + """Convert ordered results and assemble a tuple only in the binding.""" + ordered = tuple(sorted(plan.results, key=lambda item: item.result_position)) + converted = [] + nodes = [] + for result in ordered: + nodes.extend(self.visit(result, context=context, failure_cleanup=tuple(converted))) + converted.append(context.python_results[result.owner_path]) + if len(converted) == 1: + nodes.append(CReturn(CodeExpression(converted[0]))) + return tuple(nodes) + aggregate = context.python_result_name + if aggregate is None: + raise ValueError(f"{plan.owner_path!r} multiple results have no aggregate binding role") + nodes.extend( + ( + CDeclaration(aggregate, "PyObject *", CodeExpression(f"PyTuple_New({len(converted)})")), + CIf( + CodeExpression(f"{aggregate} == NULL"), + body=( + *(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in converted), + CReturn(CodeExpression("NULL")), + ), + ), + *( + CExpressionStatement(CodeExpression(f"PyTuple_SET_ITEM({aggregate}, {position}, {name})")) + for position, name in enumerate(converted) + ), + CReturn(CodeExpression(aggregate)), + ) + ) + return tuple(nodes) + def _bridge_call_statement(self, plan: FunctionPlan, context: _CFunctionContext) -> CExpressionStatement: """Return the mechanical bridge call selected by result storage.""" call = self._bridge_call(plan, context) - expression = ( - f"{context.result_name} = {call}" - if plan.result is not None and plan.result.source_kind == "direct_return" - else call - ) + expression = f"{context.result_name} = {call}" if self._direct_result(plan) is not None else call return CExpressionStatement(CodeExpression(expression)) def _lower_native_call( @@ -911,15 +964,14 @@ def _function_context(self, plan: FunctionPlan) -> _CFunctionContext: for slot in plan.native_call_slots if slot.source_kind == "result" } - if plan.result is None: - python_result = "result_obj" if plan.writeback_actions else None - return _CFunctionContext(arguments, native_outputs, None, python_result) - base = ( - native_outputs[plan.result.bridge.native_result_role] - if plan.result.source_kind == "hidden_output" - else "result" - ) - return _CFunctionContext(arguments, native_outputs, base, "result_obj") + ordered_results = tuple(sorted(plan.results, key=lambda item: item.result_position)) + python_results = { + result.owner_path: ("result_obj" if len(ordered_results) == 1 else f"result_{result.result_position}_obj") + for result in ordered_results + } + python_result = "result_obj" if ordered_results or plan.writeback_actions else None + native_result = "result" if self._direct_result(plan) is not None else None + return _CFunctionContext(arguments, native_outputs, native_result, python_result, python_results) def _keyword_declaration(self, plan: FunctionPlan) -> CDeclaration: keywords = ", ".join( @@ -945,9 +997,10 @@ def _direct_result_declaration( plan: FunctionPlan, context: _CFunctionContext, ) -> tuple[CDeclaration, ...]: - if plan.result is None or plan.result.source_kind != "direct_return" or context.result_name is None: + result = self._direct_result(plan) + if result is None or context.result_name is None: return () - scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name) + scalar_type = PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) return (CDeclaration(context.result_name, scalar_type.c_spelling),) def _native_output_declarations( @@ -1014,11 +1067,14 @@ def _bridge_prototype(self, plan: FunctionPlan) -> CFunctionPrototype: def _bridge_return_type(self, plan: FunctionPlan) -> str: """Return the direct bridge result type, or void for subroutines.""" - if plan.result is None: + result = self._direct_result(plan) + if result is None: return "void" - if plan.result.source_kind != "direct_return": - return "void" - return PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).c_spelling + return PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name).c_spelling + + def _direct_result(self, plan: FunctionPlan) -> ResultPlan | None: + """Return the sole direct native function result, when present.""" + return next((result for result in plan.results if result.source_kind == "direct_return"), None) def _bridge_argument_parameters(self, argument: ArgumentTransferPlan) -> tuple[CParameter, ...]: """Return the bridge ABI parameters for one Python argument.""" diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index 3d9f9bfb7..ab6680027 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -31,6 +31,7 @@ ModuleVariablePlan, NamespacePlan, NativeCallSlotPlan, + ResultPlan, ) from x2py.wrapper_codegen.primitive_scalar_types import PrimitiveScalarTypeRegistry from x2py.wrapper_codegen.visitor import ClassVisitor @@ -176,14 +177,13 @@ def _lower_result( plan: FunctionPlan, ) -> tuple[str | None, str | None]: """Dispatch one completed bridge result action explicitly.""" - if plan.result is None: + result = self._direct_result(plan) + if result is None: return self._lower_result_none(plan) - action = plan.result.bridge.codegen_action + action = result.bridge.codegen_action match action: case CodegenAction.DIRECT_VALUE: - return self._lower_result_direct_value(plan) - case CodegenAction.HIDDEN_OUTPUT: - return self._lower_result_hidden_output(plan) + return self._lower_result_direct_value(plan, result) raise ValueError(f"Unsupported Fortran result action for {plan.owner_path!r}: {action!r}") def _lower_result_none( @@ -196,16 +196,10 @@ def _lower_result_none( def _lower_result_direct_value( self, plan: FunctionPlan, + result: ResultPlan, ) -> tuple[str | None, str | None]: """Return the procedure shape of a direct native function result.""" - return "result", self._bridge_result_type(plan) - - def _lower_result_hidden_output( - self, - plan: FunctionPlan, - ) -> tuple[str | None, str | None]: - """Return the procedure shape of a hidden native output parameter.""" - return None, None + return "result", self._bridge_result_type(plan, result) def _visit_ModuleVariablePlan(self, plan: ModuleVariablePlan) -> tuple[FortranFunction, ...]: """Lower bridge-owned getter and setter actions into procedures.""" @@ -730,10 +724,15 @@ def _string_output_length(self, slot: NativeCallSlotPlan) -> int: raise ValueError(f"String output {slot.owner_path!r} is missing a fixed character length") return slot.character_length - def _bridge_result_type(self, plan: FunctionPlan) -> str: - if plan.result is None: + def _bridge_result_type(self, plan: FunctionPlan, result: ResultPlan | None = None) -> str: + result = result or self._direct_result(plan) + if result is None: raise ValueError(f"{plan.owner_path!r} native function has no result plan") - return PrimitiveScalarTypeRegistry.type_for(plan.result.semantic_type_name).fortran_spelling + return PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name).fortran_spelling + + def _direct_result(self, plan: FunctionPlan) -> ResultPlan | None: + """Return the sole direct result used by the Fortran function ABI.""" + return next((result for result in plan.results if result.source_kind == "direct_return"), None) def _native_module_uses(self, plan: ModulePlan) -> tuple[FortranUse, ...]: modules: dict[str, list[str]] = {} @@ -791,9 +790,10 @@ def _external_interface_procedure(self, plan: FunctionPlan) -> FortranInterfaceP ) imports = tuple(dict.fromkeys(self._iso_symbol(argument.semantic_type_name) for argument in plan.arguments)) result_name = None if plan.bridge.native_is_subroutine else "native_result" - result_type = self._bridge_result_type(plan) if result_name is not None else None - if result_type is not None and plan.result is not None: - imports = tuple(dict.fromkeys((*imports, self._iso_symbol(plan.result.semantic_type_name)))) + direct_result = self._direct_result(plan) + result_type = self._bridge_result_type(plan, direct_result) if result_name is not None else None + if result_type is not None and direct_result is not None: + imports = tuple(dict.fromkeys((*imports, self._iso_symbol(direct_result.semantic_type_name)))) return FortranInterfaceProcedure( name=plan.bridge.native_name, imports=imports, diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index 709a5b90f..e8cfb6014 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -290,8 +290,8 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost diagnostics.extend(self._native_slot_diagnostics(slot)) for argument in plan.arguments: diagnostics.extend(self._argument_diagnostics(argument, slots)) - if plan.result is not None: - diagnostics.extend(self._result_diagnostics(plan.result, slots, plan.available_roles)) + for result in plan.results: + diagnostics.extend(self._result_diagnostics(result, slots, plan.available_roles)) for action in (*plan.writeback_actions, *plan.cleanup_actions, *plan.release_actions): diagnostics.extend(self._lifecycle_diagnostics(action, plan.available_roles)) diagnostics.extend(self._writeback_phase_diagnostics(plan)) @@ -363,7 +363,7 @@ def _argument_slot_consistency_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-native-position", plan.native_position)) if plan.native_call_slot.python_position != plan.python_position: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-python-position", plan.python_position)) - if function_slots.get(plan.native_position) != plan.native_call_slot: + if function_slots.get(plan.native_position) is not plan.native_call_slot: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-function-native-slot", plan.native_position) ) @@ -544,7 +544,7 @@ def _hidden_result_diagnostics( *self._hidden_result_shape_diagnostics(plan, slot), *self._hidden_result_policy_consistency_diagnostics(plan, slot), ] - if function_slots.get(slot.native_position) != slot: + if function_slots.get(slot.native_position) is not slot: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-function-result-slot", slot.native_position) ) @@ -756,6 +756,7 @@ def _writeback_phase_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanD def _function_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Return output projection and native callable-kind diagnostics.""" diagnostics = [*self._mixed_output_diagnostics(plan)] + diagnostics.extend(self._binding_result_diagnostics(plan)) diagnostics.extend(self._writeback_result_diagnostics(plan)) diagnostics.extend(self._native_callable_kind_diagnostics(plan)) diagnostics.extend(self._unclaimed_result_diagnostics(plan)) @@ -763,10 +764,25 @@ def _function_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanD def _mixed_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Reject simultaneous public result and writeback projections.""" - if plan.result is not None and plan.writeback_actions: + if plan.results and plan.writeback_actions: return (self._diagnostic(plan.owner_path, "mixed-result-and-writeback", plan.owner_path),) return () + def _binding_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate ordered consumers and the sole direct native result.""" + diagnostics = list( + self._sequence_diagnostics( + plan.owner_path, + "binding-result", + tuple(result.result_position for result in plan.results), + len(plan.results), + ) + ) + direct_results = tuple(result for result in plan.results if result.source_kind == "direct_return") + if len(direct_results) > 1: + diagnostics.append(self._diagnostic(plan.owner_path, "multiple-direct-results", len(direct_results))) + return tuple(diagnostics) + def _writeback_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Validate contiguous Python result positions for copy-out actions.""" result_positions = tuple( @@ -783,29 +799,36 @@ def _writeback_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlan def _native_callable_kind_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Require callable kind to agree with the public result representation.""" - requires_subroutine = plan.result is None or plan.result.source_kind == "hidden_output" + requires_subroutine = not any(result.source_kind == "direct_return" for result in plan.results) if plan.bridge.native_is_subroutine != requires_subroutine: return (self._diagnostic(plan.owner_path, "inconsistent-native-callable-kind", requires_subroutine),) return () def _unclaimed_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: - """Require every native result slot to have an explicit binding consumer.""" + """Require exactly one binding or status consumer per native output.""" claimed_roles = self._claimed_result_roles(plan) - return tuple( - self._diagnostic(plan.owner_path, "unclaimed-native-result", slot.symbolic_role) - for slot in plan.native_call_slots - if slot.source_kind == "result" and slot.symbolic_role not in claimed_roles - ) + diagnostics = [] + for slot in plan.native_call_slots: + if slot.source_kind != "result": + continue + claim_count = claimed_roles[slot.symbolic_role] + if claim_count == 0: + diagnostics.append(self._diagnostic(plan.owner_path, "unclaimed-native-result", slot.symbolic_role)) + elif claim_count > 1: + diagnostics.append( + self._diagnostic(plan.owner_path, "multiple-native-result-consumers", slot.symbolic_role) + ) + return tuple(diagnostics) - def _claimed_result_roles(self, plan: FunctionPlan) -> set[str]: + def _claimed_result_roles(self, plan: FunctionPlan) -> Counter[str]: """Return public and status-policy consumers of native result slots.""" - roles = set() - if plan.result is not None and plan.result.source_kind == "hidden_output": - roles.add(plan.result.bridge.native_result_role) + roles = Counter( + result.bridge.native_result_role for result in plan.results if result.source_kind == "hidden_output" + ) if plan.binding.status_error is not None: - roles.add(plan.binding.status_error.status_role) + roles[plan.binding.status_error.status_role] += 1 if plan.binding.status_error.message_role is not None: - roles.add(plan.binding.status_error.message_role) + roles[plan.binding.status_error.message_role] += 1 return roles def _status_error_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: @@ -890,8 +913,9 @@ def _duplicate_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDi roles = [argument.binding.handoff_role for argument in plan.arguments] roles.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "literal") roles.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "result") - if plan.result is not None and plan.result.source_kind == "direct_return": - roles.append(plan.result.bridge.native_result_role) + roles.extend( + result.bridge.native_result_role for result in plan.results if result.source_kind == "direct_return" + ) return tuple( self._diagnostic(plan.owner_path, "duplicate-symbolic-role", role) for role, count in Counter(roles).items() @@ -902,8 +926,9 @@ def _available_role_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDi """Require the advertised roles to match argument and result producers.""" expected = [argument.binding.handoff_role for argument in plan.arguments] expected.extend(slot.symbolic_role for slot in plan.native_call_slots if slot.source_kind == "result") - if plan.result is not None and plan.result.source_kind == "direct_return": - expected.append(plan.result.bridge.native_result_role) + expected.extend( + result.bridge.native_result_role for result in plan.results if result.source_kind == "direct_return" + ) if Counter(plan.available_roles) != Counter(expected): return (self._diagnostic(plan.owner_path, "inconsistent-available-roles", plan.available_roles),) return () diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py index 830856631..f01080aa5 100644 --- a/x2py/wrapper_codegen/plan.py +++ b/x2py/wrapper_codegen/plan.py @@ -254,7 +254,7 @@ class FunctionPlan(StageRecord): binding: BindingFunctionPlan bridge: BridgeFunctionPlan arguments: tuple[ArgumentTransferPlan, ...] - result: ResultPlan | None + results: tuple[ResultPlan, ...] native_call_slots: tuple[NativeCallSlotPlan, ...] available_roles: tuple[str, ...] writeback_actions: tuple[LifecycleActionPlan, ...] = () diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py index a80abfaee..97d33dab2 100644 --- a/x2py/wrapper_codegen/planner.py +++ b/x2py/wrapper_codegen/planner.py @@ -186,11 +186,11 @@ def _function_plan( ) -> FunctionPlan: """Return one exported function plan from completed policy.""" native_call_slots = tuple( - self._native_slot_plan(slot, self._native_slot_role(slot, policy.result)) + self._native_slot_plan(slot, self._native_slot_role(slot, policy.results)) for slot in policy.native_call_slots ) arguments = self._argument_plans(policy, native_call_slots) - result = self._result_plan(policy, native_call_slots) + results = self._result_plans(policy, native_call_slots) return FunctionPlan( owner_path=self._export_owner_path(module_name, export.namespace, export.name), symbol_name=export.name.casefold(), @@ -206,9 +206,9 @@ def _function_plan( policy.native_is_subroutine, ), arguments=arguments, - result=result, + results=results, native_call_slots=native_call_slots, - available_roles=self._available_roles(arguments, result, native_call_slots), + available_roles=self._available_roles(arguments, results, native_call_slots), writeback_actions=tuple(self.visit(action) for action in policy.writeback_actions), cleanup_actions=tuple(self.visit(action) for action in policy.cleanup_actions), release_actions=tuple(self.visit(action) for action in policy.release_actions), @@ -228,17 +228,18 @@ def _argument_plans( for argument in policy.arguments ) - def _result_plan( + def _result_plans( self, policy: FunctionWrapperPolicy, native_call_slots: tuple[NativeCallSlotPlan, ...], - ) -> ResultPlan | None: - """Return one completed result plan when the function has a result.""" - if policy.result is None: - return None - return self.visit( - policy.result, - native_slot=self._result_native_slot(policy, native_call_slots), + ) -> tuple[ResultPlan, ...]: + """Return ordered result consumers sharing completed native slots.""" + return tuple( + self.visit( + result, + native_slot=self._result_native_slot(result, native_call_slots), + ) + for result in sorted(policy.results, key=lambda item: item.result_position) ) def _visit_ArgumentPolicy( @@ -397,37 +398,33 @@ def _planned_native_slot( def _result_native_slot( self, - function_policy: FunctionWrapperPolicy, + result_policy: ResultPolicy, native_call_slots: tuple[NativeCallSlotPlan, ...], ) -> NativeCallSlotPlan | None: """Return the completed slot for one hidden result, if any.""" - if function_policy.result is None or function_policy.result.source_kind != "hidden_output": + if result_policy.source_kind != "hidden_output": return None - return self._planned_native_slot(native_call_slots, function_policy.result.owner_path) + return self._planned_native_slot(native_call_slots, result_policy.owner_path) def _available_roles( self, arguments: tuple[ArgumentTransferPlan, ...], - result: ResultPlan | None, + results: tuple[ResultPlan, ...], native_call_slots: tuple[NativeCallSlotPlan, ...], ) -> tuple[str, ...]: """Return symbolic roles available after the native call.""" roles = [argument.binding.handoff_role for argument in arguments] roles.extend(self._native_result_roles(native_call_slots)) - roles.extend(self._direct_result_roles(result)) + roles.extend(self._direct_result_roles(results)) return tuple(dict.fromkeys(roles)) def _native_result_roles(self, native_call_slots: tuple[NativeCallSlotPlan, ...]) -> tuple[str, ...]: """Return every role produced through a native result slot.""" return tuple(slot.symbolic_role for slot in native_call_slots if slot.source_kind == "result") - def _direct_result_roles(self, result: ResultPlan | None) -> tuple[str, ...]: - """Return the direct-return role when the callable produces one.""" - if result is None: - return () - if result.source_kind != "direct_return": - return () - return (result.bridge.native_result_role,) + def _direct_result_roles(self, results: tuple[ResultPlan, ...]) -> tuple[str, ...]: + """Return direct-return roles produced by the bridge function result.""" + return tuple(result.bridge.native_result_role for result in results if result.source_kind == "direct_return") def _datatype_family(self, semantic_type_name: str) -> DatatypeFamily: """Copy the backend-relevant family of one supported semantic type.""" @@ -470,27 +467,31 @@ def _value_role(self, owner_path: str) -> str: def _native_slot_role( self, native_slot: NativeCallSlotPolicy, - result: ResultPolicy | None, + results: tuple[ResultPolicy, ...], ) -> str: """Return the symbolic role for one native-call slot.""" if native_slot.source_kind == "literal": return f"{native_slot.owner_path}:literal" - if self._is_public_result_slot(native_slot, result): - return f"{result.owner_path}:native-result" + public_result = self._public_result_for_slot(native_slot, results) + if public_result is not None: + return f"{public_result.owner_path}:native-result" if native_slot.source_kind == "result": return f"{native_slot.owner_path}:native-result" return self._value_role(native_slot.owner_path) - def _is_public_result_slot( + def _public_result_for_slot( self, native_slot: NativeCallSlotPolicy, - result: ResultPolicy | None, - ) -> bool: - """Return whether a native slot carries the Python-visible hidden result.""" + results: tuple[ResultPolicy, ...], + ) -> ResultPolicy | None: + """Return the Python-visible hidden result carried by one native slot.""" if native_slot.source_kind != "result": - return False - if result is None: - return False - if result.source_kind != "hidden_output": - return False - return result.owner_path == native_slot.owner_path + return None + return next( + ( + result + for result in results + if result.source_kind == "hidden_output" and result.owner_path == native_slot.owner_path + ), + None, + ) diff --git a/x2py/wrapper_codegen/support.py b/x2py/wrapper_codegen/support.py index a6fbf7a93..8ed18c9e2 100644 --- a/x2py/wrapper_codegen/support.py +++ b/x2py/wrapper_codegen/support.py @@ -112,12 +112,14 @@ def _output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: lanes = [] if policy.writeback_actions: lanes.append("scalar-writebacks") - result_lane = "scalar-direct-results" - if policy.result is not None and policy.result.source_kind == "hidden_output": - result_lane = "scalar-hidden-outputs" - if policy.result is not None: - lanes.append(result_lane) - if policy.result is None and not policy.writeback_actions: + source_kinds = {result.source_kind for result in policy.results} + if "direct_return" in source_kinds: + lanes.append("scalar-direct-results") + if "hidden_output" in source_kinds: + lanes.append("scalar-hidden-outputs") + if len(policy.results) > 1: + lanes.append("scalar-multiple-results") + if not policy.results and not policy.writeback_actions: lanes.append("void-calls") return tuple(lanes) From 7f239eb4405f76e811bf5f9c9a96a90aeaf5043a Mon Sep 17 00:00:00 2001 From: said Date: Mon, 13 Jul 2026 20:58:14 +0100 Subject: [PATCH 09/30] add strings and arrays --- .../wrapper-generation-pipeline.md | 170 ++- .../wrapper-plan-migration-checklist.md | 635 ++++++++- .../test_binding_handle_policy_dispatch.py | 3 +- .../test_wrapper_plan_route_selection.py | 135 +- .../policy/test_native_array_ownership.py | 4 +- .../test_policy_defaults_and_validation.py | 6 +- tests/semantics/policy/test_wrapper_policy.py | 214 ++- .../fortran/arrays/test_array_results.py | 53 + .../arrays/test_assumed_rank_arrays.py | 49 +- .../arrays/test_multidimensional_arrays.py | 66 +- .../test_native_order_contracts.py | 61 + .../function_calls/test_optional_arguments.py | 59 + .../function_calls/test_output_arguments.py | 45 + .../fortran/scalars/test_verified_baseline.py | 58 + .../strings/test_character_arguments.py | 148 +- .../strings/test_character_edge_cases.py | 142 ++ .../wrapper_codegen/test_phase0d_plan_core.py | 44 +- .../test_phase0e_backend_foundation.py | 19 + .../test_phase1a_wrapper_assembly.py | 6 +- .../test_phase2f_multiple_scalar_results.py | 4 +- .../test_phase5a_string_inputs.py | 108 ++ .../test_phase5b_fixed_string_results.py | 200 +++ .../test_phase5c_fixed_string_writeback.py | 256 ++++ .../test_phase5d_string_addresses.py | 162 +++ .../test_phase6a_array_buffers.py | 113 ++ .../test_phase6b_dense_array_shapes.py | 70 + .../test_phase6c_strided_arrays.py | 66 + .../test_phase6d_array_output_identity.py | 55 + .../test_phase6e_array_results.py | 99 ++ ...ase6f_optional_assumed_character_arrays.py | 131 ++ x2py/codegen/bindings/c_to_python.py | 25 +- x2py/codegen/bridges/fortran_to_c.py | 22 +- x2py/codegen/printers/fcode.py | 1 - x2py/pipeline/build.py | 40 + x2py/semantics/ownership.py | 37 +- x2py/semantics/policy_completion.py | 11 +- x2py/semantics/wrapper_policy.py | 696 ++++++++- x2py/wrapper_codegen/__init__.py | 6 + x2py/wrapper_codegen/c/binding.py | 1262 +++++++++++++++-- x2py/wrapper_codegen/fortran/bridge.py | 1016 ++++++++++++- x2py/wrapper_codegen/generator.py | 1014 ++++++++++++- x2py/wrapper_codegen/nodes.py | 21 +- x2py/wrapper_codegen/plan.py | 68 +- x2py/wrapper_codegen/planner.py | 135 +- x2py/wrapper_codegen/source_printers.py | 129 +- x2py/wrapper_codegen/support.py | 163 ++- 46 files changed, 7472 insertions(+), 355 deletions(-) create mode 100644 tests/wrapper_codegen/test_phase5a_string_inputs.py create mode 100644 tests/wrapper_codegen/test_phase5b_fixed_string_results.py create mode 100644 tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py create mode 100644 tests/wrapper_codegen/test_phase5d_string_addresses.py create mode 100644 tests/wrapper_codegen/test_phase6a_array_buffers.py create mode 100644 tests/wrapper_codegen/test_phase6b_dense_array_shapes.py create mode 100644 tests/wrapper_codegen/test_phase6c_strided_arrays.py create mode 100644 tests/wrapper_codegen/test_phase6d_array_output_identity.py create mode 100644 tests/wrapper_codegen/test_phase6e_array_results.py create mode 100644 tests/wrapper_codegen/test_phase6f_optional_assumed_character_arrays.py diff --git a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md index 6292c4a64..edfa1269e 100644 --- a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md +++ b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md @@ -2,19 +2,169 @@ title: Wrapper Generation Pipeline audience: maintainers prerequisites: semantic passes, code generation design -related: runtime-layer.md, ownership-tracking.md -status: planned-documentation +related: runtime-layer.md, ownership-tracking.md, ../roadmap/wrapper-plan-migration-checklist.md +status: maintained --- # Wrapper Generation Pipeline - +This page describes the direct wrapper-plan route through Phase 6. It covers +primitive scalars, scalar strings, ordinary arrays, and the supported +module-variable surface. Native allocatable and pointer handles belong to Phase +7 and are intentionally outside this contract. -## TODO +## Architectural Boundary -- TODO: Document the full generated bridge and binding pipeline with file - ownership. -- TODO: Link feature support to tests that exercise generated runtime behavior. +All semantic policy must be complete before wrapper planning or +`x2py/semantics/ir2ast.py` lowering begins. Post-IR policy completion owns +object kind, ownership, transfer, destruction, mutability, writeback, +nullability, output projection, release responsibility, storage mode, getter +behavior, native setter assignment, and Python setter exposure. + +Planning projects those completed decisions into one editable `ModulePlan`. +Validation checks that the projections agree. Binding and bridge generation +then dispatch only from completed selectors into small named lowering methods; +they do not reconstruct policy from datatype, `intent`, shape, alias flags, or +local memory checks. + +The public direct-generation boundary is: + +```python +complete_semantic_policies(module) +plan = WrapperPlanner().build(module) +artifacts = WrapperCodeGenerator().generate(plan) +``` + +`WrapperCodeGenerator.generate()` freezes the plan, runs the shared validator, +runs both backend preflight checks, lowers recursively to C and Fortran syntax +nodes, and asks the source printers to render those nodes. Build integration +compiles the rendered sources; it does not own datatype transfer policy. + +During the migration, route selection may still choose the legacy generators +for unsupported functions. The legacy mappings in `x2py/codegen/` consume the +same completed semantic action enums, but they are not dependencies of +`x2py/wrapper_codegen/`. They remain only until the final route cutover. + +## Stable Tree and Datatype-Varying Records + +The shared plan has stable module, namespace, and function orchestration: + +```text +ModulePlan + binding: BindingModulePlan + bridge: BridgeModulePlan + namespaces: NamespacePlan ... + functions: FunctionPlan ... + binding: BindingFunctionPlan + bridge: BridgeFunctionPlan + arguments: ArgumentTransferPlan ... + binding: BindingArgumentPlan + bridge: BridgeArgumentPlan + native_call_slot: NativeCallSlotPlan + results: ResultPlan ... + binding: BindingResultPlan + bridge: BridgeResultPlan + native_call_slot: NativeCallSlotPlan | None + native_call_slots: NativeCallSlotPlan ... + lifecycle actions: LifecycleActionPlan ... + variables: ModuleVariablePlan ... +``` + +Most datatype-specific work belongs to `ArgumentTransferPlan` and +`ResultPlan`. Each is one transfer with explicit binding and bridge views. +`ModuleVariablePlan` is the other intentionally datatype-sensitive surface, +because getter, setter, and native assignment behavior depend on the stored +value. + +`FunctionPlan`, `NamespacePlan`, and `ModulePlan` remain orchestration records. +They own export names, call order, result order, runtime/GIL envelopes, and +aggregation, but not datatype policy. + +`NativeCallSlotPlan` and `LifecycleActionPlan` are subordinate transfer +details. Native slots stay indexed on `FunctionPlan` because native ABI order +can interleave argument slots, result slots, literals, and helpers. Lifecycle +actions stay indexed there because copy-out, cleanup, and release order may +span several arguments and results or differ on failure. Argument and hidden +result slots are the same mutable records referenced from both their transfer +owner and the function-wide index; they are not duplicated policy. + +## One Repeatable Transfer Algorithm + +Use this sequence for scalars, strings, arrays, and future datatype families: + +1. Post-IR policy completion classifies the value with `ObjectKind` and + completes ownership, transfer, storage, nullability, mutability, projection, + barrier actions, data action, and any justified copy reason. +2. Wrapper policy records the backend-neutral transfer and the ordered native + slot. It must report a blocker instead of leaving a semantic choice for a + backend. +3. `WrapperPlanner` mechanically projects one `ArgumentTransferPlan` or + `ResultPlan`, adds symbolic handoff roles, and shares the corresponding + `NativeCallSlotPlan` reference. +4. The shared validator checks graph consistency and common invariants, then + dispatches by the completed `object_kind` to scalar, string, or ordinary- + array validation. +5. Backend preflight dispatches by the same completed kind and action selectors + and rejects combinations it cannot lower. +6. The binding lowers Python extraction or result construction. The bridge + lowers ABI declarations, representation conversion, the ordered native + call, and native result production. Both communicate through planned + symbolic roles. +7. Function-level orchestration applies status handling and ordered lifecycle + actions, aggregates Python results, and returns. Printers and build + integration remain generic. + +When adding a datatype, first extend semantic policy and its transfer record, +then add one named validator and one named lowering method per affected +backend. Do not add a parallel plan hierarchy or datatype branches to module, +namespace, or function traversal. Add a new typed action only when the existing +selectors cannot express a genuine semantic choice. + +## Selector Vocabulary + +The action axes are deliberately orthogonal: + +| Selector | Question answered | Examples | +| --- | --- | --- | +| `ObjectKind` | What kind of object follows this route? | `SCALAR`, `STRING`, `NUMPY_ARRAY` | +| `source_kind` | Where is a result produced? | `direct_return`, `hidden_output` | +| `PythonBarrierAction` | How does the binding cross the Python boundary? | `SCALAR_VALUE`, `STRING_VALUE`, `ARRAY_STORAGE` | +| `NativeBarrierAction` | What native ABI transport is used? | `PASS_VALUE`, `PASS_CALL_LOCAL_ADDRESS`, `PASS_ARRAY_BUFFER` | +| `CodegenAction` | What ownership or transfer operation occurs? | `DIRECT_VALUE`, `CALL_LOCAL_INPUT`, `COPY_IN_OUT`, `COPY_OUT` | +| `BridgeDataAction` | What happens to the representation in the bridge? | `DIRECT_TRANSFER`, `ASSOCIATE_VIEW`, `COPY_REPRESENTATION` | +| `WritebackPhase` | When does a lifecycle operation run? | native mutation, copy-out, cleanup, release | + +Hiddenness is not a transfer operation. A hidden scalar result therefore uses +`source_kind="hidden_output"` with `CodegenAction.DIRECT_VALUE`; hidden strings +and ordinary arrays use the same source kind with `CodegenAction.COPY_OUT`. + +`NativeBarrierAction.PASS_ARRAY_BUFFER` identifies the Phase 6 ordinary-array +data-buffer ABI. Its handoff plan carries data, rank, extents, strides, and +itemsize. `PASS_NATIVE_DESCRIPTOR` is reserved for Phase 7 persistent native +descriptors and handles. Neither backend may substitute one for the other. + +`DatatypeFamily` remains useful after object-kind dispatch for primitive +element spelling and conversion, such as integer versus real scalar types or +the element type of an ordinary array. It must not be used to rediscover +whether the transfer itself is a scalar, string, or array. + +## Maintainer Inspection and Acceptance + +Inspect the real records directly with normal Python prints. The primary path +is `complete_semantic_policies()` -> `WrapperPlanner.build()` -> +`WrapperCodeGenerator.generate()`. Generated artifacts from real passing +`tests/wrapper` cases are the behavioral oracle; plan unit tests cover selector +and graph invariants, while dual-route runtime tests prove legacy/direct-plan +parity. + +A Phase 5 or Phase 6 change is acceptable when: + +- semantic decisions are complete before planning; +- datatype variation is confined to transfer, result, lifecycle, or + module-variable records and their named handlers; +- scalar, string, and array routes use the same planning and validation + sequence; +- binding and bridge consume the same shared roles and native-slot records; +- no backend infers policy or silently falls back to another action; +- focused plan tests, relevant wrapper runtime tests, documentation checks, and + static analysis pass. diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 98d4d012a..5e96c06b7 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -69,12 +69,16 @@ ModulePlan arguments: ArgumentTransferPlan ... binding: BindingArgumentPlan bridge: BridgeArgumentPlan - result: ResultPlan | None + native_call_slot: NativeCallSlotPlan + results: ResultPlan ... binding: BindingResultPlan bridge: BridgeResultPlan + native_call_slot: NativeCallSlotPlan | None lifecycle: LifecycleActionPlan ... binding: BindingLifecyclePlan | None bridge: BridgeLifecyclePlan | None + native_call_slots: ordered references to argument/result slots plus + function-owned literal or helper slots ``` `ArgumentTransferPlan` remains the only argument-owner record; do not add a @@ -92,7 +96,27 @@ deliberately distinct and directly editable: referenced from `FunctionPlan.native_call_slots`, not a copied record that a maintainer must edit twice; - result and lifecycle records identify later producers, consumers, ordering, - and responsibility through their own binding and bridge views. + and responsibility through their own binding and bridge views; +- native slots and lifecycle actions are subordinate transfer details, not + parallel datatype-policy systems. They remain indexed on `FunctionPlan` + because native ABI order and success/failure lifecycle order may span more + than one argument or result. A function-owned literal, status helper, or + other ABI slot may also have no single argument/result owner. + +The action vocabulary keeps source placement, data transfer, and native ABI +transport orthogonal. `ResultPlan.source_kind` says whether a result comes from +a `direct_return` or `hidden_output`; `CodegenAction` says how the value moves +or is owned (`DIRECT_VALUE`, `COPY_OUT`, `WRAPPER_INSTANCE`, and so on). A +hidden scalar therefore remains `DIRECT_VALUE`, while hidden strings and +ordinary arrays are `COPY_OUT`; hidden descriptor-owned objects use their +completed ownership action. `HIDDEN_OUTPUT` is not a codegen action because +hiddenness is a source location, not a transfer operation. + +Likewise, `NativeBarrierAction.PASS_ARRAY_BUFFER` means the Phase 6 data-buffer +ABI whose handoff plan carries data, rank, extents, strides, and itemsize. +`NativeBarrierAction.PASS_NATIVE_DESCRIPTOR` is reserved for the persistent +native descriptors and handles introduced in Phase 7. Neither backend may use +one action as a fallback for the other. The binding input and bridge input may have different representations: a C binding commonly receives `PyObject *`, produces a C scalar or address, and @@ -326,6 +350,14 @@ Primitive dtype spelling and converter differences live in the intentionally scalar-specific `PrimitiveScalarTypeRegistry`; they do not duplicate control flow methods or select semantic policy. +Within policy, planning, support analysis, validation, and both backend +visitors, family-specific helpers stay in visibly labeled contiguous groups: +scalar helpers, string helpers, and ordinary-array helpers. Put a short section +comment above every such group so maintainers can find one datatype family +without scanning interleaved lowering methods. Generic orchestration remains +outside those groups and dispatches into them through the completed typed +actions. + ## Migration and Route Rules The legacy route remains the behavioral oracle until a lane has direct-plan @@ -337,6 +369,11 @@ An unsupported owner may select the legacy route before planning. Once the plan route is selected, planning, validation, lowering, printing, or compilation failure fails the build; it must not fall back to legacy generation. +Support reports and rollout gates keep scalar, string, and ordinary-array +input, optional, writeback, direct-result, and hidden-result lanes distinct. +Evidence for one datatype family must not make another family production +eligible accidentally. + For each lane: 1. replay an existing passing `tests/wrapper` case through the legacy route and @@ -491,9 +528,9 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 66 | -| `dual-route` | 0 | -| `legacy` | 134 | +| `wrapper-plan` | 78 | +| `dual-route` | 5 | +| `legacy` | 130 | | `not-applicable` | 95 | | `deferred-real-library` | 2 | @@ -513,6 +550,13 @@ blockers. | Phase 2D complete | 63 | 0 | 134 | 95 | 2 | 294 | | Phase 2E scalar isolation | 65 | 0 | 134 | 95 | 2 | 296 | | Phase 2F scalar result aggregation | 66 | 0 | 134 | 95 | 2 | 297 | +| Phase 5A required string values | 67 | 0 | 134 | 95 | 2 | 298 | +| Phase 5B fixed string results | 69 | 0 | 134 | 95 | 2 | 300 | +| Phase 5C fixed string writeback | 70 | 0 | 134 | 95 | 2 | 301 | +| Phase 5C assumed/optional string writeback | 71 | 0 | 134 | 95 | 2 | 302 | +| Phase 5D fixed string storage/raw addresses | 72 | 0 | 134 | 95 | 2 | 303 | +| Phase 5 production route reconciliation | 76 | 0 | 130 | 95 | 2 | 303 | +| Phase 6 ordinary arrays | 78 | 5 | 130 | 95 | 2 | 310 | Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating @@ -531,10 +575,19 @@ already covered by the new generator. | --- | --- | --- | --- | | `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | | `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/arrays/test_array_results.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | source/generated-.pyi parity | ordinary arrays; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_ordinary_array_results_match_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape ordinary array results; ranks one through fifteen; Fortran order; zero-sized results; allocation/copy/release failure paths | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing assumed-rank native unit | runtime ranks one through fifteen; mutable storage; rank validation; native-handle actuals deferred to Phase 7 | `dual-route` | | `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::*` | non-generating: legacy model/printer/policy unit coverage | legacy model/printer mechanics; ordinary arrays; native handles/descriptors | `not-applicable` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing multidimensional native unit | dense/explicit extents; positive-strided views; zero-sized axes; projected output identity; native-handle actuals deferred to Phase 7 | `dual-route` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | @@ -595,7 +648,8 @@ already covered by the new generator. | `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/snapshots; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native-call projections | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; native-call projections; arrays and derived types deferred to later lanes | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fnative_call_examples_f90` native unit | fixed mutable rank-zero NumPy bytes storage; raw fixed-string addresses; in-place mutation; rank/dtype/itemsize/writability/type validation | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `not-applicable` | | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | @@ -620,8 +674,10 @@ already covered by the new generator. | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | production plan route with deliberate legacy rollback comparison | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `dual-route` | | `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::*` | source/generated-.pyi parity or parametrized route | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | @@ -667,10 +723,17 @@ already covered by the new generator. | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | direct wrapper/build route | strings | `legacy` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity or parametrized route | strings | `legacy` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::*` | source/generated-.pyi parity or parametrized route | strings; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_required_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fmath_arrays_f90` native unit | required rank-one dense buffers; exact dtype/rank/order/alignment/writeability; zero length; native-handle actuals deferred to Phase 7 | `dual-route` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | edited semantic `.pyi` contract | strings; fixed/assumed inputs; arrays; mutable string storage | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | production plan route from source/generated-.pyi parity | fixed-form strings; fixed/assumed inputs; fixed results | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity | strings; fixed/assumed inputs; fixed/deferred results; arrays; writeback | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | fixed-width `NPY_STRING` array itemsize; rank/dtype/zero-size validation; native-handle actuals deferred to Phase 7 | `dual-route` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | required fixed/assumed scalar string inputs; default/kind-1/`c_char`; UTF-8 length and NUL validation; scalar results | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_string_results_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | direct fixed string results; trailing blanks; default/`c_char`; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy[*]` | production plan route from source/generated-.pyi parity | strings; fixed/assumed input/output; optional presence; Unicode/NUL handling | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed hidden string output; trailing blanks; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed immutable replacement and discarded identity; exact length; trailing blanks; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | assumed-length and optional immutable replacement; empty/omitted/`None`/concrete states; NUL rejection; concrete-only allocation failure | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_string_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | ## Incremental Protocol @@ -1182,50 +1245,532 @@ policy. ## Phase 5 — Strings -Scope: scalar character values, fixed-length strings, deferred-length strings, -and string copy/writeback. +Scope: non-descriptor scalar character values, fixed-length strings, assumed- +length call inputs, immutable replacement, mutable rank-zero byte storage, and +raw fixed-length character addresses. Character arrays remain in Phases 6 and +7; allocatable or pointer scalar character values remain in Phase 7; character +fields remain in Phases 8 and 9; character callbacks remain in Phase 10. + +The legacy wrapper is the behavioral oracle for this phase. In particular, +`CPythonBindingGenerator._convert_python_string_value_argument()` and +`_convert_python_string_storage_argument()` define Python conversion, +validation, allocation, and writeback behavior, while +`FortranToCBridgeGenerator._build_string_argument()`, +`_build_string_storage_argument()`, `_convert_raw_string_argument()`, and +`_convert_string_result()` define the bridge representation. The public +contract and observable oracle are +`docs/user/guide/fortran-wrapper.md`, `docs/user/guide/data-types.md`, +`docs/user/reference/semantic-pyi-format.md`, +`tests/wrapper/fortran/strings/test_character_arguments.py`, and +`tests/wrapper/fortran/strings/test_character_edge_cases.py`. Direct-plan +lowering may use different temporary names or an equivalent internal C ABI, +but it must preserve the legacy Python behavior, native argument order, +character payload, length, ownership, cleanup, and result projection. + +Strings use the same completed-policy and planning pipeline as the other +rank-zero scalar families: -- [ ] Before implementation, expand this phase under the mandatory expansion - gate, separating at least value/storage, fixed/deferred length, - input/result/inout, optionality, and ownership/lifetime cases found in the - live contract. - -- [ ] Define string handoff specs for value strings, storage strings, fixed - length, deferred length, and mutable buffers. -- [ ] Add binding and bridge actions for string value input, string storage - input, string result, and string writeback. -- [ ] Validate length/source expectations before emission. -- [ ] Keep string behavior separate from numeric scalar behavior even when both - are rank-zero. +```text +ArgumentPolicy -> ArgumentTransferPlan -> NativeCallSlotPlan +ResultPolicy -> ResultPlan +LifecyclePolicy -> LifecycleActionPlan +``` + +Do not add a parallel string plan hierarchy or plan-owned handler names. +Numeric and logical primitive families share registry-backed lowering because +their generated structure is the same. `DatatypeFamily.STRING` dispatches to +its own directly named binding and bridge lowering methods because character +conversion and ABI structure differ. The existing `STRING_VALUE`, +`STRING_STORAGE`, `PASS_CALL_LOCAL_ADDRESS`, `PASS_STORAGE_ADDRESS`, +`PASS_RAW_ADDRESS`, and generic codegen/lifecycle actions remain authoritative; +add a new typed action only if those completed actions cannot identify a real +semantic choice. + +Every string argument plan records the completed fixed positive character +length or the absence of a fixed length. The binding-to-bridge handoff records +both the payload address and encoded payload length when the bridge needs both; +this is an ABI fact in the existing argument transfer, not a new planning +stage. A fixed `String[n]` Python value must encode to exactly `n` bytes. A +plain `String` input carries its runtime UTF-8 byte length. Embedded NUL is +rejected before the native call. The bridge may copy bytes into Fortran +character storage only when `BridgeDataAction.COPY_REPRESENTATION` and its +non-empty policy reason were completed before planning. + +The phase is split into the following dependency-ordered sub-lanes. + +### Phase 5A — Required Read-Only String Values + +Included: required rank-zero `String[n]` and `String` Python `str` inputs; +default character, kind `1`, and `c_char`; fixed-length exact encoded-byte +validation; assumed-length runtime payload size; embedded-NUL rejection; and +primitive scalar or void results already supported by earlier phases. + +Excluded: writable inputs, projected replacement, optional strings, string +results, mutable `String[n][()]` storage, raw `Addr(String[n])`, arrays, +allocatable/deferred results, fields, and callbacks. + +Completed policy must provide `ObjectKind.STRING`, +`PythonBarrierAction.STRING_VALUE`, +`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, +`CodegenAction.CALL_LOCAL_INPUT`, `StorageMode.STACK`, required presence, +`BridgeDataAction.COPY_REPRESENTATION`, and the reason that C UTF-8 bytes are +materialized as Fortran character storage. Planning projects those facts into +the ordinary argument/native-slot records. The C binding method +`_lower_argument_required_string_value()` validates `str`, extracts UTF-8 plus +byte length, rejects embedded NUL, and enforces a fixed length when present. +The Fortran bridge method `_lower_argument_required_string_value()` receives +the payload address and length, associates a byte view, copies it into one +backend-local character temporary, and passes that temporary in the completed +native-call position. + +Validation requires a string-value Python action, call-local-address native +action, character-buffer handoff, one matching payload-length role, required +presence, no projected result, and a justified representation copy. Whole-unit +eligibility widens only for generation units containing this lane plus already +completed scalar/result/runtime lanes. Replay uses the existing +`fstrings_f90` native object and contract package with a reduced entry that +exports only existing read-only scalar string procedures; both routes run the +same fixed/assumed-length, kind, NumPy-string-scalar, wrong-length, and embedded +NUL assertions. The mixed original string nodes remain `legacy` because their +units also contain string results, writable strings, arrays, and allocatables. + +- [x] Complete Phase 5A policy, ordinary plan projection, validation, named C + and Fortran lowering, reduced-entry dual-route runtime parity, support + predicate, and migration-ledger evidence. + +### Phase 5B — Fixed-Length String Results And Hidden Outputs + +Included: direct fixed-length scalar character results and fixed-length hidden +`intent(out)` results, including trailing blanks. The binding receives a +NUL-terminated C-owned copy, converts the full payload to a Python-owned +`str`, and releases the temporary exactly once. The bridge allocates and fills +that copy only through completed `COPY_REPRESENTATION` policy. Deferred-length +and nullable allocatable or pointer results remain in Phase 7 because their +runtime length and allocation state are descriptor lifecycle facts, not +scalar-string conversion facts. + +Both forms reuse the ordinary ordered `ResultPolicy -> ResultPlan` path and +record the fixed positive `character_length` on the result. A direct native +function result has `source_kind="direct_return"`, +`CodegenAction.COPY_OUT`, no native-call slot, and a bridge function result of +`type(c_ptr)`. The bridge first receives the native value in backend-local +`character(kind=c_char, len=n)` storage, then allocates `n + 1` bytes through +the existing `x2py_malloc` interface, copies all `n` characters, appends +`c_null_char`, and returns the pointer. A hidden output has +`source_kind="hidden_output"`, `CodegenAction.COPY_OUT`, +`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, and references the exact same +fixed-length `NativeCallSlotPlan` used by the function. Its existing output +slot receives native character storage, then performs the same justified +allocation and copy after the native call. + +In both cases, binding lowering checks for a null allocation, converts the +NUL-terminated UTF-8 payload with the same observable behavior as the legacy +`Py_BuildValue("s", ...)` path, frees the C allocation exactly once even when +Python conversion fails, and returns the Python-owned `str`. Phase 5B supports +exactly one Python-visible string result per function; mixed or multiple +string result aggregation remains blocked until cleanup of every unconverted +native allocation is explicitly planned. A function that combines a public +fixed string result with native status-error projection is blocked for the +same reason: the status failure path must not bypass the string allocation's +planned release. + +Validation requires a fixed positive length, `ObjectKind.STRING`, Python-owned +copy-return ownership, no Python barrier action, the source-appropriate +codegen/native action, `BridgeDataAction.COPY_REPRESENTATION` with the standard +fixed-string copy reason, and matching result/native-slot lengths for hidden +outputs. Direct results must not carry a native-call slot; hidden results must +share their function slot by identity. The C and Fortran backends dispatch +`DatatypeFamily.STRING` to `_lower_result_fixed_string()` methods instead of +the primitive scalar registry. + +Replay direct results from the existing `fstrings_f90` native object with a +reduced contract entry exporting `char_result_default`, +`char_result_c_char`, `string_result_fixed`, `string_result_padded`, and +`string_result_c_char`. Replay the hidden output from the existing +`fcharacter_edges_f90.make_out` unit through another reduced entry. Run the +same trailing-blank and returned-value assertions through legacy and direct +routes. The original mixed nodes remain `legacy` on deferred results, writable +strings, optionality, or arrays. + +- [x] Complete fixed-length direct and hidden string result policy, result-plan + length facts, allocation/failure cleanup, binding conversion, bridge copy, + validation, legacy/direct parity, support widening, and ledger updates. + +### Phase 5C — Immutable String Output And Inout Replacement + +Included: fixed and assumed-length Python `str` output/inout dummies, including +the pass-by-address mutable native call. Python strings remain immutable: the +binding creates mutable call-local storage; the bridge passes that storage to +the native dummy; a declared `Returns["name", String...]` consumer returns a +replacement string; identity form discards native mutation and returns `None`. +Fixed buffers retain their complete post-call contents and trailing blanks; +assumed-length buffers use the encoded input length. Optional omitted, +explicit-`None`, and concrete-value states are handled here after required +replacement works. + +The first Phase 5C slice is required fixed-length `String[n]` only. A projected +replacement consumes completed `ObjectKind.STRING`, Python-owned +`COPY_RETURN`, `PYTHON_REFCOUNT`, stack contract storage, +`PythonBarrierAction.STRING_VALUE`, +`NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS`, +`CodegenAction.COPY_IN_OUT`, native mutation, result projection, and +`BridgeDataAction.COPY_REPRESENTATION` with the fixed-string replacement copy +reason. The ordinary argument plan records those facts plus the fixed positive +character length, and its binding and bridge views both carry the completed +codegen action. The same mutable native-call slot is referenced throughout; +there is no second output slot. + +The binding validates the input exactly as Phase 5A does, allocates one +`n + 1` byte call-local buffer through `x2py_malloc`, copies all `n` encoded +bytes, and appends NUL. Allocation failure raises `MemoryError` before native +execution. The bridge receives the mutable buffer and length, materializes +backend-local `character(kind=c_char, len=n)` storage, passes that storage to +the native dummy, then copies the complete post-call value back into the +binding buffer and restores the NUL terminator. After the call, binding +`_lower_writeback_string()` converts the replacement with the same +`Py_BuildValue("s", ...)` behavior as the legacy route and frees the call-local +buffer exactly once whether conversion succeeds or fails. + +The existing ordered lifecycle records remain authoritative: + +```text +COPY_IN (binding allocation and input copy) + -> NATIVE_MUTATION (bridge-local character call and copyback) + -> COPY_OUT (binding Python replacement conversion) + -> CLEANUP (binding call-local buffer release) +``` + +Validation requires the completed ownership/action facts, fixed result +position, one shared payload/length handoff, matching argument and native-slot +lengths/actions/copy reasons, exactly one complete lifecycle phase set, bridge +copyback ownership only for `COPY_IN_OUT`, and binding cleanup after conversion. +A replacement combined with native status-error projection stays blocked until +the status failure path also releases the mutable buffer. Multiple projected +results remain blocked by the existing single-writeback lane. + +A fixed identity contract uses the already-completed `CALL_LOCAL_INPUT` action, +the same call-local bridge character representation, no lifecycle result, and +returns `None`. Native writes affect only that temporary and are deliberately +discarded. Its binding buffer remains the borrowed read-only UTF-8 input because +the bridge never copies mutation back across the boundary. Assumed-length, +optional, mutable `String[n][()]`, and raw-address forms remain excluded from +this first slice. + +Replay both forms from the existing `fcharacter_edges_f90.fixed_inout` native +unit through a reduced edited contract that exports one projected replacement +and one identity spelling bound to the same native symbol. Run the same exact +length, trailing-blank, input-immutability, returned-value, and allocation +failure assertions through legacy and direct routes before widening the +whole-unit support predicate. + +The second Phase 5C slice keeps the same completed ownership, barrier, +representation-copy, and four-phase lifecycle records while removing the +compile-time-length restriction. For required assumed-length `String`, the +binding-recorded UTF-8 byte length is the native character length and the +replacement allocation size. A zero-byte input is valid: replacement owns a +one-byte NUL-only buffer, the bridge materializes a zero-length character +value, and binding returns the empty Python string after releasing the buffer. +Fixed strings still require the exact declared encoded length. + +Optional string values use the completed `OptionalMode.NULLABLE_VALUE`; they +do not invent a descriptor or reinterpret optionality as semantic nullability. +The binding ABI always carries the string payload pointer and runtime byte +length. Omitted and explicit `None` both send a null pointer with length zero, +so the bridge leaves the native optional dummy absent and a projected +replacement returns `None`. A concrete value is validated before native +execution, including embedded-NUL rejection and any fixed-length constraint. +Projected replacement allocates and owns the mutable `length + 1` buffer only +for that concrete value; identity form borrows the read-only payload and +discards native mutation exactly as the required identity path does. + +The bridge tests pointer association to choose the existing optional native +call branch. Only the present branch associates the payload, creates +`character(kind=c_char, len=runtime_length)` call-local storage, and invokes +the native optional dummy. Copyback is likewise guarded by pointer association, +so an absent optional never touches unassociated storage. Concrete projected +replacement restores the NUL terminator after copying all runtime-length +bytes. Binding then returns the concrete replacement and frees its allocation +exactly once, or returns `None` without calling `free` for the absent states. +Status-error combination and multiple projected replacements remain blocked +by the same explicit cleanup exclusions as the fixed required slice. + +Replay `assumed_inout` and `optional_inout` from the existing +`fcharacter_edges_f90` contract through a reduced entry module. Compare legacy +and direct routes for empty and non-empty assumed-length values, omitted, +explicit-`None`, and concrete optional states, input immutability, embedded-NUL +rejection before native execution, and allocator failure only for concrete +projected replacements. + +- [x] Complete fixed required replacement and discarded-identity policy, + writeback lifecycle, named lowering, cleanup, validation, parity, support + widening, and ledger updates. +- [x] Add assumed-length replacement and optional presence only after the fixed + required path is proven; preserve legacy empty-string and omitted/`None` + behavior and reject embedded NUL before native execution. + +### Phase 5D — Mutable Storage And Raw Fixed-Length Addresses + +Included: `String[n][()]` caller-owned rank-zero NumPy bytes storage and +`Addr(String[n])` caller-supplied integer addresses. The storage path validates +rank zero, dtype `S`, native byte order/alignment where applicable, and +writability before aliasing the caller buffer. The raw-address path does not +own or validate the pointee. Both use the declared fixed length; mutable +deferred-length scalar storage remains blocked. + +Both forms complete policy before planning and share no Phase 5C replacement +lifecycle. `String[n][()]` records `ObjectKind.STRING`, caller ownership, +`IN_PLACE`, caller destruction, alias contract/boundary storage, +`PythonBarrierAction.STRING_STORAGE`, +`NativeBarrierAction.PASS_STORAGE_ADDRESS`, `CodegenAction.IN_PLACE_ARGUMENT`, +native mutation, no result projection, and a fixed positive character length. +`Addr(String[n])` records the same caller ownership, in-place transfer, caller +destruction, mutation, and no result projection, but the contract value itself +uses stack storage while `PythonBarrierAction.RAW_ADDRESS` and +`NativeBarrierAction.PASS_RAW_ADDRESS` preserve the unsafe caller-supplied +address. The raw pointee is never adopted, released, sized, or validated by +x2py. This corrects the pre-5D raw-string decision that incorrectly retained +immutable-string call-local ownership despite the completed raw-address +barriers. + +The ordinary argument plan carries the fixed length and uses +`ArgumentHandoffMode.OPAQUE_ADDRESS` for both forms. There is one pointer ABI +field and no runtime length field: the fixed character length comes only from +the completed plan. The binding storage handler accepts exactly a rank-zero +NumPy `NPY_STRING` array whose itemsize is `n`, requires alignment and +writability, and forwards `PyArray_DATA` without allocating or copying. +The raw handler accepts an integer and uses the existing `PyLong_AsVoidPtr` +path; it deliberately does not inspect the pointee, its allocation extent, or +its lifetime. + +The native character scalar is not directly C interoperable, so both forms +record `BridgeDataAction.COPY_REPRESENTATION` with a boundary-specific reason. +The bridge associates the incoming address with exactly `n` +`character(kind=c_char)` bytes, copies them into backend-local +`character(kind=c_char, len=n)` storage, invokes the native dummy, and copies +all `n` post-call bytes back. It does not append NUL, allocate, free, infer +ownership, or create a Python result. These helper locals are emitted-code +details selected by the completed storage/raw policy. + +Optional storage/address arguments and projected returns remain blocked in +this phase. `String[()]` and `Addr(String)` are rejected by the semantic `.pyi` +contract because the bridge has no fixed extent; arrays and callback storage +remain owned by their later lanes. Validation rejects edited plans with a +missing/nonpositive length, the wrong owner/transfer/destruction/storage mode, +an inconsistent barrier or handoff, a runtime length role, an unjustified copy +reason, missing mutation, or result projection before either backend lowers. + +Replay `fixed_inout_storage` and `fixed_inout_raw` from the existing +`fnative_call_examples_f90` edited contract through a reduced entry bound to +the same `fixed_inout` native routine. Compare legacy and direct routes for +complete eight-byte mutation, rank/dtype/itemsize/writability failures, raw +integer type rejection, and lack of Python return. Keep the existing mixed +native-order test on the legacy route because its array and derived-type +neighbors belong to later phases; add the reduced replay as a separate +wrapper-plan ledger node. + +- [x] Complete mutable string-storage and raw-address policy, address handoff, + bridge association/copyback mechanics, validation, legacy/direct parity, + support widening, and ledger updates. + +### Phase 5 Completion + +Descriptor-backed scalar character values are deliberately outside this +phase. A contract such as `String | None` with +`result=Allocatable(Return(...))` carries allocation state, runtime element +length, descriptor ownership, and native release responsibility. It must enter +the direct route only through Phase 7's shared allocatable/pointer descriptor +plan; it remains a rank-zero Python `str | None` result rather than a native +array handle. Phase 5 must not add a character-only descriptor ABI or cleanup +path. + +- [x] Expand the phase under the mandatory expansion gate from the live + policies, legacy binding/bridge implementation, public string contract, and + focused wrapper tests. +- [x] Validate fixed/runtime length sources, payload/length role agreement, + result ownership, writeback consumers, and cleanup responsibility before + either backend emits source. +- [x] Keep string behavior in directly named string lowering methods while + reusing the ordinary scalar planning records and lifecycle flow. +- [x] Finish Phase 5 only when all non-descriptor scalar string sub-lanes are + proven and every affected matrix row is either `wrapper-plan` or blocked by + a later array, descriptor, field, or callback lane recorded in the ledger. ## Phase 6 — Ordinary Arrays Scope: NumPy data-buffer arrays that do not require native descriptor handles. -- [ ] Before implementation, expand this phase under the mandatory expansion - gate, separating at least input/result/inout, hidden outputs, rank/shape - forms, dtype families, C/Fortran order and striding, optionality, copy versus - view behavior, and writeability cases found in the live contract. - -- [ ] Define array handoff specs for data pointer, dtype, rank, shape, strides, - contiguity/order, itemsize, and writeability. -- [ ] Add binding actions for NumPy validation and data-buffer extraction. -- [ ] Add bridge actions for array data-buffer passing and shape/stride - forwarding. -- [ ] Represent hidden array outputs and array copy-out/writeback explicitly. -- [ ] Validate dtype/rank/shape/order expectations in the plan before emitted C - checks are generated. +The ordinary-array lane borrows or copies NumPy data buffers; it never creates +or consumes a persistent native descriptor handle. `Allocatable[T[...]]`, +`Pointer[T[...]]`, rank-zero allocatable/pointer scalars, and a native handle +used as the actual value for an ordinary array dummy all remain in Phase 7. +Derived-type arrays remain in Phase 8, fields in Phases 8 and 9, and callback +arrays in Phase 10. The full BLAS/LAPACK generation unit remains deferred until +final cutover even when individual ordinary-array shapes become supported. + +Whole-generation-unit rollout preserves that Phase 7 boundary. Output-only +ordinary array results and hidden outputs may select the production plan route +now. A generation unit with an ordinary array actual remains on the legacy +route, even after its NumPy-buffer path has direct-route parity, because route +selection cannot know whether a caller will pass a NumPy array or a supported +native descriptor handle. Those reduced array-actual rows remain `dual-route` +with the native-handle caller contract recorded as their sole Phase 7 blocker; +the direct route is forced only by the internal parity harness. + +The public behavior is defined by the NumPy array contract in +`docs/user/guide/fortran-wrapper.md`, the array spelling and metadata rules in +`docs/user/reference/semantic-pyi-format.md`, and the existing array wrapper +tests. The legacy binding validates exact dtype, rank, every expressible +extent, native byte order, alignment, layout/stride requirements, and +writeability for mutable storage before the native call. It does not cast, +byte-swap, repair alignment, de-alias overlapping storage, or silently copy a +rejected layout. Read-only source `intent(in)` storage may remain read-only; +edited `.pyi` array storage is writable unless a completed policy says +otherwise. Zero-sized dimensions are valid when the rest of the contract is +valid. + +Every ordinary array remains in the existing +`ArgumentPolicy -> ArgumentTransferPlan -> NativeCallSlotPlan` or +`ResultPolicy -> ResultPlan` flow. An argument embeds one editable array +handoff spec containing element family, concrete or runtime rank, declared +shape expressions, axis modes, order, contiguity, itemsize when relevant, +writeability, and the exact ABI roles for data, extents, upper bounds, strides, +runtime rank, or itemsize. Policy completion selects +`PythonBarrierAction.ARRAY_STORAGE`, +`NativeBarrierAction.PASS_ARRAY_BUFFER`, and either +`BridgeDataAction.ASSOCIATE_VIEW` for caller storage or an explicit copy action +and reason. Planning must not reconstruct any of those choices from rank, +shape spelling, or datatype. The binding and bridge dispatch only to directly +named array implementation methods selected by those completed facts. + +The binding checks the NumPy object before extracting `PyArray_DATA`, shape, +and element strides. The bridge receives only the fields named by the handoff +spec, associates the pointer with the completed element type and extents, and +constructs a stride slice only when the plan explicitly allows it. C-oriented +flat storage reverses bridge association extents only when the completed order +requires it. Backend-local pointer views and slice expressions are emitted-code +details; dtype, rank, extent, order, stride acceptance, mutation, projection, +copy, and ownership are semantic policy. + +The phase is dependency-ordered as follows. + +### Phase 6A — Required Rank-One Contiguous Buffers + +Included: required concrete-rank-one ordinary arrays with dense contiguous +axes (`T[:]`) for the existing bool, integer, real, and complex primitive +families; caller-owned borrowed/in-place storage; scalar or void neighbors and +results already supported by earlier phases; native position reordering; and +zero-length buffers. The binding requires an exact NumPy dtype, rank one, +native byte order, alignment, contiguity, and writeability only when completed +ownership says native code mutates the storage. It forwards the data address +and runtime extent. The bridge creates one typed rank-one pointer view with +`c_f_pointer` and passes that view in the completed native-call position. + +This slice records `ArgumentHandoffMode.ARRAY_BUFFER` and +`BridgeDataAction.ASSOCIATE_VIEW`; it performs no allocation, element copy, +writeback action, release, or Python result projection. Explicit/fixed extent +expressions, `Flat`, multidimensional order, strided axes, optionality, +projected output identity, array results, character arrays, assumed rank, and +native-handle actuals remain in later sub-lanes. Replay one existing +`fmath_arrays_f90` contiguous routine through a reduced semantic `.pyi` entry +and compare legacy/direct behavior for mutation, dtype, rank, alignment, +byte-order, contiguity, writeability, zero length, and native argument order. + +- [x] Complete required rank-one contiguous array policy, editable handoff + spec, validation, named C/Fortran lowering, reduced legacy/direct parity, + support widening, and ledger evidence. + +### Phase 6B — Declared Extents, Flat Storage, And Dense Rank + +Included: fixed and visible-symbol extent expressions, lower-bound-derived +extents, assumed-size `Flat`, ranks two through fifteen, `ORDER_F` and +`ORDER_C`, dense contiguous layout, and zero-sized axes. Shape expressions are +resolved against existing scalar handoff roles before backend emission; the +bridge association order follows the completed layout. Any expression that +cannot be represented by available roles remains blocked rather than being +recomputed in a backend. + +- [x] Complete declared-shape evaluation, flat-storage orientation, + multidimensional dense handoff, validation, parity, and ledger evidence. + +### Phase 6C — Positive-Strided Ordinary Views + +Included: `::` axes and bounded stride-aware axes, runtime upper bounds and +element strides, Fortran-oriented positive-stride slicing, contiguous views as +a valid special case, and degenerate zero-size strides. Negative, zero on an +addressable axis, incompatible C-oriented, broadcast, and otherwise invalid +layouts fail before the native call. No copy-to-contiguous fallback is inferred. + +- [x] Complete stride roles, upper bounds, positive-stride bridge slices, + layout validation, parity, and ledger evidence. + +### Phase 6D — Output Storage And Projected Identity + +Included: ordinary `intent(out)`/`intent(inout)` caller buffers and +`Returns["name", T[...]]` projections. Native code mutates the same validated +NumPy storage; the binding returns the original Python array object with one +owned reference rather than constructing a second array or copying elements. +Read-only output storage fails before the call. Multiple projections compose +with the existing ordered result aggregation only after every projected array +identity and failure-path reference is planned. + +- [x] Complete in-place output ownership, projected identity/reference + lifecycle, multiple-result aggregation, parity, and ledger evidence. + +### Phase 6E — Ordinary Array Results And Hidden Outputs + +Included: non-allocatable direct array results and hidden output arrays whose +shape and element ownership are fully expressible without persistent native +descriptors. The plan records the producer, every runtime extent, allocation +owner, copy or transfer action, Python NumPy construction, and release on +success and every failure path. Nullable allocatable/pointer results remain in +Phase 7. + +- [x] Complete ordinary result/hidden-output allocation, shape projection, + copy ownership, cleanup, parity, and ledger evidence. + +### Phase 6F — Optional, Assumed-Rank, And Character Buffers + +Included: ordinary optional NumPy arrays, numeric assumed-rank dispatch from +one through fifteen, and fixed-width NumPy bytes character arrays with planned +itemsize. Omitted ordinary optional arrays remain distinct from present +storage. Assumed-rank plans carry a runtime-rank role and validate the supported +range before bridge dispatch. Character arrays use exact `NPY_STRING` itemsize +and remain raw fixed-width bytes; deferred descriptor-backed character values +remain in Phase 7. Fixed-shape character array direct results and hidden +outputs reuse the Phase 6E copy-result path with their itemsize included in +NumPy dtype construction and bridge byte-count calculation. + +- [x] Complete optional presence, assumed-rank dispatch, character itemsize, + validation, parity, and ledger evidence. + +### Phase 6 Completion + +- [x] Expand the phase under the mandatory expansion gate from live semantic + array contracts, legacy binding/bridge lowering, public docs, and focused + wrapper tests. +- [x] Define array handoff specs for every supported data, rank, shape, stride, + order, itemsize, writeability, result, and lifecycle role. +- [x] Validate every completed array policy and handoff role before either + backend emits source. +- [x] Finish Phase 6 only when every ordinary-array matrix row is migrated or + remains blocked solely by an explicitly later descriptor, derived, field, + callback, or deferred-real-library lane. ## Phase 7 — Native Array Handles And Descriptors -Scope: `Allocatable[T[...]]`, `Pointer[T[...]]`, descriptor-backed handoffs, and -runtime native array handle objects. +Scope: `Allocatable[T[...]]`, `Pointer[T[...]]`, scalar descriptor-backed +values including allocatable or pointer `String`, descriptor-backed handoffs, +and runtime native array handle objects. - [ ] Before implementation, expand this phase under the mandatory expansion gate and reconcile it with the maintained native-array-handle checklist. Split allocatable and pointer behavior only after extracting their shared - descriptor, presence, ownership, release, module/field, argument, and result - sub-lanes. + descriptor, presence, runtime element-length, ownership, release, + module/field, argument, and result sub-lanes. Include nullable deferred- + length scalar character results in that shared audit rather than creating a + string-only descriptor path. Keep rank-zero scalar descriptor results as + copied Python scalar values; reserve native-array handles for rank-positive + array storage. - [ ] Define descriptor handoff specs for CFI descriptors, descriptor ownership, optional-absent handles, owner retention, extraction policy, and required diff --git a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py b/tests/codegen/bindings/test_binding_handle_policy_dispatch.py index 33d6878a1..0884856cf 100644 --- a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py +++ b/tests/codegen/bindings/test_binding_handle_policy_dispatch.py @@ -89,7 +89,6 @@ def test_bridge_and_binding_generators_expose_ownership_action_maps(): (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_build_borrowed_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_build_copy_return_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_build_copy_return_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_build_copy_return_array_result", } assert ( FortranToCBridgeGenerator._ALLOCATABLE_RESULT_HELPER_DISPATCHER.handlers[ @@ -173,7 +172,7 @@ def test_bridge_and_binding_generators_expose_ownership_action_maps(): NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, NativeBarrierAction.PASS_STORAGE_ADDRESS, NativeBarrierAction.PASS_RAW_ADDRESS, - NativeBarrierAction.PASS_ARRAY_DESCRIPTOR, + NativeBarrierAction.PASS_ARRAY_BUFFER, NativeBarrierAction.PASS_WRAPPER_ADDRESS, } assert set(CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER.handlers) == { diff --git a/tests/pipeline/test_wrapper_plan_route_selection.py b/tests/pipeline/test_wrapper_plan_route_selection.py index a52451bf9..a5157f1b9 100644 --- a/tests/pipeline/test_wrapper_plan_route_selection.py +++ b/tests/pipeline/test_wrapper_plan_route_selection.py @@ -74,6 +74,32 @@ def scale(x: Float64) -> Float64: ... "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" "test_multiple_scalar_results_match_both_routes_without_array_blockers", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_fixed_string_results_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" + "test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/scalars/test_verified_baseline.py::" + "test_required_array_buffers_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::" + "test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_array_results.py::" + "test_ordinary_array_results_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::" + "test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" + "test_optional_array_buffers_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/function_calls/test_output_arguments.py::" + "test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", ) assert decision.selection_reason == "wrapper-plan route forced for internal migration verification" @@ -331,7 +357,7 @@ def solve(value: Int32) -> tuple[Int32, String[32]]: ... ) -def test_route_selector_keeps_a_module_with_any_unsupported_member_entirely_legacy(): +def test_route_selector_keeps_array_buffer_lane_legacy_until_native_handle_actuals_are_supported(): module = _completed_module( """ def scale(x: Float64) -> Float64: ... @@ -348,8 +374,14 @@ def sum_values(values: Float64[:]) -> Float64: ... assert decision.selected_route == "legacy" assert decision.rollout_eligible is False - assert {blocker.owner_path for blocker in decision.blockers} == {"fmath.sum_values"} - assert decision.selection_reason == "generation unit has unsupported wrapper-plan owners" + assert decision.covered_lanes == ( + "scalar-inputs", + "scalar-direct-results", + "native-call-runtime", + "array-buffer-inputs", + ) + assert decision.blockers == () + assert decision.selection_reason == "covered lanes exceed the recorded wrapper-plan parity evidence" def test_route_selector_keeps_unimplemented_scalar_kinds_on_legacy_route(): @@ -375,14 +407,16 @@ def test_route_selector_never_silently_falls_back_when_plan_route_is_forced(): module = _completed_module( """ def scale(x: Float64) -> Float64: ... -def sum_values(values: Float64[:]) -> Float64: ... + +class sample: + value: Int32 """, module_name="fmath", ) with pytest.raises( ValueError, - match=r"cannot force wrapper-plan route.*fmath\.sum_values", + match=r"cannot force wrapper-plan route.*fmath\.sample", ): build_pipeline._select_wrapper_plan_route( module, @@ -392,6 +426,97 @@ def sum_values(values: Float64[:]) -> Float64: ... ) +def test_route_selector_keeps_existing_bind_c_direct_symbol_calls_on_legacy(): + module = _completed_module("def add_one(value: Int32) -> Int32: ...", module_name="bind_c_value") + module.functions[0].metadata["fortran_bind_c"] = True + module.functions[0].metadata["fortran_bind_c_name"] = "solver_add_one" + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "legacy" + assert [blocker.reason for blocker in decision.blockers] == [ + "existing bind(C) direct-symbol calls are not implemented by wrapper-plan lowering" + ] + + +@pytest.mark.parametrize( + ("source", "module_name", "covered_lanes"), + ( + ( + "def label(value: String[3]) -> String[3]: ...", + "fixed_strings", + ("string-value-inputs", "fixed-string-direct-results", "native-call-runtime"), + ), + ( + "def vector() -> Float64[3]: ...", + "array_result", + ("array-direct-results", "native-call-runtime"), + ), + ( + "@native_call([Return('values', 0)])\ndef hidden() -> Float64[3]: ...", + "array_hidden_output", + ("array-hidden-outputs", "native-call-runtime"), + ), + ), +) +def test_route_selector_selects_completed_string_and_array_lanes_for_production( + source: str, + module_name: str, + covered_lanes: tuple[str, ...], +): + module = _completed_module(source, module_name=module_name) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "wrapper-plan" + assert decision.rollout_eligible is True + assert decision.covered_lanes == covered_lanes + assert decision.blockers == () + + +@pytest.mark.parametrize( + ("source", "module_name", "covered_lanes"), + ( + ( + 'def fill(values: Float64[:]) -> Returns["values", Float64[:]]: ...', + "array_writeback", + ("array-buffer-inputs", "array-writebacks", "native-call-runtime"), + ), + ( + "def maybe(values: Float64[:] = ...) -> None: ...", + "optional_array", + ("array-buffer-inputs", "array-optional-inputs", "void-calls", "native-call-runtime"), + ), + ), +) +def test_route_selector_keeps_array_actual_lanes_legacy_until_phase7( + source: str, + module_name: str, + covered_lanes: tuple[str, ...], +): + module = _completed_module(source, module_name=module_name) + + decision = build_pipeline._select_wrapper_plan_route( + module, + makefile=False, + strict_wrapper_names=False, + ) + + assert decision.selected_route == "legacy" + assert decision.rollout_eligible is False + assert decision.covered_lanes == covered_lanes + assert decision.blockers == () + assert decision.selection_reason == "covered lanes exceed the recorded wrapper-plan parity evidence" + + def test_route_selector_keeps_an_explicitly_forced_legacy_module_entirely_legacy(): module = _completed_module( """ diff --git a/tests/semantics/policy/test_native_array_ownership.py b/tests/semantics/policy/test_native_array_ownership.py index 1d4b78173..4cb50b127 100644 --- a/tests/semantics/policy/test_native_array_ownership.py +++ b/tests/semantics/policy/test_native_array_ownership.py @@ -128,8 +128,8 @@ def make_values() -> Allocatable[Float64[:]]: ... assert decision.owner is OwnershipOwner.WRAPPER assert decision.transfer is TransferMode.WRAPPER_INSTANCE assert decision.destruction is DestructionPolicy.WRAPPER_DEALLOC - assert decision.codegen_action is CodegenAction.HIDDEN_OUTPUT - assert decision.native_barrier_action is NativeBarrierAction.PASS_ARRAY_DESCRIPTOR + assert decision.codegen_action is CodegenAction.WRAPPER_INSTANCE + assert decision.native_barrier_action is NativeBarrierAction.PASS_NATIVE_DESCRIPTOR assert policy.handle_kind == "owned_result_descriptor" assert policy.origin == "projected_result" assert policy.owner_retention == "wrapper_owner_storage" diff --git a/tests/semantics/policy/test_policy_defaults_and_validation.py b/tests/semantics/policy/test_policy_defaults_and_validation.py index dc98eae20..ad2d9ae8c 100644 --- a/tests/semantics/policy/test_policy_defaults_and_validation.py +++ b/tests/semantics/policy/test_policy_defaults_and_validation.py @@ -152,7 +152,7 @@ def test_default_policy_decisions_cover_public_object_kinds(): _hidden_output_context(projects_result=True, python_visible=False), ) assert hidden_derived_output.transfer is TransferMode.WRAPPER_INSTANCE - assert hidden_derived_output.codegen_action is CodegenAction.HIDDEN_OUTPUT + assert hidden_derived_output.codegen_action is CodegenAction.WRAPPER_INSTANCE derived_field = resolver.decide_semantic_type(_derived_type(), OwnershipContext.field()) assert derived_field.owner is OwnershipOwner.WRAPPER @@ -197,7 +197,7 @@ def test_default_policy_completes_python_and_native_barrier_actions(): _array_type(), _read_only_argument_context(), PythonBarrierAction.ARRAY_STORAGE, - NativeBarrierAction.PASS_ARRAY_DESCRIPTOR, + NativeBarrierAction.PASS_ARRAY_BUFFER, ), ( "string_value", @@ -433,7 +433,7 @@ def test_immutable_derived_output_selects_wrapper_instance_and_replacement_block assert output.owner is OwnershipOwner.WRAPPER assert output.transfer is TransferMode.WRAPPER_INSTANCE assert output.destruction is DestructionPolicy.WRAPPER_DEALLOC - assert output.codegen_action is CodegenAction.HIDDEN_OUTPUT + assert output.codegen_action is CodegenAction.WRAPPER_INSTANCE replacement = default_ownership_policy.decide_semantic_type( semantic_type, diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index d6fba726c..320a28495 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -19,14 +19,18 @@ from x2py.semantics.ownership import ( AssignmentMode, CodegenAction, + DestructionPolicy, NativeBarrierAction, ObjectKind, + OwnershipOwner, PythonBarrierAction, StorageMode, SetterAction, + TransferMode, ) from x2py.semantics.policy_completion import complete_semantic_policies from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, BridgeDataAction, FunctionWrapperPolicy, ModuleGetterAction, @@ -34,6 +38,8 @@ NativeStatusErrorPolicy, OptionalMode, PythonExceptionKind, + RAW_STRING_ADDRESS_COPY_REASON, + STRING_STORAGE_COPY_REASON, WritebackPhase, completed_function_wrapper_policy, ) @@ -518,7 +524,7 @@ def tagged(x: Float64) -> Float64: ... assert "native-call literal slot 1 uses unsupported first-lane literal type 'String[1]'" in policy.blockers -def test_wrapper_policy_blocks_non_primitive_arguments_before_planning(): +def test_wrapper_policy_completes_required_rank_one_array_buffer_handoff(): module = parse_pyi_text( """ def sum_values(values: Float64[:]) -> Float64: ... @@ -530,16 +536,23 @@ def sum_values(values: Float64[:]) -> Float64: ... policy = function.metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] assert isinstance(policy, FunctionWrapperPolicy) - assert policy.supported is False - assert "argument 'values' is not a first-lane primitive scalar" in policy.blockers - assert "argument 'values' has no completed bridge data action" in policy.blockers - assert policy.arguments[0].bridge_data_action is BridgeDataAction.BLOCKED - - with pytest.raises(ValueError, match="blocked wrapper policy"): - completed_function_wrapper_policy(function) - - -def test_wrapper_policy_keeps_string_arguments_blocked_until_bridge_data_action_is_completed(): + assert policy.supported is True + assert policy.blockers == () + argument = policy.arguments[0] + assert argument.ownership.kind is ObjectKind.NUMPY_ARRAY + assert argument.python_barrier_action is PythonBarrierAction.ARRAY_STORAGE + assert argument.native_barrier_action is NativeBarrierAction.PASS_ARRAY_BUFFER + assert argument.bridge_data_action is BridgeDataAction.ASSOCIATE_VIEW + assert argument.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER + assert argument.array is not None + assert argument.array.rank == 1 + assert argument.array.shape == (":",) + assert argument.array.axes == ("dense",) + assert argument.array.contiguous is True + assert policy.native_call_slots[0].array == argument.array + + +def test_wrapper_policy_completes_required_read_only_string_value_handoff(): module = parse_pyi_text( "def consume(value: String) -> None: ...", module_name="string_argument", @@ -547,9 +560,182 @@ def test_wrapper_policy_keeps_string_arguments_blocked_until_bridge_data_action_ complete_semantic_policies(module) policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] - assert policy.supported is False - assert "argument 'value' has no completed bridge data action" in policy.blockers - assert policy.arguments[0].bridge_data_action is BridgeDataAction.BLOCKED + assert policy.supported is True + assert policy.blockers == () + argument = policy.arguments[0] + assert argument.python_barrier_action is PythonBarrierAction.STRING_VALUE + assert argument.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert argument.codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert argument.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER + assert argument.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert argument.bridge_copy_reason == ("materialize Fortran character storage from the binding UTF-8 byte buffer") + + +def test_wrapper_policy_completes_fixed_string_direct_and_hidden_copy_results(): + module = parse_pyi_text( + """ +def direct_label() -> String[8]: ... + +@native_call([Return("label", 0)]) +def hidden_label() -> String[8]: ... +""", + module_name="fixed_string_results", + ) + complete_semantic_policies(module) + direct_policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + hidden_policy = module.functions[1].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert direct_policy.supported is True + direct = direct_policy.results[0] + assert direct.ownership.kind is ObjectKind.STRING + assert direct.codegen_action is CodegenAction.COPY_OUT + assert direct.native_barrier_action is NativeBarrierAction.NONE + assert direct.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert direct.character_length == 8 + + assert hidden_policy.supported is True + hidden = hidden_policy.results[0] + assert hidden.ownership.kind is ObjectKind.STRING + assert hidden.codegen_action is CodegenAction.COPY_OUT + assert hidden.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert hidden.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert hidden.character_length == 8 + assert hidden_policy.native_call_slots[0].character_length == hidden.character_length + + +def test_wrapper_policy_completes_fixed_string_replacement_and_discarded_identity(): + module = parse_pyi_text( + """ +def replace_name(name: String[8]) -> Returns["name", String[8]]: ... +def discard_name(name: String[8]) -> None: ... +""", + module_name="fixed_string_writeback", + ) + complete_semantic_policies(module) + replacement = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + identity = module.functions[1].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert replacement.supported is True + argument = replacement.arguments[0] + assert argument.ownership.kind is ObjectKind.STRING + assert argument.ownership.owner is OwnershipOwner.PYTHON + assert argument.ownership.transfer is TransferMode.COPY_RETURN + assert argument.ownership.destruction is DestructionPolicy.PYTHON_REFCOUNT + assert argument.codegen_action is CodegenAction.COPY_IN_OUT + assert argument.character_length == 8 + assert argument.projects_result is True + assert argument.writable is True + assert tuple(action.phase for action in replacement.writeback_actions) == tuple(WritebackPhase) + + assert identity.supported is True + assert identity.arguments[0].codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert identity.arguments[0].projects_result is False + assert identity.writeback_actions == () + + +def test_wrapper_policy_completes_assumed_optional_replacements_and_blocks_unreleased_status_cleanup(): + module = parse_pyi_text( + """ +def assumed(name: String) -> Returns["name", String]: ... +def optional(label: String = ...) -> Returns["label", String] | None: ... +def optional_fixed(label: String[8] = ...) -> Returns["label", String[8]] | None: ... +def optional_identity(label: String = ...) -> None: ... + +@raises(status="status", success=0) +@native_call([Arg(0), Return("status", 1)]) +def with_status( + name: String[8] +) -> tuple[Returns["name", String[8]], Returns["status", Int32]]: ... +""", + module_name="blocked_string_writeback", + ) + complete_semantic_policies(module) + assumed = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + optional = module.functions[1].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + optional_fixed = module.functions[2].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + optional_identity = module.functions[3].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + with_status = module.functions[4].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert assumed.supported is True + assert assumed.arguments[0].character_length is None + assert assumed.arguments[0].codegen_action is CodegenAction.COPY_IN_OUT + assert optional.supported is True + assert optional.arguments[0].optional_mode is OptionalMode.NULLABLE_VALUE + assert optional.arguments[0].nullable is False + assert optional.arguments[0].character_length is None + assert optional_fixed.supported is True + assert optional_fixed.arguments[0].character_length == 8 + assert optional_identity.supported is True + assert optional_identity.arguments[0].optional_mode is OptionalMode.NULLABLE_VALUE + assert optional_identity.arguments[0].codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert optional_identity.writeback_actions == () + assert with_status.supported is False + assert "string replacement with native status error requires planned failure-path cleanup" in (with_status.blockers) + + +def test_wrapper_policy_completes_fixed_string_storage_and_raw_address_ownership(): + module = parse_pyi_text( + """ +def storage(label: String[8][()]) -> None: ... +def raw(label: Addr(String[8])) -> None: ... +""", + module_name="fixed_string_addresses", + ) + complete_semantic_policies(module) + storage = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + raw = module.functions[1].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert storage.supported is True + storage_argument = storage.arguments[0] + assert storage_argument.ownership.kind is ObjectKind.STRING + assert storage_argument.ownership.owner is OwnershipOwner.CALLER + assert storage_argument.ownership.transfer is TransferMode.IN_PLACE + assert storage_argument.ownership.destruction is DestructionPolicy.CALLER + assert storage_argument.storage_mode is StorageMode.ALIAS + assert storage_argument.boundary_storage_mode is StorageMode.ALIAS + assert storage_argument.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert storage_argument.python_barrier_action is PythonBarrierAction.STRING_STORAGE + assert storage_argument.native_barrier_action is NativeBarrierAction.PASS_STORAGE_ADDRESS + assert storage_argument.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert storage_argument.bridge_copy_reason == STRING_STORAGE_COPY_REASON + assert storage_argument.character_length == 8 + assert storage.writeback_actions == () + + assert raw.supported is True + raw_argument = raw.arguments[0] + assert raw_argument.ownership.kind is ObjectKind.STRING + assert raw_argument.ownership.owner is OwnershipOwner.CALLER + assert raw_argument.ownership.transfer is TransferMode.IN_PLACE + assert raw_argument.ownership.destruction is DestructionPolicy.CALLER + assert raw_argument.storage_mode is StorageMode.STACK + assert raw_argument.boundary_storage_mode is StorageMode.STACK + assert raw_argument.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert raw_argument.python_barrier_action is PythonBarrierAction.RAW_ADDRESS + assert raw_argument.native_barrier_action is NativeBarrierAction.PASS_RAW_ADDRESS + assert raw_argument.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert raw_argument.bridge_copy_reason == RAW_STRING_ADDRESS_COPY_REASON + assert raw_argument.character_length == 8 + assert raw.writeback_actions == () + + +def test_wrapper_policy_blocks_optional_or_projected_string_address_forms(): + module = parse_pyi_text( + """ +def optional_storage(label: String[8][()] = ...) -> None: ... +def optional_raw(label: Addr(String[8]) = ...) -> None: ... +def projected_storage(label: String[8][()]) -> Returns["label", String[8][()]]: ... +def projected_raw(label: Addr(String[8])) -> Returns["label", String[8]]: ... +""", + module_name="blocked_string_addresses", + ) + complete_semantic_policies(module) + policies = [function.metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] for function in module.functions] + + assert all(policy.supported is False for policy in policies) + assert "optional string storage is unsupported" in "; ".join(policies[0].blockers) + assert "optional raw string address is unsupported" in "; ".join(policies[1].blockers) + assert "string storage unexpectedly projects a result" in "; ".join(policies[2].blockers) + assert "raw string address unexpectedly projects a result" in "; ".join(policies[3].blockers) def test_missing_wrapper_policy_fails_before_planning(): diff --git a/tests/wrapper/fortran/arrays/test_array_results.py b/tests/wrapper/fortran/arrays/test_array_results.py index 64dc55e1b..5e19c65f7 100644 --- a/tests/wrapper/fortran/arrays/test_array_results.py +++ b/tests/wrapper/fortran/arrays/test_array_results.py @@ -2,12 +2,18 @@ import gc from pathlib import Path +import shutil import numpy as np +import pytest +from x2py import build_pyi_extension from x2py.runtime.handles import AllocatableArray from tests.wrapper.fortran._support import ( _build_source_or_generated_pyi_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, wrapper_source, ) @@ -93,3 +99,50 @@ def test_array_results_follow_data_buffer_and_descriptor_handle_contracts( np.testing.assert_allclose(cube, expected_cube) for result, expected in rank_results: np.testing.assert_allclose(result, expected) + + +def test_ordinary_array_results_match_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): + """Replay fixed-shape direct results without descriptor-backed neighbors.""" + native_object = _compile_native_object(ARRAY_RESULTS_F90_SOURCE, tmp_path / "native") + modules = {} + selected = ( + "fixed_vector", + "automatic_vector", + "automatic_matrix", + "rank3_cube", + "rank15_result", + "zero_vector", + ) + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_ordinary_array_results" + shutil.copytree(CONTRACT_FIXTURES / "farray_results_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "".join(f"from .farray_results_f90 import {name}\n" for name in selected), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "fixed_vector") else _sole_native_module(module) + + for module in modules.values(): + np.testing.assert_array_equal(module.fixed_vector(), np.array([1.0, 2.0, 3.0])) + np.testing.assert_array_equal(module.automatic_vector(np.int32(3)), np.array([2.0, 4.0, 6.0])) + matrix = module.automatic_matrix(np.int32(2), np.int32(3)) + np.testing.assert_array_equal(matrix, np.array([[12.0, 13.0, 14.0], [22.0, 23.0, 24.0]])) + assert matrix.flags.f_contiguous + assert module.rank3_cube(np.int32(0), np.int32(2), np.int32(3)).shape == (0, 2, 3) + assert module.rank15_result().shape == (2, *([1] * 14)) + assert module.zero_vector().shape == (0,) + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + with pytest.raises(MemoryError, match="Unable to allocate copy-return output array"): + modules["wrapper_plan"].fixed_vector() diff --git a/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py b/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py index cdf6930a1..6309a8a06 100644 --- a/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py +++ b/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py @@ -1,11 +1,19 @@ """Assumed-rank array dispatch and supported-rank boundary tests.""" from pathlib import Path +import shutil import numpy as np import pytest -from tests.wrapper.fortran._support import _build_source_or_generated_pyi_and_import, wrapper_source +from x2py import build_pyi_extension +from tests.wrapper.fortran._support import ( + _build_source_or_generated_pyi_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, + wrapper_source, +) from x2py.runtime.handles import _NativeArrayHandoff, AllocatableArray, PointerArray ASSUMED_RANK_F90_SOURCE = wrapper_source("fassumed_rank_f90.f90") @@ -114,3 +122,42 @@ def test_assumed_rank_bridge_dispatches_each_runtime_rank_argument( right = np.ones(right_shape, dtype=np.float64, order="F") assert module.rank_pair_score(left, right) == 100 * left_rank + right_rank + 4 + + +def test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay runtime ranks one through fifteen through explicit bridge branches.""" + native_object = _compile_native_object(ASSUMED_RANK_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_assumed_rank" + shutil.copytree(CONTRACT_FIXTURES / "fassumed_rank_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fassumed_rank_f90 import rank_weighted_sum\nfrom .fassumed_rank_f90 import bump_assumed_rank\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "rank_weighted_sum") else _sole_native_module(module) + + for module in modules.values(): + for rank in (1, 2, 15): + shape = (2, *([1] * (rank - 1))) + values = np.ones(shape, dtype=np.float64, order="F") + assert module.rank_weighted_sum(values) == np.float64(rank + 2) + assert module.bump_assumed_rank(values) is None + np.testing.assert_array_equal(values, np.full(shape, rank + 1.0, order="F")) + + direct = modules["wrapper_plan"] + with pytest.raises(TypeError): + direct.rank_weighted_sum(np.float64(1.0)) + with pytest.raises(TypeError): + direct.rank_weighted_sum(np.empty((1,) * 16, dtype=np.float64, order="F")) diff --git a/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py b/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py index 97ab6c95c..3ac9bfd27 100644 --- a/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py +++ b/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py @@ -1,9 +1,17 @@ from pathlib import Path +import shutil import numpy as np import pytest -from tests.wrapper.fortran._support import _build_source_or_generated_pyi_and_import, wrapper_source +from x2py import build_pyi_extension +from tests.wrapper.fortran._support import ( + _build_source_or_generated_pyi_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, + wrapper_source, +) SOURCE = wrapper_source("multid_arrays.f90") @@ -248,3 +256,59 @@ def test_rank3_assumed_shape_accepts_fortran_ordered_strided_views(module): module.shift3_strided(c_ordered_strided_source, contiguous_out) with pytest.raises(TypeError, match=r"expected ordering \(F\)"): module.checksum3_strided(c_ordered_strided_source, contiguous_checksum) + + +def test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay dense extents, positive strides, and returned storage identity.""" + native_object = _compile_native_object(SOURCE, tmp_path / "native") + modules = {} + selected = ("scale2_contiguous", "scale2_strided", "scale2_explicit", "shift3_strided") + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_multid_arrays" + shutil.copytree(CONTRACT_FIXTURES / "multid_arrays", contract_package) + (contract_package / "__init__.pyi").write_text( + "".join(f"from .multid_arrays import {name}\n" for name in selected), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "scale2_contiguous") else _sole_native_module(module) + + for module in modules.values(): + dense = _matrix() + dense_out = np.zeros_like(dense, order="F") + assert module.scale2_contiguous(dense, dense_out) is dense_out + np.testing.assert_allclose(dense_out, 2.0 * dense) + + explicit_out = np.zeros_like(dense, order="F") + assert module.scale2_explicit(np.int32(4), np.int32(3), dense, explicit_out) is explicit_out + np.testing.assert_allclose(explicit_out, 4.0 * dense) + + strided = _strided_matrix() + strided_out = _strided_matrix_output(strided.shape) + assert module.scale2_strided(strided, strided_out) is strided_out + np.testing.assert_allclose(strided_out, 3.0 * strided) + + empty = _strided_matrix(0, 3) + empty_out = _strided_matrix_output(empty.shape) + assert module.scale2_strided(empty, empty_out) is empty_out + assert empty_out.shape == (0, 3) + + direct = modules["wrapper_plan"] + dense = _matrix() + output = np.zeros_like(dense, order="F") + with pytest.raises(TypeError): + direct.scale2_strided(_reversed_fortran_matrix(), output) + with pytest.raises(TypeError): + direct.scale2_strided(_broadcast_fortran_like_matrix(), output) + with pytest.raises(TypeError): + direct.scale2_contiguous(np.array(dense, order="C"), output) diff --git a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py index 9aa9271a1..1578ef1f0 100644 --- a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py +++ b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py @@ -94,3 +94,64 @@ def test_editable_contract_can_use_native_order_arguments_without_native_call(tm assert module.make_point(scale, point) is None assert point.total == np.float64(7.5) assert point.code == np.int32(107) + + +def test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay both fixed address boundaries through one existing native routine.""" + native_object = _compile_native_object(NATIVE_CALL_EXAMPLES_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_fixed_string_addresses" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + "from .fnative_call_examples_f90 import fixed_inout_raw, fixed_inout_storage\n", + encoding="utf-8", + ) + (contract_package / "fnative_call_examples_f90.pyi").write_text( + """from x2py.contracts import Addr, String, bind + +@bind("fixed_inout") +def fixed_inout_raw(label: Addr(String[8])) -> None: ... + +@bind("fixed_inout") +def fixed_inout_storage(label: String[8][()]) -> None: ... +""", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "fixed_inout_raw") else _sole_native_module(module) + + for module in modules.values(): + raw_label = ctypes.create_string_buffer(8) + raw_label.raw = b"abc " + assert module.fixed_inout_raw(ctypes.addressof(raw_label)) is None + assert raw_label.raw == b"Xbc !" + + storage_label = np.array("abc ", dtype="S8") + assert module.fixed_inout_storage(storage_label) is None + assert storage_label[()] == b"Xbc !" + + with pytest.raises(TypeError): + module.fixed_inout_raw("abc ") + with pytest.raises(TypeError, match="itemsize 8"): + module.fixed_inout_storage(np.array("abc", dtype="S3")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array([b"abc "], dtype="S8")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array("abc ", dtype="U8")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array(b"abc ", dtype=object)) + read_only = np.array("abc ", dtype="S8") + read_only.flags.writeable = False + with pytest.raises(TypeError, match="writeable"): + module.fixed_inout_storage(read_only) diff --git a/tests/wrapper/fortran/function_calls/test_optional_arguments.py b/tests/wrapper/fortran/function_calls/test_optional_arguments.py index fa7d64542..de80b5eb7 100644 --- a/tests/wrapper/fortran/function_calls/test_optional_arguments.py +++ b/tests/wrapper/fortran/function_calls/test_optional_arguments.py @@ -1,6 +1,7 @@ """Optional argument runtime wrapper tests.""" from pathlib import Path +import shutil import numpy as np import pytest @@ -256,3 +257,61 @@ def test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states(tm assert module.optional_scale(base=np.int32(3), factor=np.int32(6)) == np.int32(9) with pytest.raises(TypeError): module.optional_scale(np.int32(3), "bad") + + +def test_optional_array_buffers_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay omitted, explicit-None, and present ordinary array storage.""" + native_object = _compile_native_object(OPTIONAL_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_optional_arrays" + shutil.copytree(CONTRACT_FIXTURES / "foptional_f90", contract_package) + (contract_package / "foptional_f90.pyi").write_text( + """from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call + +@native_call([Arg(0), Addr(Arg(1))]) +def mutate_optional( + values: Float64[::] = ..., + amount: Float64 = ... +) -> None: ... + +@native_call([Addr(Arg(0)), Arg(1)]) +def fill_optional( + n: Int32, + values: Float64[::] = ... +) -> Returns["values", Float64[::]] | None: ... +""", + encoding="utf-8", + ) + (contract_package / "__init__.pyi").write_text( + "from .foptional_f90 import mutate_optional\nfrom .foptional_f90 import fill_optional\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "mutate_optional") else _sole_native_module(module) + + for module in modules.values(): + assert module.mutate_optional() is None + assert module.mutate_optional(None, np.float64(2.0)) is None + values = np.array([1.0, 2.0], dtype=np.float64) + assert module.mutate_optional(values, np.float64(2.5)) is None + np.testing.assert_array_equal(values, np.array([3.5, 4.5])) + + output = np.empty(3, dtype=np.float64) + assert module.fill_optional(np.int32(3), output) is output + np.testing.assert_array_equal(output, np.array([11.0, 12.0, 13.0])) + assert module.fill_optional(np.int32(3)) is None + assert module.fill_optional(np.int32(3), None) is None + + with pytest.raises(TypeError): + modules["wrapper_plan"].fill_optional(np.int32(3), np.empty(3, dtype=np.float32)) diff --git a/tests/wrapper/fortran/function_calls/test_output_arguments.py b/tests/wrapper/fortran/function_calls/test_output_arguments.py index dbcde427d..115a3a9aa 100644 --- a/tests/wrapper/fortran/function_calls/test_output_arguments.py +++ b/tests/wrapper/fortran/function_calls/test_output_arguments.py @@ -1,13 +1,18 @@ """Output argument and multiple-result runtime wrapper tests.""" from pathlib import Path +import shutil import numpy as np import pytest +from x2py import build_pyi_extension from x2py.runtime.handles import AllocatableArray from tests.wrapper.fortran._support import ( _build_source_or_generated_pyi_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, wrapper_source, ) @@ -106,3 +111,43 @@ def test_output_arguments_and_multiple_results_follow_python_projection_rules( module.fill_vector(np.int32(4), np.empty(3, dtype=np.float64)) with pytest.raises(TypeError): module.fill_matrix(np.int32(2), np.int32(3), np.empty((2, 3), dtype=np.float64, order="C")) + + +def test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): + """Replay an existing fixed-shape native output as a hidden result.""" + native_object = _compile_native_object(OUTPUTS_F90_SOURCE, tmp_path / "native") + modules = {} + contract_text = """\ +from x2py.contracts import Addr, Arg, Float64, Int32, Return, native_call + +@native_call([Addr(Arg(0)), Return("values", 0)]) +def fill_vector(n: Int32) -> Float64[n]: ... +""" + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_hidden_array_output" + shutil.copytree(CONTRACT_FIXTURES / "foutputs_f90", contract_package) + (contract_package / "foutputs_f90.pyi").write_text(contract_text, encoding="utf-8") + (contract_package / "__init__.pyi").write_text( + "from .foutputs_f90 import fill_vector\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "fill_vector") else _sole_native_module(module) + + for module in modules.values(): + np.testing.assert_array_equal(module.fill_vector(np.int32(4)), np.array([2.0, 4.0, 6.0, 8.0])) + assert module.fill_vector(np.int32(0)).shape == (0,) + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + with pytest.raises(MemoryError, match="Unable to allocate copy-return output array"): + modules["wrapper_plan"].fill_vector(np.int32(2)) diff --git a/tests/wrapper/fortran/scalars/test_verified_baseline.py b/tests/wrapper/fortran/scalars/test_verified_baseline.py index 2b1c09a8b..3b9af517f 100644 --- a/tests/wrapper/fortran/scalars/test_verified_baseline.py +++ b/tests/wrapper/fortran/scalars/test_verified_baseline.py @@ -1,6 +1,7 @@ """Verified baseline runtime wrapper tests.""" from pathlib import Path +import shutil import numpy as np import pytest @@ -12,8 +13,12 @@ _build_source_legacy_and_import, _build_source_or_generated_pyi_and_import, _build_source_wrapper_plan_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, wrapper_source, ) +from x2py import build_pyi_extension CONTRACT_FIXTURES = Path(__file__).parent / "contracts" SCALAR_FIXED_SOURCE = wrapper_source("fmath.f") @@ -157,3 +162,56 @@ def test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts( _assert_fmath_array_examples(module, suffix="_CONTIGUOUS", strided=False) _assert_array_rejects_strided_views(module, "SQUARE_R4_CONTIGUOUS") _assert_fmath_array_examples(module, suffix="_STRIDED", strided=True) + + +def test_required_array_buffers_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay one existing dense rank-one routine through a reduced contract.""" + native_object = _compile_native_object(ARRAY_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_required_array" + shutil.copytree(CONTRACT_FIXTURES / "fmath_arrays_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fmath_arrays_f90 import square_r8_contiguous\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "square_r8_contiguous") else _sole_native_module(module) + + for module in modules.values(): + values = np.array([2.0, 3.0, -4.0], dtype=np.float64) + output = np.zeros_like(values) + assert module.square_r8_contiguous(np.int32(values.size), values, output) is None + np.testing.assert_array_equal(output, values**2) + + empty = np.empty(0, dtype=np.float64) + assert module.square_r8_contiguous(np.int32(0), empty, empty.copy()) is None + + module = modules["wrapper_plan"] + valid = np.arange(4, dtype=np.float64) + output = np.zeros_like(valid) + invalid_cases = ( + np.arange(4, dtype=np.float32), + valid.reshape(2, 2), + np.arange(8, dtype=np.float64)[::2], + np.arange(4, dtype=">f8"), + np.ndarray(4, dtype=np.float64, buffer=bytearray(33), offset=1), + ) + for invalid in invalid_cases: + with pytest.raises(TypeError): + module.square_r8_contiguous(np.int32(4), invalid, output) + + read_only = valid.copy() + read_only.flags.writeable = False + with pytest.raises(TypeError, match="writeable"): + module.square_r8_contiguous(np.int32(4), read_only, output) diff --git a/tests/wrapper/fortran/strings/test_character_arguments.py b/tests/wrapper/fortran/strings/test_character_arguments.py index 61e05152e..dcdff3bdf 100644 --- a/tests/wrapper/fortran/strings/test_character_arguments.py +++ b/tests/wrapper/fortran/strings/test_character_arguments.py @@ -1,14 +1,15 @@ """Legacy and modern scalar character argument/result tests.""" from pathlib import Path +import shutil import numpy as np +import pytest from tests.wrapper.fortran._support import ( _build_source_or_generated_pyi_and_import, _compile_native_object, _import_from_build_dir, - _normalized_fortran_source, _assert_legacy_string_examples, _assert_modern_string_examples, _sole_native_module, @@ -34,16 +35,6 @@ def test_legacy_fortran_character_arguments_and_results(pyi_parity_build_mode: s pyi_parity_build_mode, ) - if pyi_parity_build_mode == "source": - bind_c_source = _normalized_fortran_source(tmp_path / "source_build" / "bind_c_fstrings_wrapper.f90") - assert "C = transfer(C_0001, C)" in bind_c_source - assert "C = transfer(C_0001, C) C_fixed = C" not in bind_c_source - assert ( - "CHAR_RESULT_DEFAULT_ptr = transfer(" - "CHAR_RESULT_DEFAULT_0001, CHAR_RESULT_DEFAULT_ptr, CHAR_RESULT_DEFAULT_len)" - ) in bind_c_source - assert "do Dummy_" not in bind_c_source - _assert_legacy_string_examples(module) @@ -81,3 +72,138 @@ def test_edited_modern_string_contract_wraps_full_axis_spelling_set(tmp_path: Pa label = np.array("abcdefgh", dtype="S8") assert module.rewrite_storage(label) is None assert label[()] == b"Ybcdefg?" + + +def test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Replay one ordinary fixed-width NumPy bytes array without descriptors.""" + native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_character_array" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90_axes", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fstrings_f90 import fixed_array_extent\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "fixed_array_extent") else _sole_native_module(module) + + for module in modules.values(): + labels = np.array([b"first", b"second"], dtype="S8") + assert module.fixed_array_extent(labels) == 16 + assert module.fixed_array_extent(np.empty(0, dtype="S8")) == 0 + + direct = modules["wrapper_plan"] + with pytest.raises(TypeError): + direct.fixed_array_extent(np.array([b"short"], dtype="S7")) + with pytest.raises(TypeError): + direct.fixed_array_extent(np.array([[b"label"]], dtype="S8")) + + +def test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + """Reuse the existing modern string unit through one scalar-input-only entry.""" + native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") + modules = [] + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_string_inputs" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "\n".join( + ( + "from .fstrings_f90 import char_code_default", + "from .fstrings_f90 import char_code_len1", + "from .fstrings_f90 import char_code_kind1", + "from .fstrings_f90 import char_code_c_char", + "from .fstrings_f90 import string_len_fixed", + "from .fstrings_f90 import string_len_assumed", + "from .fstrings_f90 import string_len_c_char", + "", + ) + ), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules.append(module if hasattr(module, "char_code_default") else _sole_native_module(module)) + + for module in modules: + assert module.char_code_default("A") == ord("A") + assert module.char_code_len1(np.str_("B")) == ord("B") + assert module.char_code_kind1("C") == ord("C") + assert module.char_code_c_char("D") == ord("D") + assert module.string_len_fixed("short ") == 5 + assert module.string_len_assumed("variable length") == 15 + assert module.string_len_assumed("") == 0 + assert module.string_len_assumed("café") == 5 + assert module.string_len_c_char("c-char ") == 6 + + with pytest.raises(TypeError, match="str"): + module.string_len_assumed(b"bytes") + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.string_len_fixed("short") + with pytest.raises(TypeError, match="embedded NUL"): + module.string_len_assumed("a\0b") + + +def test_fixed_string_results_match_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): + """Replay existing fixed direct results through a result-only contract entry.""" + native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_string_results" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "\n".join( + ( + "from .fstrings_f90 import char_result_default", + "from .fstrings_f90 import char_result_c_char", + "from .fstrings_f90 import string_result_fixed", + "from .fstrings_f90 import string_result_padded", + "from .fstrings_f90 import string_result_c_char", + "", + ) + ), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "char_result_default") else _sole_native_module(module) + + for module in modules.values(): + assert module.char_result_default() == "M" + assert module.char_result_c_char() == "C" + assert module.string_result_fixed() == "MODERN!!" + assert module.string_result_padded() == "PAD " + assert module.string_result_c_char() == "C-CHAR!!" + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + with pytest.raises(MemoryError, match="Unable to allocate copy-return output string"): + modules["wrapper_plan"].string_result_fixed() diff --git a/tests/wrapper/fortran/strings/test_character_edge_cases.py b/tests/wrapper/fortran/strings/test_character_edge_cases.py index 2a75752d5..9151f305c 100644 --- a/tests/wrapper/fortran/strings/test_character_edge_cases.py +++ b/tests/wrapper/fortran/strings/test_character_edge_cases.py @@ -1,13 +1,18 @@ """Character copy-in/copy-out, length, Unicode, and NUL tests.""" from pathlib import Path +import shutil import pytest from tests.wrapper.fortran._support import ( _build_source_or_generated_pyi_and_import, + _compile_native_object, + _import_from_build_dir, + _sole_native_module, wrapper_source, ) +from x2py import build_pyi_extension CHARACTER_EDGES_F90_SOURCE = wrapper_source("fcharacter_edges_f90.f90") CONTRACT_FIXTURES = Path(__file__).parent / "contracts" @@ -49,3 +54,140 @@ def test_fortran_character_edge_cases_follow_copy_in_copy_out_policy( module.assumed_inout("a\0b") with pytest.raises(TypeError, match="embedded NUL"): module.unicode_echo("a\0b") + + +def test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): + """Replay the existing hidden output through a reduced contract entry.""" + native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_hidden_string_result" + shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import make_out\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "make_out") else _sole_native_module(module) + + assert modules["legacy"].make_out() == "go " + assert modules["wrapper_plan"].make_out() == "go " + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + with pytest.raises(MemoryError, match="Unable to allocate copy-return output string"): + modules["wrapper_plan"].make_out() + + +def test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes( + tmp_path: Path, + monkeypatch, +): + """Replay projected and discarded mutation against one existing native routine.""" + native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_fixed_string_writeback" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import fixed_discard, fixed_replacement\n", + encoding="utf-8", + ) + (contract_package / "fcharacter_edges_f90.pyi").write_text( + """from x2py.contracts import Returns, String, bind + +@bind("fixed_inout") +def fixed_replacement(name: String[8]) -> Returns["name", String[8]]: ... + +@bind("fixed_inout") +def fixed_discard(name: String[8]) -> None: ... +""", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "fixed_replacement") else _sole_native_module(module) + + for module in modules.values(): + original = "abc " + assert module.fixed_replacement(original) == "Zbc !" + assert original == "abc " + assert module.fixed_discard(original) is None + assert original == "abc " + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.fixed_replacement("abc") + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.fixed_discard("abcdefghi") + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument name"): + modules["wrapper_plan"].fixed_replacement("abc ") + + +def test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes( + tmp_path: Path, + monkeypatch, +): + """Replay runtime-length and absent/concrete presence through a reduced entry.""" + native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") + modules = {} + for route, route_kwargs in ( + ("legacy", {"_force_legacy_wrapper_route": True}), + ("wrapper_plan", {"_force_wrapper_plan_route": True}), + ): + contract_package = tmp_path / f"{route}_assumed_optional_string_writeback" + shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import assumed_inout, optional_inout\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / route, + **route_kwargs, + ) + module = _import_from_build_dir(result.module_name, result.output_dir) + modules[route] = module if hasattr(module, "assumed_inout") else _sole_native_module(module) + + for module in modules.values(): + assumed_original = "abc" + optional_original = "abc" + assert module.assumed_inout(assumed_original) == "Qbc" + assert module.assumed_inout("") == "" + assert module.optional_inout() is None + assert module.optional_inout(None) is None + assert module.optional_inout(optional_original) == "Pbc" + assert assumed_original == "abc" + assert optional_original == "abc" + with pytest.raises(TypeError, match="embedded NUL"): + module.assumed_inout("a\0b") + with pytest.raises(TypeError, match="embedded NUL"): + module.optional_inout("a\0b") + + monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") + assert modules["wrapper_plan"].optional_inout() is None + assert modules["wrapper_plan"].optional_inout(None) is None + with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument name"): + modules["wrapper_plan"].assumed_inout("abc") + with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument label"): + modules["wrapper_plan"].optional_inout("abc") diff --git a/tests/wrapper_codegen/test_phase0d_plan_core.py b/tests/wrapper_codegen/test_phase0d_plan_core.py index a13b2c796..216cab1d1 100644 --- a/tests/wrapper_codegen/test_phase0d_plan_core.py +++ b/tests/wrapper_codegen/test_phase0d_plan_core.py @@ -8,7 +8,7 @@ from tests._shared.ownership_policy_support import parse_pyi_text from x2py.semantics.models import PYTHON_EXPORTS_METADATA -from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, PythonBarrierAction +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, ObjectKind, PythonBarrierAction from x2py.semantics.policy_completion import complete_semantic_policies from x2py.wrapper_codegen import ( DatatypeFamily, @@ -77,6 +77,7 @@ def test_planner_projects_one_shared_tree_with_explicit_backend_views(): assert first.bridge.native_name == "x" assert first.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS assert first.binding.handoff_role == first.bridge.handoff_role == first.native_call_slot.symbolic_role + assert first.object_kind is first.native_call_slot.object_kind is ObjectKind.SCALAR assert first.native_call_slot.codegen_action is CodegenAction.CALL_LOCAL_INPUT assert function.results[0].binding.codegen_action is CodegenAction.DIRECT_VALUE assert function.results[0].bridge.native_result_role in function.available_roles @@ -95,6 +96,12 @@ def test_planner_records_hidden_literals_and_hidden_result_slots(): assert function.results[0].source_kind == "hidden_output" assert function.results[0].bridge.abi_position == 3 assert function.results[0].native_call_slot is function.native_call_slots[3] + assert [slot.object_kind for slot in function.native_call_slots] == [ + None, + ObjectKind.SCALAR, + None, + ObjectKind.SCALAR, + ] def test_generator_rejects_hidden_result_native_action_disagreement(): @@ -122,7 +129,7 @@ def test_generator_rejects_hidden_result_slot_codegen_action_disagreement(): plan = _hidden_result_plan() function = plan.namespaces[0].functions[0] result = function.results[0] - edited_slot = replace(result.native_call_slot, codegen_action=CodegenAction.DIRECT_VALUE) + edited_slot = replace(result.native_call_slot, codegen_action=CodegenAction.COPY_OUT) invalid = _edit_first_function( plan, lambda item: replace( @@ -139,6 +146,24 @@ def test_generator_rejects_hidden_result_slot_codegen_action_disagreement(): WrapperCodeGenerator().generate(invalid) +def test_generator_rejects_argument_native_slot_object_kind_disagreement(): + plan = _scalar_plan() + argument = plan.namespaces[0].functions[0].arguments[0] + argument.native_call_slot.object_kind = ObjectKind.STRING + + with pytest.raises(ValueError, match="inconsistent-argument-object-kind"): + WrapperCodeGenerator().generate(plan) + + +def test_generator_rejects_result_native_slot_object_kind_disagreement(): + plan = _hidden_result_plan() + result = plan.namespaces[0].functions[0].results[0] + result.native_call_slot.object_kind = ObjectKind.STRING + + with pytest.raises(ValueError, match="inconsistent-result-object-kind"): + WrapperCodeGenerator().generate(plan) + + def test_generator_rejects_advertised_role_without_a_plan_producer(): invalid = _edit_first_function( _scalar_plan(), @@ -209,7 +234,7 @@ def hidden(x: Int32) -> Int32: ... assert [function.binding.python_name for function in plan.namespaces[0].functions] == ["visible"] -def test_support_analyzer_reports_unsupported_generation_units(): +def test_support_analyzer_reports_required_array_buffer_lane(): module = parse_pyi_text( """ def sum_values(values: Float64[:]) -> Float64: ... @@ -220,11 +245,14 @@ def sum_values(values: Float64[:]) -> Float64: ... report = WrapperPlanSupportAnalyzer().analyze(module) - assert report.supported is False - assert report.blockers[0].owner_path == "array_argument.sum_values" - assert "not a first-lane primitive scalar" in report.blockers[0].reason - with pytest.raises(ValueError, match="Unsupported wrapper-plan generation unit"): - WrapperPlanner().build(module) + assert report.supported is True + assert report.covered_lanes == ( + "array-buffer-inputs", + "scalar-direct-results", + "native-call-runtime", + ) + assert report.blockers == () + assert WrapperPlanner().build(module).namespaces[0].functions[0].arguments[0].array is not None def test_planner_fails_when_post_ir_policy_has_not_completed(): diff --git a/tests/wrapper_codegen/test_phase0e_backend_foundation.py b/tests/wrapper_codegen/test_phase0e_backend_foundation.py index 29840387c..4c5decb66 100644 --- a/tests/wrapper_codegen/test_phase0e_backend_foundation.py +++ b/tests/wrapper_codegen/test_phase0e_backend_foundation.py @@ -24,6 +24,7 @@ CSourcePrinter, CodeExpression, FortranAssignment, + FortranCall, FortranFunction, FortranModule, FortranParameter, @@ -112,6 +113,24 @@ def test_source_printers_reject_wrapper_plan_models(): FortranSourcePrinter().doprint(plan) +def test_fortran_source_printer_wraps_long_parenthesized_call_arguments(): + slices = ", ".join(f"1:values_upper_bound_{axis} + 1:values_stride_{axis}" for axis in range(4)) + source = FortranSourcePrinter().doprint( + FortranCall( + "native_scale", + ( + CodeExpression(f"values_base({slices})"), + CodeExpression(f"out_base({slices})"), + ), + ) + ) + + assert "& values_base(&" in source + assert "& 1:values_upper_bound_3 + 1:values_stride_3), &" in source + assert "& out_base(&" in source + assert max(map(len, source.splitlines())) <= 124 + + def test_source_printers_do_not_import_wrapper_plan_models(): path = REPO_ROOT / "x2py" / "wrapper_codegen" / "source_printers.py" imports = { diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index 6c9b82638..7a34a9bea 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -120,7 +120,11 @@ def test_supported_writeback_actions_select_scalar_result_behavior(codegen_actio ) function = plan.namespaces[0].functions[0] actions = tuple( - replace(action, binding=replace(action.binding, codegen_action=codegen_action)) + replace( + action, + codegen_action=codegen_action, + binding=replace(action.binding, codegen_action=codegen_action), + ) if action.phase is WritebackPhase.COPY_OUT else action for action in function.writeback_actions diff --git a/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py b/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py index ed7665c08..7fd17d2dc 100644 --- a/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py +++ b/tests/wrapper_codegen/test_phase2f_multiple_scalar_results.py @@ -7,7 +7,7 @@ import pytest from tests._shared.ownership_policy_support import parse_pyi_text -from x2py.semantics.ownership import NativeBarrierAction +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction from x2py.semantics.policy_completion import complete_semantic_policies from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner @@ -36,6 +36,8 @@ def test_multiple_scalar_result_plan_has_ordered_binding_consumers_and_shared_hi ] assert direct.native_call_slot is None assert hidden.native_call_slot is function.native_call_slots[hidden.bridge.abi_position] + assert direct.binding.codegen_action is CodegenAction.DIRECT_VALUE + assert hidden.binding.codegen_action is CodegenAction.DIRECT_VALUE assert hidden.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS assert direct.bridge.native_result_role in function.available_roles assert hidden.bridge.native_result_role in function.available_roles diff --git a/tests/wrapper_codegen/test_phase5a_string_inputs.py b/tests/wrapper_codegen/test_phase5a_string_inputs.py new file mode 100644 index 000000000..05a5217da --- /dev/null +++ b/tests/wrapper_codegen/test_phase5a_string_inputs.py @@ -0,0 +1,108 @@ +"""Direct-plan required scalar string-value input lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, PythonBarrierAction +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import ArgumentHandoffMode, BridgeDataAction +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanSupportAnalyzer, WrapperPlanner +from x2py.wrapper_codegen.plan import DatatypeFamily + + +def _string_input_module(): + module = parse_pyi_text( + """ +def fixed(text: String[8]) -> Int32: ... +def assumed(text: String) -> Int32: ... +""", + module_name="string_inputs", + ) + complete_semantic_policies(module) + return module + + +def _string_input_plan(): + return WrapperPlanner().build(_string_input_module()) + + +def test_required_string_values_reuse_argument_plan_with_character_handoff_facts(): + module = _string_input_module() + report = WrapperPlanSupportAnalyzer().analyze(module) + assert report.supported + assert "string-value-inputs" in report.covered_lanes + + plan = WrapperPlanner().build(module) + functions = {function.binding.python_name: function for function in plan.namespaces[0].functions} + fixed = functions["fixed"].arguments[0] + assumed = functions["assumed"].arguments[0] + + for function_name, argument in (("fixed", fixed), ("assumed", assumed)): + assert argument.datatype_family is DatatypeFamily.STRING + assert argument.binding.python_action is PythonBarrierAction.STRING_VALUE + assert argument.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert argument.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER + assert argument.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert argument.bridge.copy_reason == ( + "materialize Fortran character storage from the binding UTF-8 byte buffer" + ) + assert argument.native_call_slot.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION + assert argument.binding.length_handoff_role == argument.bridge.length_handoff_role + assert argument.binding.length_handoff_role == f"{argument.owner_path}:length" + assert argument.native_call_slot is functions[function_name].native_call_slots[0] + assert argument.native_call_slot.codegen_action is CodegenAction.CALL_LOCAL_INPUT + + assert fixed.native_call_slot.character_length == 8 + assert assumed.native_call_slot.character_length is None + + +def test_required_string_values_dispatch_to_named_binding_and_bridge_lowering(): + artifacts = WrapperCodeGenerator().generate(_string_input_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "#include " in c_source + assert "const char * text = NULL;" in c_source + assert "text = PyUnicode_AsUTF8AndSize(text_obj, &text_length);" in c_source + assert "strlen(text) != text_length" in c_source + assert "text_length != 8" in c_source + assert "must encode to exactly 8 bytes" in c_source + assert "bind_c_fixed(text, (int64_t)text_length)" in c_source + assert "bind_c_assumed(text, (int64_t)text_length)" in c_source + + assert "type(c_ptr), value :: bound_text" in bridge_source + assert "integer(c_int64_t), value :: text_length" in bridge_source + assert "character(kind=c_char), pointer, dimension(:) :: text_bytes" in bridge_source + assert "character(kind=c_char, len=text_length) :: text" in bridge_source + assert "call c_f_pointer(bound_text, text_bytes, [text_length])" in bridge_source + assert "text = transfer(text_bytes, text)" in bridge_source + assert "native_fixed(text)" in bridge_source + assert "native_assumed(text)" in bridge_source + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("missing-length", "missing-string-length-handoff"), + ("wrong-handoff", "invalid-string-handoff"), + ("wrong-copy", "invalid-string-data-action"), + ], +) +def test_string_handoff_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _string_input_plan() + argument = plan.namespaces[0].functions[0].arguments[0] + if edit == "missing-length": + argument.binding.length_handoff_role = None + argument.bridge.length_handoff_role = None + elif edit == "wrong-handoff": + argument.bridge.handoff_mode = ArgumentHandoffMode.TYPED_REFERENCE + else: + argument.bridge.data_action = BridgeDataAction.DIRECT_TRANSFER + argument.native_call_slot.bridge_data_action = BridgeDataAction.DIRECT_TRANSFER + argument.bridge.copy_reason = None + argument.native_call_slot.bridge_copy_reason = None + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase5b_fixed_string_results.py b/tests/wrapper_codegen/test_phase5b_fixed_string_results.py new file mode 100644 index 000000000..5247b59ff --- /dev/null +++ b/tests/wrapper_codegen/test_phase5b_fixed_string_results.py @@ -0,0 +1,200 @@ +"""Direct-plan fixed string result and hidden-output lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.models import RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA +from x2py.semantics.ownership import ( + CodegenAction, + DestructionPolicy, + NativeBarrierAction, + ObjectKind, + OwnershipOwner, + PythonBarrierAction, + StorageMode, + TransferMode, +) +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import BridgeDataAction +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanSupportAnalyzer, WrapperPlanner +from x2py.wrapper_codegen.plan import DatatypeFamily + + +_COPY_REASON = "copy fixed-length Fortran character output into C-owned null-terminated storage" + + +def _fixed_string_module(): + module = parse_pyi_text( + """ +def direct_label() -> String[8]: ... + +@native_call([Return("label", 0)]) +def hidden_label() -> String[8]: ... +""", + module_name="fixed_string_results", + ) + complete_semantic_policies(module) + return module + + +def _fixed_string_plan(): + return WrapperPlanner().build(_fixed_string_module()) + + +def test_fixed_strings_reuse_ordered_result_plans_with_completed_length_and_copy_facts(): + module = _fixed_string_module() + reports = {function.name: WrapperPlanSupportAnalyzer().analyze(function) for function in module.functions} + assert reports["direct_label"].covered_lanes == ( + "fixed-string-direct-results", + "native-call-runtime", + ) + assert reports["hidden_label"].covered_lanes == ( + "fixed-string-hidden-outputs", + "native-call-runtime", + ) + + plan = WrapperPlanner().build(module) + functions = {function.binding.python_name: function for function in plan.namespaces[0].functions} + direct = functions["direct_label"].results[0] + hidden = functions["hidden_label"].results[0] + + for result in (direct, hidden): + assert result.datatype_family is DatatypeFamily.STRING + assert result.character_length == 8 + assert result.object_kind is ObjectKind.STRING + assert result.ownership_owner is OwnershipOwner.PYTHON + assert result.transfer_mode is TransferMode.COPY_RETURN + assert result.destruction_policy is DestructionPolicy.PYTHON_REFCOUNT + assert result.storage_mode is StorageMode.STACK + assert result.boundary_storage_mode is StorageMode.STACK + assert result.nullable is False + assert result.binding.python_action is PythonBarrierAction.NONE + assert result.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert result.bridge.copy_reason == _COPY_REASON + + assert direct.source_kind == "direct_return" + assert direct.binding.codegen_action is CodegenAction.COPY_OUT + assert direct.bridge.native_action is NativeBarrierAction.NONE + assert direct.native_call_slot is None + + assert hidden.source_kind == "hidden_output" + assert hidden.binding.codegen_action is CodegenAction.COPY_OUT + assert hidden.bridge.native_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert hidden.native_call_slot is functions["hidden_label"].native_call_slots[0] + assert hidden.native_call_slot.object_kind is ObjectKind.STRING + assert hidden.native_call_slot.character_length == hidden.character_length + + +def test_fixed_string_results_dispatch_to_named_binding_and_bridge_copy_lowering(): + artifacts = WrapperCodeGenerator().generate(_fixed_string_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void * bind_c_direct_label(void);" in c_source + assert "void * result = NULL;" in c_source + assert "result = bind_c_direct_label();" in c_source + assert "if (result == NULL)" in c_source + assert 'Py_BuildValue("s", (const char *)result)' in c_source + assert "free(result);" in c_source + assert "void bind_c_hidden_label(void ** label);" in c_source + assert "bind_c_hidden_label(&label);" in c_source + assert 'Py_BuildValue("s", (const char *)label)' in c_source + assert "free(label);" in c_source + + assert 'function bind_c_direct_label() result(result) bind(c, name="bind_c_direct_label")' in bridge_source + assert "type(c_ptr) :: result" in bridge_source + assert "character(kind=c_char, len=8) :: result_value" in bridge_source + assert "result_value = native_direct_label()" in bridge_source + assert "result = c_malloc(9_c_size_t)" in bridge_source + assert "result_copy(1:8) = transfer(result_value, result_copy(1:8))" in bridge_source + assert "result_copy(9) = c_null_char" in bridge_source + assert 'subroutine bind_c_hidden_label(label) bind(c, name="bind_c_hidden_label")' in bridge_source + assert "character(kind=c_char, len=8) :: label_value" in bridge_source + assert "call native_hidden_label(label_value)" in bridge_source + assert "label = c_malloc(9_c_size_t)" in bridge_source + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("missing-length", "invalid-result-character-length"), + ("wrong-copy", "invalid-string-result-data-action"), + ("wrong-copy-reason", "invalid-string-result-copy-reason"), + ("wrong-object-kind", "invalid-scalar-result-datatype-family"), + ("wrong-owner", "invalid-string-result-owner"), + ("wrong-transfer", "invalid-string-result-transfer"), + ("wrong-destruction", "invalid-string-result-destruction"), + ("wrong-storage", "invalid-string-result-storage"), + ("wrong-boundary-storage", "invalid-string-result-boundary-storage"), + ("nullable", "nullable-fixed-string-result"), + ("slot-length-drift", "inconsistent-result-character-length"), + ], +) +def test_fixed_string_result_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _fixed_string_plan() + direct, hidden = ( + plan.namespaces[0].functions[0].results[0], + plan.namespaces[0].functions[1].results[0], + ) + if edit == "missing-length": + direct.character_length = None + elif edit == "wrong-copy": + direct.bridge.data_action = BridgeDataAction.DIRECT_TRANSFER + direct.bridge.copy_reason = None + elif edit == "wrong-copy-reason": + direct.bridge.copy_reason = "an edited reason" + elif edit == "wrong-object-kind": + direct.object_kind = ObjectKind.SCALAR + elif edit == "wrong-owner": + direct.ownership_owner = OwnershipOwner.NATIVE + elif edit == "wrong-transfer": + direct.transfer_mode = TransferMode.BORROWED_VIEW + elif edit == "wrong-destruction": + direct.destruction_policy = DestructionPolicy.NATIVE_OWNER + elif edit == "wrong-storage": + direct.storage_mode = StorageMode.HEAP + elif edit == "wrong-boundary-storage": + direct.boundary_storage_mode = StorageMode.HEAP + elif edit == "nullable": + direct.nullable = True + else: + hidden.native_call_slot.character_length = 7 + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) + + +def test_fixed_string_result_policy_blocks_mixed_result_aggregation_until_cleanup_is_planned(): + module = parse_pyi_text( + """ +@native_call([Return("status", 1)]) +def mixed() -> tuple[String[8], Int32]: ... +""", + module_name="mixed_string_results", + ) + complete_semantic_policies(module) + policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert policy.supported is False + assert "fixed string result lane requires exactly one Python-visible result" in policy.blockers + assert policy.results[0].ownership.kind is ObjectKind.STRING + + +def test_fixed_string_result_policy_blocks_status_error_until_failure_release_is_planned(): + module = parse_pyi_text( + """ +@raises(status="status", success=0) +@native_call([Return("label", 0), Return("status", 1)]) +def label() -> tuple[String[8], Int32]: ... +""", + module_name="string_result_with_status", + ) + complete_semantic_policies(module) + policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] + + assert policy.supported is False + assert "fixed string result with native status error requires planned failure-path release" in policy.blockers + assert len(policy.results) == 1 + assert policy.results[0].ownership.kind is ObjectKind.STRING diff --git a/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py b/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py new file mode 100644 index 000000000..d0de9eca3 --- /dev/null +++ b/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py @@ -0,0 +1,256 @@ +"""Direct-plan immutable string replacement and identity lowering.""" + +from __future__ import annotations + +from dataclasses import replace + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import ( + CodegenAction, + DestructionPolicy, + ObjectKind, + OwnershipOwner, + StorageMode, + TransferMode, +) +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import ( + BridgeDataAction, + OptionalMode, + PythonExceptionKind, + STRING_REPLACEMENT_COPY_REASON, + WritebackPhase, +) +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanSupportAnalyzer, WrapperPlanner +from x2py.wrapper_codegen.plan import BindingStatusErrorPlan, DatatypeFamily + + +def _fixed_writeback_module(): + module = parse_pyi_text( + """ +def replace_name(name: String[8]) -> Returns["name", String[8]]: ... +def discard_name(name: String[8]) -> None: ... +""", + module_name="fixed_string_writeback", + ) + complete_semantic_policies(module) + return module + + +def _fixed_writeback_plan(): + return WrapperPlanner().build(_fixed_writeback_module()) + + +def _functions(plan): + return {function.binding.python_name: function for function in plan.namespaces[0].functions} + + +def test_fixed_replacement_projects_completed_argument_and_lifecycle_facts(): + module = _fixed_writeback_module() + reports = {function.name: WrapperPlanSupportAnalyzer().analyze(function) for function in module.functions} + assert reports["replace_name"].covered_lanes == ( + "string-value-inputs", + "string-writebacks", + "native-call-runtime", + ) + assert reports["discard_name"].covered_lanes == ( + "string-value-inputs", + "void-calls", + "native-call-runtime", + ) + + functions = _functions(WrapperPlanner().build(module)) + replacement = functions["replace_name"] + argument = replacement.arguments[0] + assert argument.character_length == 8 + assert argument.object_kind is ObjectKind.STRING + assert argument.ownership_owner is OwnershipOwner.PYTHON + assert argument.transfer_mode is TransferMode.COPY_RETURN + assert argument.destruction_policy is DestructionPolicy.PYTHON_REFCOUNT + assert argument.storage_mode is StorageMode.STACK + assert argument.boundary_storage_mode is StorageMode.STACK + assert argument.nullable is False + assert argument.mutates_native is True + assert argument.projects_result is True + assert argument.result_position == 0 + assert argument.binding.codegen_action is CodegenAction.COPY_IN_OUT + assert argument.bridge.codegen_action is CodegenAction.COPY_IN_OUT + assert argument.native_call_slot.codegen_action is CodegenAction.COPY_IN_OUT + assert argument.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert argument.bridge.copy_reason == STRING_REPLACEMENT_COPY_REASON + assert tuple(action.phase for action in replacement.writeback_actions) == tuple(WritebackPhase) + assert all(action.semantic_type_name == "String" for action in replacement.writeback_actions) + assert all(action.datatype_family is DatatypeFamily.STRING for action in replacement.writeback_actions) + + identity = functions["discard_name"] + assert identity.arguments[0].binding.codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert identity.arguments[0].bridge.codegen_action is CodegenAction.CALL_LOCAL_INPUT + assert identity.arguments[0].projects_result is False + assert identity.writeback_actions == () + + +def test_fixed_string_writeback_dispatches_to_named_binding_and_bridge_lowering(): + artifacts = WrapperCodeGenerator().generate(_fixed_writeback_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void bind_c_replace_name(char * name, int64_t name_length);" in c_source + assert "const char * name_source = NULL;" in c_source + assert "char * name = NULL;" in c_source + assert "name = (char *)x2py_malloc((size_t)name_length + 1);" in c_source + assert 'PyExc_MemoryError, "Unable to allocate mutable string buffer for argument name."' in c_source + assert "memcpy(name, name_source, (size_t)name_length);" in c_source + assert "name[name_length] = '\\0';" in c_source + assert "bind_c_replace_name(name, (int64_t)name_length);" in c_source + assert 'Py_BuildValue("s", (const char *)name)' in c_source + assert c_source.index('Py_BuildValue("s", (const char *)name)') < c_source.index("free(name);") + assert c_source.index("free(name);") < c_source.index("if (result_obj == NULL)") + assert "void bind_c_discard_name(const char * name, int64_t name_length);" in c_source + assert "bind_c_discard_name(name, (int64_t)name_length);" in c_source + + assert "call c_f_pointer(bound_name, name_bytes, [name_length + 1])" in bridge_source + assert "name = transfer(name_bytes(1:name_length), name)" in bridge_source + assert "call native_replace_name(name)" in bridge_source + assert "name_bytes(1:name_length) = transfer(name, name_bytes(1:name_length))" in bridge_source + assert "name_bytes(name_length + 1) = c_null_char" in bridge_source + assert "call c_f_pointer(bound_name, name_bytes, [name_length])" in bridge_source + assert "call native_discard_name(name)" in bridge_source + + +def test_fixed_string_replacement_allocation_runs_after_other_argument_conversions(): + module = parse_pyi_text( + 'def replace_name(name: String[8], count: Int32) -> Returns["name", String[8]]: ...', + module_name="fixed_string_cleanup_order", + ) + complete_semantic_policies(module) + artifacts = WrapperCodeGenerator().generate(WrapperPlanner().build(module)) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + + assert c_source.index("PyArray_IsScalar(count_obj, Int)") < c_source.index( + "name = (char *)x2py_malloc((size_t)name_length + 1)" + ) + + +def test_assumed_and_optional_string_replacements_reuse_runtime_length_and_presence_facts(): + module = parse_pyi_text( + """ +def assumed(name: String) -> Returns["name", String]: ... +def optional(label: String = ...) -> Returns["label", String] | None: ... +def optional_identity(label: String = ...) -> None: ... +""", + module_name="assumed_optional_string_writeback", + ) + complete_semantic_policies(module) + reports = {function.name: WrapperPlanSupportAnalyzer().analyze(function) for function in module.functions} + assert reports["assumed"].covered_lanes == ( + "string-value-inputs", + "string-writebacks", + "native-call-runtime", + ) + assert reports["optional"].covered_lanes == ( + "string-value-inputs", + "string-optional-inputs", + "string-writebacks", + "native-call-runtime", + ) + assert reports["optional_identity"].covered_lanes == ( + "string-value-inputs", + "string-optional-inputs", + "void-calls", + "native-call-runtime", + ) + + functions = _functions(WrapperPlanner().build(module)) + for name in ("assumed", "optional", "optional_identity"): + argument = functions[name].arguments[0] + assert argument.character_length is None + assert argument.native_call_slot.character_length is None + assert functions["assumed"].arguments[0].binding.optional_mode is OptionalMode.REQUIRED + assert functions["optional"].arguments[0].binding.optional_mode is OptionalMode.NULLABLE_VALUE + assert functions["optional"].arguments[0].nullable is False + assert functions["optional_identity"].arguments[0].binding.codegen_action is CodegenAction.CALL_LOCAL_INPUT + + +def test_assumed_and_optional_string_lowering_guards_presence_copyback_and_cleanup(): + module = parse_pyi_text( + """ +def assumed(name: String) -> Returns["name", String]: ... +def optional(label: String = ...) -> Returns["label", String] | None: ... +def optional_identity(label: String = ...) -> None: ... +""", + module_name="assumed_optional_string_writeback", + ) + complete_semantic_policies(module) + artifacts = WrapperCodeGenerator().generate(WrapperPlanner().build(module)) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void bind_c_assumed(char * name, int64_t name_length);" in c_source + assert "void bind_c_optional(char * label, int64_t label_length);" in c_source + assert "PyObject * label_obj = Py_None;" in c_source + assert "if (label_obj != Py_None)" in c_source + assert "bind_c_optional(label, (int64_t)label_length);" in c_source + assert "if (label == NULL)" in c_source + assert "Py_INCREF(Py_None);" in c_source + assert 'result_obj = Py_BuildValue("s", (const char *)label);' in c_source + assert "void bind_c_optional_identity(const char * label, int64_t label_length);" in c_source + + assert "character(kind=c_char, len=name_length) :: name" in bridge_source + assert "if (c_associated(bound_label)) then" in bridge_source + assert "call native_optional(label=label)" in bridge_source + assert "call native_optional()" in bridge_source + assert "label_bytes(1:label_length) = transfer(label, label_bytes(1:label_length))" in bridge_source + assert "label_bytes(label_length + 1) = c_null_char" in bridge_source + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("wrong-owner", "invalid-string-replacement-owner"), + ("wrong-copy-reason", "invalid-string-copy-reason"), + ("missing-cleanup", "missing-writeback-phase"), + ("lifecycle-type-drift", "inconsistent-lifecycle-type"), + ("descriptor-presence", "invalid-string-optional-mode"), + ], +) +def test_fixed_string_writeback_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _fixed_writeback_plan() + function = _functions(plan)["replace_name"] + argument = function.arguments[0] + if edit == "wrong-owner": + argument.ownership_owner = OwnershipOwner.NATIVE + elif edit == "wrong-copy-reason": + argument.bridge.copy_reason = "an edited copy reason" + argument.native_call_slot.bridge_copy_reason = "an edited copy reason" + elif edit == "missing-cleanup": + function.writeback_actions = tuple( + action for action in function.writeback_actions if action.phase is not WritebackPhase.CLEANUP + ) + elif edit == "lifecycle-type-drift": + copy_out = next(action for action in function.writeback_actions if action.phase is WritebackPhase.COPY_OUT) + copy_out.semantic_type_name = "Int32" + else: + argument.binding.optional_mode = OptionalMode.DESCRIPTOR + argument.bridge.optional_mode = OptionalMode.DESCRIPTOR + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) + + +def test_fixed_string_writeback_status_edit_fails_at_generator_validation(): + plan = _fixed_writeback_plan() + function = _functions(plan)["replace_name"] + function.binding = replace( + function.binding, + status_error=BindingStatusErrorPlan( + status_role="missing:status", + message_role=None, + success=0, + exception_kind=PythonExceptionKind.RUNTIME_ERROR, + ), + ) + + with pytest.raises(ValueError, match="string-writeback-with-status-error"): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase5d_string_addresses.py b/tests/wrapper_codegen/test_phase5d_string_addresses.py new file mode 100644 index 000000000..e1ae9ddb5 --- /dev/null +++ b/tests/wrapper_codegen/test_phase5d_string_addresses.py @@ -0,0 +1,162 @@ +"""Direct-plan mutable fixed string storage and raw-address lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import ( + CodegenAction, + DestructionPolicy, + NativeBarrierAction, + ObjectKind, + OwnershipOwner, + PythonBarrierAction, + StorageMode, + TransferMode, +) +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + BridgeDataAction, + RAW_STRING_ADDRESS_COPY_REASON, + STRING_STORAGE_COPY_REASON, +) +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanSupportAnalyzer, WrapperPlanner + + +def _string_address_module(): + module = parse_pyi_text( + """ +def storage(label: String[8][()]) -> None: ... +def raw(label: Addr(String[8])) -> None: ... +""", + module_name="fixed_string_addresses", + ) + complete_semantic_policies(module) + return module + + +def _string_address_plan(): + return WrapperPlanner().build(_string_address_module()) + + +def _functions(plan): + return {function.binding.python_name: function for function in plan.namespaces[0].functions} + + +def test_string_address_plans_keep_completed_ownership_length_and_copy_facts(): + module = _string_address_module() + reports = {function.name: WrapperPlanSupportAnalyzer().analyze(function) for function in module.functions} + assert reports["storage"].covered_lanes == ( + "string-storage-inputs", + "void-calls", + "native-call-runtime", + ) + assert reports["raw"].covered_lanes == ( + "string-raw-address-inputs", + "void-calls", + "native-call-runtime", + ) + + functions = _functions(WrapperPlanner().build(module)) + storage = functions["storage"].arguments[0] + raw = functions["raw"].arguments[0] + for argument in (storage, raw): + assert argument.character_length == 8 + assert argument.object_kind is ObjectKind.STRING + assert argument.ownership_owner is OwnershipOwner.CALLER + assert argument.transfer_mode is TransferMode.IN_PLACE + assert argument.destruction_policy is DestructionPolicy.CALLER + assert argument.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert argument.bridge.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + assert argument.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert argument.binding.length_handoff_role is None + assert argument.bridge.length_handoff_role is None + assert argument.mutates_native is True + assert argument.projects_result is False + + assert storage.binding.python_action is PythonBarrierAction.STRING_STORAGE + assert storage.bridge.native_action is NativeBarrierAction.PASS_STORAGE_ADDRESS + assert storage.storage_mode is StorageMode.ALIAS + assert storage.boundary_storage_mode is StorageMode.ALIAS + assert storage.bridge.copy_reason == STRING_STORAGE_COPY_REASON + assert raw.binding.python_action is PythonBarrierAction.RAW_ADDRESS + assert raw.bridge.native_action is NativeBarrierAction.PASS_RAW_ADDRESS + assert raw.storage_mode is StorageMode.STACK + assert raw.boundary_storage_mode is StorageMode.STACK + assert raw.bridge.copy_reason == RAW_STRING_ADDRESS_COPY_REASON + + +def test_string_addresses_dispatch_to_named_binding_and_bridge_lowering(): + artifacts = WrapperCodeGenerator().generate(_string_address_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void bind_c_storage(void * label);" in c_source + assert "PyArray_TYPE((PyArrayObject *)label_obj) != NPY_STRING" in c_source + assert "PyArray_NDIM((PyArrayObject *)label_obj) != 0" in c_source + assert "PyArray_ITEMSIZE((PyArrayObject *)label_obj) != 8" in c_source + assert "PyArray_ISNOTSWAPPED((PyArrayObject *)label_obj)" in c_source + assert "PyArray_ISALIGNED((PyArrayObject *)label_obj)" in c_source + assert "PyArray_ISWRITEABLE((PyArrayObject *)label_obj)" in c_source + assert "label = PyArray_DATA((PyArrayObject *)label_obj);" in c_source + assert "void bind_c_raw(void * label);" in c_source + assert "if (!PyLong_Check(label_obj))" in c_source + assert "label = PyLong_AsVoidPtr(label_obj);" in c_source + assert "x2py_malloc" not in c_source + + assert 'subroutine bind_c_storage(bound_label) bind(c, name="bind_c_storage")' in bridge_source + assert 'subroutine bind_c_raw(bound_label) bind(c, name="bind_c_raw")' in bridge_source + assert bridge_source.count("type(c_ptr), value :: bound_label") == 2 + assert bridge_source.count("character(kind=c_char, len=8) :: label") == 2 + assert bridge_source.count("call c_f_pointer(bound_label, label_bytes, [8])") == 2 + assert bridge_source.count("label = transfer(label_bytes, label)") == 2 + assert "call native_storage(label)" in bridge_source + assert "call native_raw(label)" in bridge_source + assert bridge_source.count("label_bytes(1:8) = transfer(label, label_bytes(1:8))") == 2 + assert "label_length" not in bridge_source + assert "c_null_char" not in "\n".join(line for line in bridge_source.splitlines() if "label_bytes" in line) + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("missing-length", "invalid-string-storage-length"), + ("wrong-owner", "invalid-string-storage-owner"), + ("runtime-length-role", "unexpected-string-storage-length-handoff"), + ("wrong-copy-reason", "invalid-string-storage-copy-reason"), + ("missing-mutation", "string-storage-without-mutation"), + ("raw-alias-storage", "invalid-string-raw-address-storage"), + ("raw-projection", "string-raw-address-projects-result"), + ], +) +def test_string_address_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _string_address_plan() + functions = _functions(plan) + storage = functions["storage"].arguments[0] + raw = functions["raw"].arguments[0] + if edit == "missing-length": + storage.character_length = None + storage.native_call_slot.character_length = None + elif edit == "wrong-owner": + storage.ownership_owner = OwnershipOwner.NATIVE + elif edit == "runtime-length-role": + role = f"{storage.owner_path}:length" + storage.binding.length_handoff_role = role + storage.bridge.length_handoff_role = role + elif edit == "wrong-copy-reason": + storage.bridge.copy_reason = "an edited reason" + storage.native_call_slot.bridge_copy_reason = "an edited reason" + elif edit == "missing-mutation": + storage.mutates_native = False + storage.binding.writable = False + elif edit == "raw-alias-storage": + raw.storage_mode = StorageMode.ALIAS + else: + raw.projects_result = True + raw.result_position = 0 + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase6a_array_buffers.py b/tests/wrapper_codegen/test_phase6a_array_buffers.py new file mode 100644 index 000000000..dbb023167 --- /dev/null +++ b/tests/wrapper_codegen/test_phase6a_array_buffers.py @@ -0,0 +1,113 @@ +"""Direct-plan required dense rank-one primitive-array input lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import ( + CodegenAction, + DestructionPolicy, + NativeBarrierAction, + ObjectKind, + OwnershipOwner, + PythonBarrierAction, + StorageMode, + TransferMode, +) +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import ArgumentHandoffMode, BridgeDataAction +from x2py.wrapper_codegen import ArrayHandoffPlan, WrapperCodeGenerator, WrapperPlanner +from x2py.wrapper_codegen.plan import DatatypeFamily + + +def _array_module(): + module = parse_pyi_text( + "def sum_values(values: Float64[:]) -> Float64: ...\n", + module_name="array_buffers", + ) + complete_semantic_policies(module) + return module + + +def _array_plan(): + return WrapperPlanner().build(_array_module()) + + +def test_required_array_buffer_has_one_printable_editable_handoff_plan(): + function = _array_plan().namespaces[0].functions[0] + argument = function.arguments[0] + + assert argument.object_kind is ObjectKind.NUMPY_ARRAY + assert argument.ownership_owner is OwnershipOwner.CALLER + assert argument.transfer_mode is TransferMode.IN_PLACE + assert argument.destruction_policy is DestructionPolicy.CALLER + assert argument.storage_mode is StorageMode.STACK + assert argument.boundary_storage_mode is StorageMode.STACK + assert argument.datatype_family is DatatypeFamily.REAL + assert argument.binding.python_action is PythonBarrierAction.ARRAY_STORAGE + assert argument.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert argument.bridge.native_action is NativeBarrierAction.PASS_ARRAY_BUFFER + assert argument.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER + assert argument.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + + assert isinstance(argument.array, ArrayHandoffPlan) + assert argument.array is argument.native_call_slot.array + assert argument.native_call_slot.object_kind is ObjectKind.NUMPY_ARRAY + assert argument.array.rank == 1 + assert argument.array.shape == (":",) + assert argument.array.axes == ("dense",) + assert argument.array.contiguous is True + assert argument.array.data_role == argument.binding.handoff_role + assert argument.array.extent_roles == (f"{argument.owner_path}:extent:0",) + assert argument.array.upper_bound_roles == () + assert argument.array.stride_roles == () + + +def test_required_array_buffer_dispatches_through_named_binding_and_bridge_methods(): + artifacts = WrapperCodeGenerator().generate(_array_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "double bind_c_sum_values(void * values, int64_t values_extent_0);" in c_source + assert "PyArray_TYPE((PyArrayObject *)values_obj) != NPY_FLOAT64" in c_source + assert "PyArray_NDIM((PyArrayObject *)values_obj) != 1" in c_source + assert "PyArray_ISNOTSWAPPED((PyArrayObject *)values_obj)" in c_source + assert "PyArray_ISALIGNED((PyArrayObject *)values_obj)" in c_source + assert "PyArray_IS_C_CONTIGUOUS((PyArrayObject *)values_obj)" in c_source + assert "PyArray_IS_F_CONTIGUOUS((PyArrayObject *)values_obj)" in c_source + assert "PyArray_ISWRITEABLE((PyArrayObject *)values_obj)" in c_source + assert "values = PyArray_DATA((PyArrayObject *)values_obj);" in c_source + assert "values_extent_0 = (int64_t)PyArray_DIM((PyArrayObject *)values_obj, 0);" in c_source + assert "result = bind_c_sum_values(values, values_extent_0);" in c_source + + assert "type(c_ptr), value :: bound_values" in bridge_source + assert "integer(c_int64_t), value :: values_extent_0" in bridge_source + assert "real(c_double), pointer, dimension(:) :: values" in bridge_source + assert "call c_f_pointer(bound_values, values, [values_extent_0])" in bridge_source + assert "result = native_sum_values(values)" in bridge_source + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("rank", "inconsistent-array-rank"), + ("axis", "invalid-array-axis-modes"), + ("role", "inconsistent-array-data-role"), + ("action", "invalid-array-data-action"), + ], +) +def test_array_handoff_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _array_plan() + argument = plan.namespaces[0].functions[0].arguments[0] + if edit == "rank": + argument.array.rank = 2 + elif edit == "axis": + argument.array.axes = ("strided",) + elif edit == "role": + argument.array.data_role = "edited:data-role" + else: + argument.bridge.data_action = BridgeDataAction.DIRECT_TRANSFER + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py new file mode 100644 index 000000000..dc2d0564c --- /dev/null +++ b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py @@ -0,0 +1,70 @@ +"""Declared extents, flat storage, dense rank, and order lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + +def _dense_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Annotated, Flat, Float64, Int32, ORDER_C, ORDER_F + +def dense_f(rows: Int32, cols: Int32, values: Annotated[Float64[rows, cols], ORDER_F]) -> None: ... +def dense_c(rows: Int32, cols: Int32, values: Annotated[Float64[rows, cols], ORDER_C]) -> None: ... +def flat(n: Int32, values: Float64[Flat]) -> None: ... +""", + module_name="dense_array_shapes", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_dense_array_plan_records_extent_dependencies_flat_storage_and_order(): + functions = {function.binding.python_name: function for function in _dense_plan().namespaces[0].functions} + dense_f = functions["dense_f"].arguments[-1].array + dense_c = functions["dense_c"].arguments[-1].array + flat = functions["flat"].arguments[-1].array + + assert dense_f is not None + assert dense_f.rank == 2 + assert dense_f.shape == ("rows", "cols") + assert dense_f.order == "ORDER_F" + assert dense_f.extent_reference_roles == ( + ("dense_array_shapes.dense_f.rows:value",), + ("dense_array_shapes.dense_f.cols:value",), + ) + assert dense_c is not None + assert dense_c.order == "ORDER_C" + assert flat is not None + assert flat.rank == 1 + assert flat.shape == (":",) + assert flat.category == "assumed_size" + + +def test_dense_array_lowering_uses_planned_shape_checks_and_bridge_orientation(): + artifacts = WrapperCodeGenerator().generate(_dense_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "PyArray_DIM((PyArrayObject *)values_obj, 0) != (npy_intp)(rows)" in c_source + assert "PyArray_DIM((PyArrayObject *)values_obj, 1) != (npy_intp)(cols)" in c_source + assert "!PyArray_IS_F_CONTIGUOUS((PyArrayObject *)values_obj)" in c_source + assert "!PyArray_IS_C_CONTIGUOUS((PyArrayObject *)values_obj)" in c_source + assert "call c_f_pointer(bound_values, values, [values_extent_0, values_extent_1])" in bridge_source + assert "call c_f_pointer(bound_values, values, [values_extent_1, values_extent_0])" in bridge_source + assert "subroutine bind_c_flat(n, bound_values, values_extent_0)" in bridge_source + + +def test_unavailable_dense_extent_role_fails_before_backend_lowering(): + plan = _dense_plan() + array = plan.namespaces[0].functions[0].arguments[-1].array + assert array is not None + array.extent_reference_roles = (("edited.missing:value",), array.extent_reference_roles[1]) + + with pytest.raises(ValueError, match="unavailable-array-extent-reference"): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase6c_strided_arrays.py b/tests/wrapper_codegen/test_phase6c_strided_arrays.py new file mode 100644 index 000000000..a40d97ed8 --- /dev/null +++ b/tests/wrapper_codegen/test_phase6c_strided_arrays.py @@ -0,0 +1,66 @@ +"""Positive-strided ordinary array view lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + +def _strided_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Annotated, Float64, ORDER_F + +def strided(values: Annotated[Float64[::, ::], ORDER_F]) -> None: ... +""", + module_name="strided_arrays", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_strided_array_plan_names_bounds_and_element_strides_explicitly(): + argument = _strided_plan().namespaces[0].functions[0].arguments[0] + array = argument.array + + assert array is not None + assert array.rank == 2 + assert array.axes == ("strided", "strided") + assert array.contiguous is False + assert array.upper_bound_roles == ( + f"{argument.owner_path}:upper-bound:0", + f"{argument.owner_path}:upper-bound:1", + ) + assert array.stride_roles == ( + f"{argument.owner_path}:stride:0", + f"{argument.owner_path}:stride:1", + ) + + +def test_strided_array_lowering_validates_and_passes_one_explicit_bridge_slice(): + artifacts = WrapperCodeGenerator().generate(_strided_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "must use positive element strides" in c_source + assert "must use a Fortran-oriented non-overlapping view" in c_source + assert "values_upper_bound_0" in c_source + assert "values_stride_1" in c_source + assert "real(c_double), pointer, dimension(:, :) :: values_base" in bridge_source + assert ( + "values_base(1:values_upper_bound_0 + 1:values_stride_0, 1:values_upper_bound_1 + 1:values_stride_1)" + ) in bridge_source + assert max(map(len, bridge_source.splitlines())) <= 132 + + +def test_strided_role_edit_fails_before_backend_lowering(): + plan = _strided_plan() + array = plan.namespaces[0].functions[0].arguments[0].array + assert array is not None + array.stride_roles = array.stride_roles[:1] + + with pytest.raises(ValueError, match="invalid-array-stride-roles"): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase6d_array_output_identity.py b/tests/wrapper_codegen/test_phase6d_array_output_identity.py new file mode 100644 index 000000000..818e87de8 --- /dev/null +++ b/tests/wrapper_codegen/test_phase6d_array_output_identity.py @@ -0,0 +1,55 @@ +"""Projected ordinary outputs preserve their original Python array identity.""" + +from __future__ import annotations + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import CodegenAction, ObjectKind, OwnershipOwner, TransferMode +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner +from x2py.wrapper_codegen.plan import WritebackPhase + + +def _output_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Float64, Int32, Returns + +def fill(n: Int32, values: Float64[n]) -> Returns["values", Float64[n]]: ... +def fill_two( + n: Int32, + left: Float64[n], + right: Float64[n], +) -> tuple[Returns["left", Float64[n]], Returns["right", Float64[n]]]: ... +""", + module_name="array_output_identity", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_projected_array_identity_uses_one_completed_in_place_copy_out_action(): + function = _output_plan().namespaces[0].functions[0] + argument = function.arguments[-1] + action = function.writeback_actions[0] + + assert argument.object_kind is ObjectKind.NUMPY_ARRAY + assert argument.ownership_owner is OwnershipOwner.CALLER + assert argument.transfer_mode is TransferMode.IN_PLACE + assert argument.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + assert action.object_kind is ObjectKind.NUMPY_ARRAY + assert action.phase is WritebackPhase.COPY_OUT + assert action.binding is not None + assert action.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + + +def test_projected_array_lowering_increfs_original_objects_and_reuses_tuple_aggregation(): + artifacts = WrapperCodeGenerator().generate(_output_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + + assert "PyObject * result_obj = values_obj;" in c_source + assert "Py_INCREF(result_obj);" in c_source + assert "PyObject * result_0_obj = left_obj;" in c_source + assert "PyObject * result_1_obj = right_obj;" in c_source + assert "PyTuple_New(2)" in c_source + assert "PyTuple_SET_ITEM(result_obj, 0, result_0_obj)" in c_source + assert "PyTuple_SET_ITEM(result_obj, 1, result_1_obj)" in c_source diff --git a/tests/wrapper_codegen/test_phase6e_array_results.py b/tests/wrapper_codegen/test_phase6e_array_results.py new file mode 100644 index 000000000..6d4dc68dc --- /dev/null +++ b/tests/wrapper_codegen/test_phase6e_array_results.py @@ -0,0 +1,99 @@ +"""Fixed-shape direct and hidden ordinary array result lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, ObjectKind, OwnershipOwner, TransferMode +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import BridgeDataAction, ORDINARY_ARRAY_RESULT_COPY_REASON +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + +def _result_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Float64, Int32, Return, native_call + +def direct(n: Int32) -> Float64[n]: ... + +@native_call([Return("out", 0)]) +def hidden() -> Float64[3]: ... +""", + module_name="ordinary_array_results", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_array_results_record_producer_shape_copy_ownership_and_shared_hidden_slot(): + direct_function, hidden_function = _result_plan().namespaces[0].functions + direct = direct_function.results[0] + hidden = hidden_function.results[0] + + for result in (direct, hidden): + assert result.object_kind is ObjectKind.NUMPY_ARRAY + assert result.ownership_owner is OwnershipOwner.PYTHON + assert result.transfer_mode is TransferMode.COPY_RETURN + assert result.array is not None + assert result.array.rank == 1 + assert result.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert result.bridge.copy_reason == ORDINARY_ARRAY_RESULT_COPY_REASON + assert direct.source_kind == "direct_return" + assert direct.binding.codegen_action is CodegenAction.COPY_OUT + assert direct.bridge.native_action is NativeBarrierAction.NONE + assert direct.native_call_slot is None + assert hidden.source_kind == "hidden_output" + assert hidden.binding.codegen_action is CodegenAction.COPY_OUT + assert hidden.bridge.native_action is NativeBarrierAction.PASS_ARRAY_BUFFER + assert hidden.array is hidden.native_call_slot.array + assert hidden.native_call_slot.object_kind is ObjectKind.NUMPY_ARRAY + + +def test_array_result_lowering_allocates_bridge_copy_then_python_owned_numpy_storage(): + artifacts = WrapperCodeGenerator().generate(_result_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "void * bind_c_direct(int32_t n);" in c_source + assert "PyArray_EMPTY(1, result_obj_dims, NPY_FLOAT64, 0)" in c_source + assert "memcpy(PyArray_DATA((PyArrayObject *)result_obj), result" in c_source + assert "free(result);" in c_source + assert "void bind_c_hidden(void ** out);" in c_source + assert "free(out);" in c_source + assert "real(c_double), dimension(n) :: result_value" in bridge_source + assert "result = c_malloc(max(1_c_size_t, c_sizeof(result_value)))" in bridge_source + assert "result_copy = reshape(result_value, [size(result_value)])" in bridge_source + assert "real(c_double), dimension(3) :: out_value" in bridge_source + assert "call native_hidden(out_value)" in bridge_source + + +@pytest.mark.parametrize( + ("edit", "diagnostic"), + [ + ("rank", "invalid-array-result-rank"), + ("order", "invalid-array-result-order"), + ("copy", "invalid-array-result-copy-reason"), + ("slot", "inconsistent-result-array-handoff"), + ], +) +def test_array_result_plan_edits_fail_before_backend_lowering(edit: str, diagnostic: str): + plan = _result_plan() + direct = plan.namespaces[0].functions[0].results[0] + hidden = plan.namespaces[0].functions[1].results[0] + if edit == "rank": + direct.array.rank = None + elif edit == "order": + direct.array.order = "ORDER_C" + direct.array.rank = 2 + direct.array.shape = ("2", "2") + direct.array.extent_roles = ("edited:extent:0", "edited:extent:1") + direct.array.extent_reference_roles = ((), ()) + elif edit == "copy": + direct.bridge.copy_reason = "edited" + else: + hidden.native_call_slot.array = direct.array + + with pytest.raises(ValueError, match=diagnostic): + WrapperCodeGenerator().generate(plan) diff --git a/tests/wrapper_codegen/test_phase6f_optional_assumed_character_arrays.py b/tests/wrapper_codegen/test_phase6f_optional_assumed_character_arrays.py new file mode 100644 index 000000000..6afa654b3 --- /dev/null +++ b/tests/wrapper_codegen/test_phase6f_optional_assumed_character_arrays.py @@ -0,0 +1,131 @@ +"""Optional, assumed-rank, and fixed-width character array lowering.""" + +from __future__ import annotations + +import pytest + +from tests._shared.ownership_policy_support import parse_pyi_text +from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, ObjectKind +from x2py.semantics.policy_completion import complete_semantic_policies +from x2py.semantics.wrapper_policy import BridgeDataAction, OptionalMode +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner +from x2py.wrapper_codegen.plan import DatatypeFamily + + +def _later_array_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Float64, String + +def optional(values: Float64[:] = ...) -> None: ... +def any_rank(values: Float64[...]) -> Float64: ... +def labels(values: String[8][:]) -> None: ... +""", + module_name="later_array_buffers", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def _character_array_result_plan(): + module = parse_pyi_text( + """ +def direct_labels() -> String[5][3]: ... + +@native_call([Return("labels", 0)]) +def hidden_labels() -> String[4][2]: ... +""", + module_name="character_array_results", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + +def test_optional_assumed_rank_and_character_arrays_have_explicit_distinct_roles(): + functions = {function.binding.python_name: function for function in _later_array_plan().namespaces[0].functions} + optional = functions["optional"].arguments[0] + assumed = functions["any_rank"].arguments[0].array + character = functions["labels"].arguments[0].array + + assert optional.binding.optional_mode is OptionalMode.NULLABLE_VALUE + assert optional.bridge.optional_mode is OptionalMode.NULLABLE_VALUE + assert assumed is not None + assert assumed.rank is None + assert assumed.runtime_rank_role == "later_array_buffers.any_rank.values:rank" + assert len(assumed.extent_roles) == 15 + assert character is not None + assert character.rank == 1 + assert character.itemsize == 8 + assert character.itemsize_role == "later_array_buffers.labels.values:itemsize" + + +def test_optional_assumed_rank_and_character_lowering_follow_named_plan_fields(): + artifacts = WrapperCodeGenerator().generate(_later_array_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "PyObject * values_obj = Py_None;" in c_source + assert "if (values_obj != Py_None)" in c_source + assert "PyArray_NDIM((PyArrayObject *)values_obj) < 1" in c_source + assert "values_rank = (int64_t)PyArray_NDIM" in c_source + assert "PyArray_TYPE((PyArrayObject *)values_obj) != NPY_STRING" in c_source + assert "values_itemsize != 8" in c_source + assert "if (c_associated(bound_values)) then" in bridge_source + assert "select case (values_rank)" in bridge_source + assert "case (1)" in bridge_source + assert "case (15)" in bridge_source + assert "character(kind=c_char, len=8), pointer, dimension(:) :: values" in bridge_source + assert max(map(len, bridge_source.splitlines())) <= 132 + + +def test_character_itemsize_edit_fails_before_backend_lowering(): + plan = _later_array_plan() + character = plan.namespaces[0].functions[2].arguments[0].array + assert character is not None + character.itemsize_role = None + + with pytest.raises(ValueError, match="invalid-array-itemsize"): + WrapperCodeGenerator().generate(plan) + + +def test_fixed_width_character_array_results_reuse_the_ordinary_array_copy_plan(): + direct_function, hidden_function = _character_array_result_plan().namespaces[0].functions + direct = direct_function.results[0] + hidden = hidden_function.results[0] + + for result, itemsize in ((direct, 5), (hidden, 4)): + assert result.object_kind is ObjectKind.NUMPY_ARRAY + assert result.datatype_family is DatatypeFamily.STRING + assert result.array is not None + assert result.array.itemsize == itemsize + assert result.character_length == itemsize + assert result.binding.codegen_action is CodegenAction.COPY_OUT + assert result.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + assert direct.bridge.native_action is NativeBarrierAction.NONE + assert hidden.bridge.native_action is NativeBarrierAction.PASS_ARRAY_BUFFER + assert hidden.native_call_slot.object_kind is ObjectKind.NUMPY_ARRAY + + +def test_fixed_width_character_array_results_lower_itemsize_into_both_backends(): + artifacts = WrapperCodeGenerator().generate(_character_array_result_plan()) + c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + assert "PyArray_New(&PyArray_Type, 1, result_obj_dims, NPY_STRING, NULL, NULL, 5, 0, NULL)" in c_source + assert "PyArray_New(&PyArray_Type, 1, result_obj_dims, NPY_STRING, NULL, NULL, 4, 0, NULL)" in c_source + assert "character(kind=c_char, len=5), dimension(3) :: result_value" in bridge_source + assert "character(kind=c_char), pointer, dimension(:) :: result_copy" in bridge_source + assert "5_c_size_t * size(result_value, kind=c_size_t)" in bridge_source + assert "result_copy = transfer(result_value, result_copy, 5 * size(result_value))" in bridge_source + assert "character(kind=c_char, len=4), dimension(2) :: labels_value" in bridge_source + assert "labels_copy = transfer(labels_value, labels_copy, 4 * size(labels_value))" in bridge_source + assert max(map(len, bridge_source.splitlines())) <= 132 + + +def test_fixed_width_character_array_result_itemsize_edit_fails_before_lowering(): + plan = _character_array_result_plan() + result = plan.namespaces[0].functions[0].results[0] + result.array.itemsize = None + + with pytest.raises(ValueError, match="invalid-array-result-itemsize"): + WrapperCodeGenerator().generate(plan) diff --git a/x2py/codegen/bindings/c_to_python.py b/x2py/codegen/bindings/c_to_python.py index 2d473f51f..0a00b3380 100644 --- a/x2py/codegen/bindings/c_to_python.py +++ b/x2py/codegen/bindings/c_to_python.py @@ -346,22 +346,18 @@ class CPythonBindingGenerator(BindingGenerator): _RESULT_POLICY_DISPATCHER = PolicyActionDispatcher( { (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT): "_convert_policy_scalar_result", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_policy_scalar_result", (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_policy_scalar_result", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_policy_scalar_result", (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_convert_snapshot_policy_scalar_result", (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_convert_policy_scalar_result", (ObjectKind.STRING, CodegenAction.COPY_OUT): "_convert_policy_string_result", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT): "_convert_policy_string_result", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_policy_string_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_convert_policy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_convert_policy_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_policy_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_convert_policy_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_convert_policy_array_result", (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_convert_policy_custom_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT): "_convert_policy_custom_result", (ObjectKind.DERIVED_TYPE, CodegenAction.SNAPSHOT_COPY): "_convert_snapshot_policy_custom_result", (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_convert_policy_custom_result", } @@ -374,25 +370,22 @@ class CPythonBindingGenerator(BindingGenerator): _RESULT_DETAIL_DISPATCHER = PolicyActionDispatcher( { (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_default_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT): "_default_result_detail_lines", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_detail_lines", (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_default_result_detail_lines", (ObjectKind.STRING, CodegenAction.COPY_OUT): "_default_result_detail_lines", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT): "_default_result_detail_lines", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_detail_lines", (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_default_result_detail_lines", + (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_default_result_detail_lines", (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_default_result_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT): "_default_result_detail_lines", (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", (ObjectKind.DERIVED_TYPE, CodegenAction.SNAPSHOT_COPY): "_default_result_detail_lines", @@ -402,25 +395,22 @@ class CPythonBindingGenerator(BindingGenerator): _RESULT_NOTE_DISPATCHER = PolicyActionDispatcher( { (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_empty_result_notes", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT): "_empty_result_notes", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_empty_result_notes", (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_notes", (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_empty_result_notes", (ObjectKind.STRING, CodegenAction.COPY_OUT): "_empty_result_notes", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT): "_empty_result_notes", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_empty_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_copy_return_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_empty_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_copy_return_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_copy_return_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_notes", (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_borrowed_view_result_notes", + (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_copy_return_result_notes", (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_empty_result_notes", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT): "_empty_result_notes", (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", (ObjectKind.DERIVED_TYPE, CodegenAction.SNAPSHOT_COPY): "_empty_result_notes", @@ -457,7 +447,7 @@ class CPythonBindingGenerator(BindingGenerator): (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True): ( "_bind_projected_native_array_handle_result" ), - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT, True): ( + (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE, True): ( "_bind_materialized_native_array_handle_result" ), (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT, False): ("_bind_materialized_native_array_handle_result"), @@ -486,14 +476,14 @@ class CPythonBindingGenerator(BindingGenerator): (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT, True): "_project_native_argument_return", + (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE, True): "_project_native_argument_return", (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT, True): "_project_native_argument_return", + (ObjectKind.STRING, CodegenAction.COPY_OUT, True): "_project_native_argument_return", (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, True): "_project_visible_argument_return", (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", @@ -501,13 +491,14 @@ class CPythonBindingGenerator(BindingGenerator): (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT, True): "_project_native_argument_return", + (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT, True): "_project_native_argument_return", + (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE, True): "_project_native_argument_return", (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT, True): "_project_native_argument_return", + (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE, True): "_project_native_argument_return", } ) _PROJECTED_ARGUMENT_OBJECT_DISPATCHER = PolicyProjectionDispatcher( diff --git a/x2py/codegen/bridges/fortran_to_c.py b/x2py/codegen/bridges/fortran_to_c.py index 2c894b685..9252f173e 100644 --- a/x2py/codegen/bridges/fortran_to_c.py +++ b/x2py/codegen/bridges/fortran_to_c.py @@ -152,29 +152,25 @@ class FortranToCBridgeGenerator(BridgeGenerator): NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: "_convert_native_call_local_address_argument", NativeBarrierAction.PASS_STORAGE_ADDRESS: "_convert_native_storage_address_argument", NativeBarrierAction.PASS_RAW_ADDRESS: "_convert_native_raw_address_argument", - NativeBarrierAction.PASS_ARRAY_DESCRIPTOR: "_convert_native_array_descriptor_argument", + NativeBarrierAction.PASS_ARRAY_BUFFER: "_convert_native_array_buffer_argument", NativeBarrierAction.PASS_WRAPPER_ADDRESS: "_convert_native_wrapper_address_argument", } ) _RESULT_POLICY_DISPATCHER = PolicyActionDispatcher( { (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_scalar_result", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT): "_convert_scalar_result", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_scalar_result", (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_scalar_result", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_scalar_result", (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_convert_snapshot_scalar_result", (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_convert_scalar_result", (ObjectKind.STRING, CodegenAction.COPY_OUT): "_convert_string_result", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT): "_convert_string_result", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_string_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_convert_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_convert_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_convert_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_convert_array_result", (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_convert_owned_custom_type_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT): "_convert_owned_custom_type_result", (ObjectKind.DERIVED_TYPE, CodegenAction.SNAPSHOT_COPY): "_convert_owned_custom_type_result", (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_convert_borrowed_custom_type_result", } @@ -186,21 +182,17 @@ class FortranToCBridgeGenerator(BridgeGenerator): (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.SCALAR, CodegenAction.HIDDEN_OUTPUT): "_convert_hidden_function_argument", (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.STRING, CodegenAction.HIDDEN_OUTPUT): "_convert_hidden_function_argument", (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_convert_hidden_function_argument", (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.HIDDEN_OUTPUT): "_convert_hidden_function_argument", } ) _REPLACEMENT_RESULT_DISPATCHER = PolicyActionDispatcher( @@ -216,13 +208,12 @@ class FortranToCBridgeGenerator(BridgeGenerator): (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_build_borrowed_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_build_copy_return_array_result", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_build_copy_return_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_build_copy_return_array_result", } ) _ALLOCATABLE_RESULT_HELPER_DISPATCHER = PolicyActionDispatcher( { (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_uses_heap_allocatable_result_helper", - (ObjectKind.NUMPY_ARRAY, CodegenAction.HIDDEN_OUTPUT): "_skips_allocatable_result_helper", + (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_skips_allocatable_result_helper", (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_skips_allocatable_result_helper", (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_skips_allocatable_result_helper", (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_skips_allocatable_result_helper", @@ -606,6 +597,9 @@ def _convert_function_argument(self, argument, function): """Convert one function argument and its optional projected result.""" if isinstance(argument.var, FunctionAddress): return self._convert_argument(argument, function), None + decision = ownership_decision_for_codegen_variable(argument.var) + if decision.projects_result and not decision.python_visible: + return self._convert_hidden_function_argument(argument.var, decision, argument, function) return self._FUNCTION_ARGUMENT_POLICY_DISPATCHER.dispatch( self, argument.var, @@ -1361,7 +1355,7 @@ def _native_array_descriptor_output_argument(self, subject, policy): DestructionPolicy.CALLER, storage_mode=StorageMode.ALIAS, boundary_storage_mode=StorageMode.ALIAS, - codegen_action=CodegenAction.HIDDEN_OUTPUT, + codegen_action=CodegenAction.IDENTITY_OUTPUT, mutates_native=True, reason="generated pointer descriptor-view operation writes a caller-established C descriptor", ) @@ -3103,8 +3097,8 @@ def _convert_native_wrapper_address_argument(self, var, decision, func): self.scope.insert_variable(f_arg) return {"c_arg": BindCVariable(new_var, var), "f_arg": f_arg, "body": body} - def _convert_native_array_descriptor_argument(self, var, decision, func): - """Pass a packed array descriptor through the native call boundary.""" + def _convert_native_array_buffer_argument(self, var, decision, func): + """Pass ordinary array-buffer fields through the native boundary.""" if decision.codegen_action is CodegenAction.COPY_IN_OUT: return self._convert_native_array_replacement_argument(var, decision, func) return self._convert_native_array_storage_argument(var, decision, func) diff --git a/x2py/codegen/printers/fcode.py b/x2py/codegen/printers/fcode.py index 638f358b3..5ec799247 100644 --- a/x2py/codegen/printers/fcode.py +++ b/x2py/codegen/printers/fcode.py @@ -76,7 +76,6 @@ CodegenAction.CALL_LOCAL_INPUT: "read", CodegenAction.IN_PLACE_ARGUMENT: "readwrite", CodegenAction.IDENTITY_OUTPUT: "write", - CodegenAction.HIDDEN_OUTPUT: "write", CodegenAction.COPY_IN_OUT: "readwrite", CodegenAction.COPY_OUT: "write", CodegenAction.SNAPSHOT_COPY: "read", diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 646f2ad09..eaeb8da37 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -78,6 +78,7 @@ } _WRAPPER_PLAN_COMPLETED_LANES = frozenset( { + # Scalar rollout lanes. "scalar-inputs", "scalar-storage-inputs", "scalar-raw-address-inputs", @@ -88,6 +89,19 @@ "scalar-descriptor-inputs", "scalar-writebacks", "scalar-module-variables", + # String rollout lanes. + "string-value-inputs", + "string-storage-inputs", + "string-raw-address-inputs", + "string-optional-inputs", + "string-writebacks", + "fixed-string-direct-results", + "fixed-string-hidden-outputs", + # Ordinary-array output-only rollout lanes. Array actual arguments stay + # gated until Phase 7 can preserve the native-handle caller contract. + "array-direct-results", + "array-hidden-outputs", + # Cross-cutting rollout lanes. "void-calls", "python-namespaces", "native-call-runtime", @@ -118,6 +132,32 @@ "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" "test_multiple_scalar_results_match_both_routes_without_array_blockers", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_fixed_string_results_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_edge_cases.py::" + "test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" + "test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/scalars/test_verified_baseline.py::" + "test_required_array_buffers_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::" + "test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_array_results.py::" + "test_ordinary_array_results_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::" + "test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" + "test_optional_array_buffers_match_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/function_calls/test_output_arguments.py::" + "test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes", + "tests/wrapper/fortran/strings/test_character_arguments.py::" + "test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", ) diff --git a/x2py/semantics/ownership.py b/x2py/semantics/ownership.py index a548c096a..9465b54d1 100644 --- a/x2py/semantics/ownership.py +++ b/x2py/semantics/ownership.py @@ -90,7 +90,6 @@ class CodegenAction(str, Enum): CALL_LOCAL_INPUT = "call_local_input" IN_PLACE_ARGUMENT = "in_place_argument" IDENTITY_OUTPUT = "identity_output" - HIDDEN_OUTPUT = "hidden_output" COPY_IN_OUT = "copy_in_out" COPY_OUT = "copy_out" SNAPSHOT_COPY = "snapshot_copy" @@ -116,7 +115,8 @@ class NativeBarrierAction(str, Enum): PASS_CALL_LOCAL_ADDRESS = "pass_call_local_address" PASS_STORAGE_ADDRESS = "pass_storage_address" PASS_RAW_ADDRESS = "pass_raw_address" - PASS_ARRAY_DESCRIPTOR = "pass_array_descriptor" + PASS_ARRAY_BUFFER = "pass_array_buffer" + PASS_NATIVE_DESCRIPTOR = "pass_native_descriptor" PASS_WRAPPER_ADDRESS = "pass_wrapper_address" NONE = "none" BLOCKED = "blocked" @@ -788,6 +788,15 @@ def _address_projection_scalar_decision(facts: _StorageFacts, context: Ownership mutates_native=True, reason="address-projected scalar value uses mutable native storage and a replacement return", ) + if context.writes_argument and context.projects_result and not context.python_visible: + return OwnershipDecision( + ObjectKind.SCALAR, + OwnershipOwner.PYTHON, + TransferMode.BY_VALUE, + DestructionPolicy.PYTHON_REFCOUNT, + mutates_native=True, + reason="address-projected hidden scalar output is returned as a Python value", + ) return OwnershipDecision( ObjectKind.SCALAR, OwnershipOwner.CALLER, @@ -903,6 +912,15 @@ def _function_scalar_descriptor_decision( ) def _string_decision(self, facts: _StorageFacts, context: OwnershipContext) -> OwnershipDecision: + if facts.address_role == ADDRESS_ROLE_RAW: + return OwnershipDecision( + ObjectKind.STRING, + OwnershipOwner.CALLER, + TransferMode.IN_PLACE, + DestructionPolicy.CALLER, + mutates_native=True, + reason="raw string address aliases caller-owned fixed-width storage", + ) if facts.scalar_storage: if context.is_result: return OwnershipDecision( @@ -999,6 +1017,15 @@ def _array_decision(self, facts: _StorageFacts, context: OwnershipContext) -> Ow DestructionPolicy.PYTHON_REFCOUNT, reason="array result is returned as Python-owned NumPy storage", ) + if context.writes_argument and context.projects_result and not context.python_visible: + return OwnershipDecision( + ObjectKind.NUMPY_ARRAY, + OwnershipOwner.PYTHON, + TransferMode.COPY_RETURN, + DestructionPolicy.PYTHON_REFCOUNT, + mutates_native=True, + reason="hidden array output is copied into Python-owned NumPy storage", + ) if context.writes_argument: return OwnershipDecision( ObjectKind.NUMPY_ARRAY, @@ -1683,7 +1710,7 @@ def _codegen_action(decision: OwnershipDecision, context: OwnershipContext) -> C return CodegenAction.IDENTITY_OUTPUT if context.python_visible and decision.transfer is TransferMode.IN_PLACE: return CodegenAction.IDENTITY_OUTPUT - return CodegenAction.HIDDEN_OUTPUT + return _CODEGEN_ACTION_BY_TRANSFER[decision.transfer] if ( context.is_argument and context.writes_argument @@ -1740,7 +1767,9 @@ def _native_barrier_action( if OwnershipPolicyResolver._passes_scalar_alias_address(decision, facts): return NativeBarrierAction.PASS_STORAGE_ADDRESS if decision.kind is ObjectKind.NUMPY_ARRAY: - return NativeBarrierAction.PASS_ARRAY_DESCRIPTOR + if decision.descriptor_boundary: + return NativeBarrierAction.PASS_NATIVE_DESCRIPTOR + return NativeBarrierAction.PASS_ARRAY_BUFFER if decision.kind is ObjectKind.STRING: return NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS if decision.kind is ObjectKind.DERIVED_TYPE: diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index 7af88a64f..e8908cb03 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -15,6 +15,7 @@ OwnershipDecision, OwnershipContext, OwnershipOwner, + ObjectKind, SetterAction, TransferMode, default_ownership_policy, @@ -297,11 +298,11 @@ def _native_status_output( def _is_compatible_status_handoff(decision: OwnershipDecision) -> bool: - return bool( - decision.projects_result - and not decision.python_visible - and decision.codegen_action is CodegenAction.HIDDEN_OUTPUT - ) + expected_action = { + ObjectKind.SCALAR: CodegenAction.DIRECT_VALUE, + ObjectKind.STRING: CodegenAction.COPY_OUT, + }.get(decision.kind) + return bool(decision.projects_result and not decision.python_visible and decision.codegen_action is expected_action) def _is_scalar_integer_status(semantic_type_name: str) -> bool: diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index b34269aca..87b2c773d 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -12,12 +12,15 @@ from x2py.semantics.ownership import ( AssignmentMode, CodegenAction, + DestructionPolicy, NativeBarrierAction, ObjectKind, OwnershipDecision, + OwnershipOwner, PythonBarrierAction, SetterAction, StorageMode, + TransferMode, ) @@ -35,6 +38,19 @@ } ) +FIXED_STRING_RESULT_COPY_REASON = "copy fixed-length Fortran character output into C-owned null-terminated storage" +ORDINARY_ARRAY_RESULT_COPY_REASON = "copy non-descriptor Fortran array output into C-owned contiguous storage" +STRING_INPUT_COPY_REASON = "materialize Fortran character storage from the binding UTF-8 byte buffer" +STRING_REPLACEMENT_COPY_REASON = ( + "materialize mutable Fortran character storage and copy post-call bytes back to binding storage" +) +STRING_STORAGE_COPY_REASON = ( + "materialize fixed-length Fortran character storage from caller-owned NumPy bytes and copy mutation back" +) +RAW_STRING_ADDRESS_COPY_REASON = ( + "materialize fixed-length Fortran character storage from a caller-supplied raw address and copy mutation back" +) + class OptionalMode(str, Enum): """Completed ABI behavior for one argument's presence states.""" @@ -45,11 +61,13 @@ class OptionalMode(str, Enum): class ArgumentHandoffMode(str, Enum): - """Completed binding-to-bridge ABI shape for one scalar argument.""" + """Completed binding-to-bridge ABI shape for one argument.""" VALUE = "value" TYPED_REFERENCE = "typed_reference" OPAQUE_ADDRESS = "opaque_address" + CHARACTER_BUFFER = "character_buffer" + ARRAY_BUFFER = "array_buffer" class BridgeDataAction(str, Enum): @@ -141,6 +159,21 @@ class LifecyclePolicy: codegen_action: CodegenAction semantic_type_name: str result_position: int + object_kind: ObjectKind + + +@dataclass(frozen=True) +class ArrayHandoffPolicy: + """Completed ordinary-array storage and layout facts.""" + + rank: int | None + shape: tuple[str, ...] + axes: tuple[str, ...] + order: str | None + contiguous: bool | None + itemsize: int | None = None + category: str | None = None + extent_references: tuple[tuple[str, ...], ...] = () @dataclass(frozen=True) @@ -172,6 +205,8 @@ class ArgumentPolicy: projects_result: bool python_visible: bool result_position: int | None + character_length: int | None + array: ArrayHandoffPolicy | None = None @dataclass(frozen=True) @@ -189,6 +224,8 @@ class ResultPolicy: boundary_storage_mode: StorageMode bridge_data_action: BridgeDataAction bridge_copy_reason: str | None + character_length: int | None = None + array: ArrayHandoffPolicy | None = None source_kind: str = "direct_return" native_name: str | None = None native_position: int | None = None @@ -197,7 +234,11 @@ class ResultPolicy: @dataclass(frozen=True) class NativeCallSlotPolicy: - """Completed native-call slot consumed by wrapper planning.""" + """Completed native-call slot consumed by wrapper planning. + + ``object_kind`` is copied from the owning transfer decision. Literal slots + have no transfer owner and therefore use ``None``. + """ owner_path: str native_position: int @@ -210,11 +251,13 @@ class NativeCallSlotPolicy: codegen_action: CodegenAction bridge_data_action: BridgeDataAction bridge_copy_reason: str | None + object_kind: ObjectKind | None literal_type: str | None = None literal_value: Any = None result_position: int | None = None semantic_type_name: str | None = None character_length: int | None = None + array: ArrayHandoffPolicy | None = None @dataclass(frozen=True) @@ -330,7 +373,10 @@ def build_function_wrapper_policy( + result_blockers + slot_blockers + lifecycle_blockers + + _array_extent_reference_blockers(function, arguments, results) + _runtime_status_plan_blockers(status_error) + + _string_result_status_blockers(results, status_error) + + _string_writeback_status_blockers(arguments, status_error) ) return FunctionWrapperPolicy( owner_path=owner_path, @@ -417,6 +463,8 @@ def _argument_policies( projects_result=decision.projects_result, python_visible=decision.python_visible, result_position=_argument_result_position(function, current_python_position), + character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), ) ) return policies, tuple(blockers) @@ -433,7 +481,11 @@ def _result_policies( if function.return_type is None: projected_arguments = _visible_projected_arguments(function) if hidden_results and not projected_arguments: - return hidden_results, (*hidden_blockers, *_result_position_blockers(hidden_results)) + return hidden_results, ( + *hidden_blockers, + *_result_position_blockers(hidden_results), + *_string_result_aggregation_blockers(hidden_results), + ) if projected_arguments and not hidden_results: return (), hidden_blockers if not hidden_results and not projected_arguments: @@ -459,11 +511,18 @@ def _result_policies( boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, bridge_data_action=bridge_data_action, bridge_copy_reason=bridge_copy_reason, + character_length=_character_length(function.return_type), + array=_array_handoff_policy(function.return_type), ) results = (direct_result, *hidden_results) return ( results, - (*blockers, *hidden_blockers, *_result_position_blockers(results)), + ( + *blockers, + *hidden_blockers, + *_result_position_blockers(results), + *_string_result_aggregation_blockers(results), + ), ) @@ -509,6 +568,8 @@ def _hidden_result_policies( boundary_storage_mode=decision.boundary_storage_mode or decision.storage_mode, bridge_data_action=bridge_data_action, bridge_copy_reason=bridge_copy_reason, + character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), source_kind="hidden_output", native_name=mapping.native_name or argument.name, native_position=mapping.native_position, @@ -604,9 +665,11 @@ def _projected_native_call_slot_policies( codegen_action=decision.codegen_action, bridge_data_action=bridge_data_action, bridge_copy_reason=bridge_copy_reason, + object_kind=decision.kind, result_position=mapping.result_position, semantic_type_name=argument.semantic_type.name, character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), ) ) blockers.extend(_native_position_blockers(slot.native_position for slot in slots)) @@ -635,6 +698,7 @@ def _hidden_result_native_call_slot_policy( codegen_action=CodegenAction.BLOCKED, bridge_data_action=BridgeDataAction.BLOCKED, bridge_copy_reason=None, + object_kind=None, result_position=mapping.result_position, ), (f"native-call result slot {native_position} has no hidden argument {mapping.python_name!r}",), @@ -654,9 +718,11 @@ def _hidden_result_native_call_slot_policy( codegen_action=CodegenAction.BLOCKED, bridge_data_action=BridgeDataAction.BLOCKED, bridge_copy_reason=None, + object_kind=None, result_position=mapping.result_position, semantic_type_name=argument.semantic_type.name, character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), ), (f"native-call result slot {native_position} references argument without completed policy",), ) @@ -679,9 +745,11 @@ def _hidden_result_native_call_slot_policy( codegen_action=decision.codegen_action, bridge_data_action=bridge_data_action, bridge_copy_reason=bridge_copy_reason, + object_kind=decision.kind, result_position=mapping.result_position, semantic_type_name=argument.semantic_type.name, character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), ), blockers, ) @@ -707,6 +775,7 @@ def _literal_native_call_slot_policy( codegen_action=CodegenAction.DIRECT_VALUE, bridge_data_action=BridgeDataAction.DIRECT_TRANSFER, bridge_copy_reason=None, + object_kind=None, literal_type=literal_type, literal_value=literal_value, semantic_type_name=literal_type, @@ -768,7 +837,10 @@ def _implicit_native_call_slot_policies( codegen_action=decision.codegen_action, bridge_data_action=bridge_data_action, bridge_copy_reason=bridge_copy_reason, + object_kind=decision.kind, semantic_type_name=argument.semantic_type.name, + character_length=_character_length(argument.semantic_type), + array=_array_handoff_policy(argument.semantic_type), ) ) return positions, tuple(slots), tuple(blockers) @@ -794,17 +866,41 @@ def _argument_shape_blockers( decision: OwnershipDecision, ) -> tuple[str, ...]: """Return ownership, type, and visibility blockers for one argument.""" + if int(argument.semantic_type.rank or 0) > 0: + return _array_argument_shape_blockers(argument, decision) blockers: list[str] = [] if decision.is_blocked: blockers.append( f"argument {argument.name!r} has blocked ownership policy: {decision.blocker or decision.reason}" ) - if not _is_first_lane_scalar_type(argument.semantic_type): + string_value = _is_plan_string_value_type(argument.semantic_type) + if not (_is_first_lane_scalar_type(argument.semantic_type) or string_value): blockers.append(f"argument {argument.name!r} is not a first-lane primitive scalar") if not decision.python_visible: blockers.append(f"argument {argument.name!r} is not Python-visible") - if decision.kind is not ObjectKind.SCALAR: - blockers.append(f"argument {argument.name!r} policy kind is {decision.kind.value}, not scalar") + expected_kind = ObjectKind.STRING if string_value else ObjectKind.SCALAR + if decision.kind is not expected_kind: + blockers.append(f"argument {argument.name!r} policy kind is {decision.kind.value}, not {expected_kind.value}") + return tuple(blockers) + + +# Ordinary-array argument policy. +def _array_argument_shape_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require one supported non-descriptor ordinary array value.""" + blockers: list[str] = [] + if decision.is_blocked: + blockers.append( + f"argument {argument.name!r} has blocked ownership policy: {decision.blocker or decision.reason}" + ) + if not _is_phase6_ordinary_array_type(argument.semantic_type): + blockers.append(f"argument {argument.name!r} is outside ordinary array buffer support") + if not decision.python_visible: + blockers.append(f"argument {argument.name!r} is not Python-visible") + if decision.kind is not ObjectKind.NUMPY_ARRAY: + blockers.append(f"argument {argument.name!r} policy kind is {decision.kind.value}, not numpy_array") return tuple(blockers) @@ -814,6 +910,10 @@ def _argument_boundary_blockers( ) -> tuple[str, ...]: """Return Python/native boundary-action blockers for one argument.""" blockers: list[str] = [] + if decision.kind is ObjectKind.STRING: + return _string_boundary_blockers(argument, decision) + if decision.kind is ObjectKind.NUMPY_ARRAY: + return _array_boundary_blockers(argument, decision) if decision.python_barrier_action not in { PythonBarrierAction.SCALAR_VALUE, PythonBarrierAction.SCALAR_STORAGE, @@ -847,6 +947,207 @@ def _argument_boundary_blockers( return tuple(blockers) +# Ordinary-array boundary policy. +def _array_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require one caller-owned ordinary NumPy buffer handoff.""" + blockers = [] + if decision.owner is not OwnershipOwner.CALLER: + blockers.append(f"argument {argument.name!r} array owner is {decision.owner.value}, not caller") + expected_transfer = TransferMode.IN_PLACE if decision.mutates_native else TransferMode.CALL_LOCAL + if decision.transfer is not expected_transfer: + blockers.append( + f"argument {argument.name!r} array transfer is {decision.transfer.value}, not {expected_transfer.value}" + ) + expected_destruction = DestructionPolicy.CALLER if decision.mutates_native else DestructionPolicy.NONE + if decision.destruction is not expected_destruction: + blockers.append( + f"argument {argument.name!r} array destruction is {decision.destruction.value}, " + f"not {expected_destruction.value}" + ) + if decision.storage_mode is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} array storage is {decision.storage_mode.value}, not stack") + if (decision.boundary_storage_mode or decision.storage_mode) is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} array boundary storage is not stack") + if decision.python_barrier_action is not PythonBarrierAction.ARRAY_STORAGE: + blockers.append( + f"argument {argument.name!r} array Python action is " + f"{decision.python_barrier_action.value}, not array_storage" + ) + if decision.native_barrier_action is not NativeBarrierAction.PASS_ARRAY_BUFFER: + blockers.append( + f"argument {argument.name!r} array native action is " + f"{decision.native_barrier_action.value}, not pass_array_buffer" + ) + expected_actions = { + CodegenAction.CALL_LOCAL_INPUT, + CodegenAction.IN_PLACE_ARGUMENT, + CodegenAction.IDENTITY_OUTPUT, + } + if decision.codegen_action not in expected_actions: + blockers.append( + f"argument {argument.name!r} array action is {decision.codegen_action.value}, not a borrowed buffer action" + ) + if decision.nullable or decision.descriptor_boundary: + blockers.append(f"argument {argument.name!r} ordinary array must be non-descriptor storage") + array_policy = _array_handoff_policy(argument.semantic_type) + if argument.optional and array_policy is not None and array_policy.rank is None: + blockers.append(f"argument {argument.name!r} optional assumed-rank combination is not supported") + return tuple(blockers) + + +# String argument policy. +def _string_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Dispatch one completed string boundary without backend inference.""" + action = decision.python_barrier_action + if action is PythonBarrierAction.STRING_VALUE: + return _string_value_boundary_blockers(argument, decision) + if action is PythonBarrierAction.STRING_STORAGE: + return _string_storage_boundary_blockers(argument, decision) + if action is PythonBarrierAction.RAW_ADDRESS: + return _raw_string_address_boundary_blockers(argument, decision) + return (f"argument {argument.name!r} has unsupported string Python action {action.value}",) + + +def _string_value_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Return completed string-value input or replacement blockers.""" + blockers = [] + if decision.python_barrier_action is not PythonBarrierAction.STRING_VALUE: + blockers.append( + f"argument {argument.name!r} has unsupported string Python action {decision.python_barrier_action.value}" + ) + if decision.native_barrier_action is not NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: + blockers.append( + f"argument {argument.name!r} native action is {decision.native_barrier_action.value}, " + "not a string call-local address handoff" + ) + if decision.codegen_action not in {CodegenAction.CALL_LOCAL_INPUT, CodegenAction.COPY_IN_OUT}: + blockers.append( + f"argument {argument.name!r} string action is {decision.codegen_action.value}, " + "not a call-local input or copy-in/out replacement" + ) + if decision.codegen_action is CodegenAction.CALL_LOCAL_INPUT and decision.projects_result: + blockers.append(f"argument {argument.name!r} call-local string input unexpectedly projects a result") + if decision.codegen_action is CodegenAction.COPY_IN_OUT: + blockers.extend(_string_replacement_blockers(argument, decision)) + return tuple(blockers) + + +def _string_storage_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require caller-owned fixed mutable NumPy bytes storage.""" + blockers = list( + _string_address_ownership_blockers( + argument, + decision, + expected_storage=StorageMode.ALIAS, + label="string storage", + ) + ) + if decision.native_barrier_action is not NativeBarrierAction.PASS_STORAGE_ADDRESS: + blockers.append( + f"argument {argument.name!r} string storage native action is " + f"{decision.native_barrier_action.value}, not pass_storage_address" + ) + return tuple(blockers) + + +def _raw_string_address_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require a caller-owned unsafe fixed string address contract.""" + blockers = list( + _string_address_ownership_blockers( + argument, + decision, + expected_storage=StorageMode.STACK, + label="raw string address", + ) + ) + if decision.native_barrier_action is not NativeBarrierAction.PASS_RAW_ADDRESS: + blockers.append( + f"argument {argument.name!r} raw string native action is " + f"{decision.native_barrier_action.value}, not pass_raw_address" + ) + return tuple(blockers) + + +def _string_address_ownership_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, + *, + expected_storage: StorageMode, + label: str, +) -> tuple[str, ...]: + """Validate ownership shared by fixed storage and raw-address forms.""" + blockers = [] + if _character_length(argument.semantic_type) is None: + blockers.append(f"argument {argument.name!r} {label} requires a fixed positive character length") + if decision.owner is not OwnershipOwner.CALLER: + blockers.append(f"argument {argument.name!r} {label} owner is {decision.owner.value}, not caller") + if decision.transfer is not TransferMode.IN_PLACE: + blockers.append(f"argument {argument.name!r} {label} transfer is {decision.transfer.value}, not in_place") + if decision.destruction is not DestructionPolicy.CALLER: + blockers.append(f"argument {argument.name!r} {label} destruction is {decision.destruction.value}, not caller") + if decision.storage_mode is not expected_storage: + blockers.append( + f"argument {argument.name!r} {label} storage is {decision.storage_mode.value}, not {expected_storage.value}" + ) + if (decision.boundary_storage_mode or decision.storage_mode) is not expected_storage: + blockers.append(f"argument {argument.name!r} {label} boundary storage is not {expected_storage.value}") + if decision.codegen_action is not CodegenAction.IN_PLACE_ARGUMENT: + blockers.append( + f"argument {argument.name!r} {label} action is {decision.codegen_action.value}, not in_place_argument" + ) + if not decision.mutates_native: + blockers.append(f"argument {argument.name!r} {label} does not record native mutation") + if decision.projects_result: + blockers.append(f"argument {argument.name!r} {label} unexpectedly projects a result") + if decision.nullable or argument.optional: + blockers.append(f"argument {argument.name!r} optional {label} is unsupported") + return tuple(blockers) + + +def _string_replacement_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require completed Phase 5C replacement ownership and projection.""" + blockers = [] + if decision.owner is not OwnershipOwner.PYTHON: + blockers.append(f"argument {argument.name!r} replacement owner is {decision.owner.value}, not python") + if decision.transfer is not TransferMode.COPY_RETURN: + blockers.append( + f"argument {argument.name!r} replacement transfer is {decision.transfer.value}, not copy_return" + ) + if decision.destruction is not DestructionPolicy.PYTHON_REFCOUNT: + blockers.append( + f"argument {argument.name!r} replacement destruction is {decision.destruction.value}, not python_refcount" + ) + if decision.storage_mode is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} replacement storage is {decision.storage_mode.value}, not stack") + if (decision.boundary_storage_mode or decision.storage_mode) is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} replacement boundary storage is not stack") + if not decision.mutates_native: + blockers.append(f"argument {argument.name!r} replacement does not record native mutation") + if not decision.projects_result: + blockers.append(f"argument {argument.name!r} replacement does not project a Python result") + if decision.nullable: + blockers.append(f"argument {argument.name!r} replacement uses semantic nullability for optional presence") + return tuple(blockers) + + def _argument_bridge_data_blockers( argument: models.SemanticArgument, bridge_data_action: BridgeDataAction, @@ -879,6 +1180,10 @@ def _argument_projection_blockers( def _result_blockers(semantic_type: models.SemanticType, decision: OwnershipDecision) -> tuple[str, ...]: + if _is_phase6_ordinary_array_type(semantic_type): + return _ordinary_array_result_blockers(semantic_type, decision, "result") + if _is_fixed_plan_string_result_type(semantic_type): + return _fixed_string_result_blockers(decision) blockers: list[str] = [] if decision.is_blocked: blockers.append(f"result has blocked ownership policy: {decision.blocker or decision.reason}") @@ -900,7 +1205,11 @@ def _hidden_result_blockers( decision: OwnershipDecision, mapping: models.ProjectionMapping, ) -> tuple[str, ...]: - """Return blockers for one hidden primitive scalar result projection.""" + """Return blockers for one hidden result projection.""" + if _is_phase6_ordinary_array_type(argument.semantic_type): + return _ordinary_array_hidden_result_blockers(argument, decision, mapping) + if _is_fixed_plan_string_result_type(argument.semantic_type): + return _fixed_string_hidden_result_blockers(argument, decision, mapping) blockers: list[str] = [] if decision.is_blocked: blockers.append(f"hidden result {argument.name!r} has blocked ownership policy: {decision.blocker}") @@ -908,9 +1217,9 @@ def _hidden_result_blockers( blockers.append(f"hidden result {argument.name!r} is not a primitive scalar") if decision.kind is not ObjectKind.SCALAR: blockers.append(f"hidden result {argument.name!r} policy kind is {decision.kind.value}, not scalar") - if decision.codegen_action is not CodegenAction.HIDDEN_OUTPUT: + if decision.codegen_action is not CodegenAction.DIRECT_VALUE: blockers.append( - f"hidden result {argument.name!r} codegen action is {decision.codegen_action.value}, not hidden_output" + f"hidden result {argument.name!r} codegen action is {decision.codegen_action.value}, not direct_value" ) if decision.python_barrier_action is not PythonBarrierAction.NONE: blockers.append( @@ -936,6 +1245,172 @@ def _hidden_result_blockers( return tuple(blockers) +# Ordinary-array result policy. +def _ordinary_array_result_blockers( + semantic_type: models.SemanticType, + decision: OwnershipDecision, + label: str, +) -> tuple[str, ...]: + """Require one Python-owned fixed-shape ordinary array copy result.""" + blockers = [] + if decision.is_blocked: + blockers.append(f"{label} has blocked ownership policy: {decision.blocker or decision.reason}") + if decision.kind is not ObjectKind.NUMPY_ARRAY: + blockers.append(f"{label} policy kind is {decision.kind.value}, not numpy_array") + if decision.owner is not OwnershipOwner.PYTHON: + blockers.append(f"{label} owner is {decision.owner.value}, not python") + if decision.transfer is not TransferMode.COPY_RETURN: + blockers.append(f"{label} transfer is {decision.transfer.value}, not copy_return") + if decision.destruction is not DestructionPolicy.PYTHON_REFCOUNT: + blockers.append(f"{label} destruction is {decision.destruction.value}, not python_refcount") + if decision.storage_mode is not StorageMode.STACK: + blockers.append(f"{label} storage is {decision.storage_mode.value}, not stack") + if decision.codegen_action is not CodegenAction.COPY_OUT: + blockers.append(f"{label} action is {decision.codegen_action.value}, not copy_out") + if decision.python_barrier_action is not PythonBarrierAction.NONE: + blockers.append(f"{label} Python action is {decision.python_barrier_action.value}, not none") + if decision.native_barrier_action is not NativeBarrierAction.NONE: + blockers.append(f"{label} native action is {decision.native_barrier_action.value}, not none") + if decision.nullable or decision.descriptor_boundary: + blockers.append(f"{label} is descriptor-backed or nullable") + array = _array_handoff_policy(semantic_type) + if array is None or array.rank is None or any(shape in {":", "::Strided", "...", "Flat"} for shape in array.shape): + blockers.append(f"{label} ordinary array shape is not fully expressible") + elif array.order == "ORDER_C" and array.rank > 1: + blockers.append(f"{label} ordinary array copy requires Fortran element order") + return tuple(blockers) + + +def _ordinary_array_hidden_result_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, + mapping: models.ProjectionMapping, +) -> tuple[str, ...]: + """Require one hidden fixed-shape array copied through bridge-owned storage.""" + label = f"hidden result {argument.name!r}" + blockers = list(_ordinary_array_result_blockers(argument.semantic_type, decision, label)) + blockers = [item for item in blockers if " native action is " not in item] + if decision.native_barrier_action is not NativeBarrierAction.PASS_ARRAY_BUFFER: + blockers.append(f"{label} native action is {decision.native_barrier_action.value}, not array buffer") + if decision.python_visible or not decision.projects_result: + blockers.append(f"{label} projection visibility is inconsistent") + if not isinstance(mapping.native_position, int): + blockers.append(f"{label} is missing a native position") + if not isinstance(mapping.result_position, int) or isinstance(mapping.result_position, bool): + blockers.append(f"{label} has no integer result position") + elif mapping.result_position < 0: + blockers.append(f"{label} has negative result position {mapping.result_position}") + return tuple(blockers) + + +# String result and writeback policy. +def _fixed_string_result_blockers(decision: OwnershipDecision) -> tuple[str, ...]: + """Require the completed copy-return policy for one direct fixed string.""" + blockers = list(_fixed_string_result_ownership_blockers(decision, "result")) + if decision.is_blocked: + blockers.append(f"result has blocked ownership policy: {decision.blocker or decision.reason}") + if decision.kind is not ObjectKind.STRING: + blockers.append(f"result policy kind is {decision.kind.value}, not string") + if decision.codegen_action is not CodegenAction.COPY_OUT: + blockers.append(f"result codegen action is {decision.codegen_action.value}, not copy_out") + if decision.python_barrier_action is not PythonBarrierAction.NONE: + blockers.append(f"result Python action is {decision.python_barrier_action.value}, not none") + if decision.native_barrier_action is not NativeBarrierAction.NONE: + blockers.append(f"result native action is {decision.native_barrier_action.value}, not none") + return tuple(blockers) + + +def _fixed_string_hidden_result_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, + mapping: models.ProjectionMapping, +) -> tuple[str, ...]: + """Require one fixed string hidden output and its completed projection.""" + label = f"hidden result {argument.name!r}" + blockers = list(_fixed_string_result_ownership_blockers(decision, label)) + if decision.is_blocked: + blockers.append( + f"hidden result {argument.name!r} has blocked ownership policy: {decision.blocker or decision.reason}" + ) + if decision.kind is not ObjectKind.STRING: + blockers.append(f"hidden result {argument.name!r} policy kind is {decision.kind.value}, not string") + if decision.codegen_action is not CodegenAction.COPY_OUT: + blockers.append( + f"hidden result {argument.name!r} codegen action is {decision.codegen_action.value}, not copy_out" + ) + if decision.python_barrier_action is not PythonBarrierAction.NONE: + blockers.append( + f"hidden result {argument.name!r} Python action is {decision.python_barrier_action.value}, not none" + ) + if decision.native_barrier_action is not NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: + blockers.append( + f"hidden result {argument.name!r} native action is " + f"{decision.native_barrier_action.value}, not call-local address" + ) + if decision.python_visible: + blockers.append(f"hidden result {argument.name!r} is Python-visible") + if not decision.projects_result: + blockers.append(f"hidden result {argument.name!r} does not project a Python result") + if not isinstance(mapping.native_position, int): + blockers.append(f"hidden result {argument.name!r} is missing a native position") + if not isinstance(mapping.result_position, int) or isinstance(mapping.result_position, bool): + blockers.append(f"hidden result {argument.name!r} has no integer result position") + elif mapping.result_position < 0: + blockers.append(f"hidden result {argument.name!r} has negative result position {mapping.result_position}") + return tuple(blockers) + + +def _fixed_string_result_ownership_blockers( + decision: OwnershipDecision, + label: str, +) -> tuple[str, ...]: + """Require Python-owned stack-to-copy-return fixed string ownership.""" + blockers = [] + if decision.owner is not OwnershipOwner.PYTHON: + blockers.append(f"{label} owner is {decision.owner.value}, not python") + if decision.transfer is not TransferMode.COPY_RETURN: + blockers.append(f"{label} transfer is {decision.transfer.value}, not copy_return") + if decision.destruction is not DestructionPolicy.PYTHON_REFCOUNT: + blockers.append(f"{label} destruction is {decision.destruction.value}, not python_refcount") + if decision.storage_mode is not StorageMode.STACK: + blockers.append(f"{label} storage is {decision.storage_mode.value}, not stack") + if (decision.boundary_storage_mode or decision.storage_mode) is not StorageMode.STACK: + blockers.append(f"{label} boundary storage is not stack") + if decision.nullable: + blockers.append(f"{label} is nullable outside deferred string results") + return tuple(blockers) + + +def _string_result_aggregation_blockers(results: tuple[ResultPolicy, ...]) -> tuple[str, ...]: + """Keep native string-allocation cleanup single-result in Phase 5B.""" + if any(result.ownership.kind is ObjectKind.STRING for result in results) and len(results) != 1: + return ("fixed string result lane requires exactly one Python-visible result",) + return () + + +def _string_result_status_blockers( + results: tuple[ResultPolicy, ...], + status_error: NativeStatusErrorPolicy | None, +) -> tuple[str, ...]: + """Block status exits until public string-result release is planned.""" + if status_error is not None and any(result.ownership.kind is ObjectKind.STRING for result in results): + return ("fixed string result with native status error requires planned failure-path release",) + return () + + +def _string_writeback_status_blockers( + arguments: list[ArgumentPolicy], + status_error: NativeStatusErrorPolicy | None, +) -> tuple[str, ...]: + """Block status exits until mutable string-buffer cleanup is planned there.""" + if status_error is not None and any( + argument.ownership.kind is ObjectKind.STRING and argument.codegen_action is CodegenAction.COPY_IN_OUT + for argument in arguments + ): + return ("string replacement with native status error requires planned failure-path cleanup",) + return () + + def _result_position_blockers(results: tuple[ResultPolicy, ...]) -> tuple[str, ...]: """Require completed Python results to cover one contiguous order.""" positions = tuple(result.result_position for result in results) @@ -998,7 +1473,7 @@ def _character_length(semantic_type: models.SemanticType) -> int | None: def _lifecycle_policies( arguments: list[ArgumentPolicy], ) -> tuple[tuple[LifecyclePolicy, ...], tuple[str, ...]]: - """Return completed scalar writeback actions and structural blockers.""" + """Return completed replacement/writeback actions and structural blockers.""" actions = [] blockers: list[str] = [] for argument in arguments: @@ -1007,6 +1482,9 @@ def _lifecycle_policies( if argument.result_position is None: blockers.append(f"argument {argument.name!r} writeback is missing a result position") continue + phases = ( + (WritebackPhase.COPY_OUT,) if argument.ownership.kind is ObjectKind.NUMPY_ARRAY else tuple(WritebackPhase) + ) actions.extend( LifecyclePolicy( owner_path=argument.owner_path, @@ -1015,11 +1493,18 @@ def _lifecycle_policies( codegen_action=argument.codegen_action, semantic_type_name=argument.semantic_type_name, result_position=argument.result_position, + object_kind=argument.ownership.kind, ) - for phase in WritebackPhase + for phase in phases ) - if len(actions) > len(WritebackPhase): - blockers.append("scalar writeback lane currently requires exactly one projected scalar result") + non_array_actions = [ + action + for action in actions + if next(item for item in arguments if item.owner_path == action.owner_path).ownership.kind + is not ObjectKind.NUMPY_ARRAY + ] + if len(non_array_actions) > len(WritebackPhase): + blockers.append("replacement lane currently requires exactly one non-array projected result") return tuple(actions), tuple(blockers) @@ -1044,11 +1529,23 @@ def _is_first_lane_scalar_type(semantic_type: models.SemanticType) -> bool: ) +def _is_plan_string_value_type(semantic_type: models.SemanticType) -> bool: + """Return whether one semantic type is a scalar Python string value.""" + return bool(int(semantic_type.rank or 0) == 0 and semantic_type.name == "String") + + +def _is_fixed_plan_string_result_type(semantic_type: models.SemanticType) -> bool: + """Return whether one result is a fixed positive scalar string.""" + length = _character_length(semantic_type) + return bool(_is_plan_string_value_type(semantic_type) and length is not None and length > 0) + + def _is_first_lane_literal_type(literal_type: str) -> bool: """Return whether a hidden literal type belongs to the scalar input lane.""" return literal_type in {"Bool", "Int32", "Float32", "Float64", "Complex64", "Complex128"} +# Scalar module-variable policy. def _scalar_module_variable_blockers( variable: models.SemanticVariable, getter: OwnershipDecision | None, @@ -1186,6 +1683,45 @@ def _argument_bridge_data_action( value_kind: str | None, ) -> tuple[BridgeDataAction, str | None]: """Complete whether the bridge reuses, views, or copies one input payload.""" + if decision.kind is ObjectKind.NUMPY_ARRAY: + if ( + optional_mode in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE} + and decision.python_barrier_action is PythonBarrierAction.ARRAY_STORAGE + and decision.native_barrier_action is NativeBarrierAction.PASS_ARRAY_BUFFER + and decision.codegen_action + in { + CodegenAction.CALL_LOCAL_INPUT, + CodegenAction.IN_PLACE_ARGUMENT, + CodegenAction.IDENTITY_OUTPUT, + } + ): + return BridgeDataAction.ASSOCIATE_VIEW, None + return BridgeDataAction.BLOCKED, None + if decision.kind is ObjectKind.STRING: + if ( + optional_mode in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE} + and decision.python_barrier_action is PythonBarrierAction.STRING_VALUE + and decision.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + ): + if decision.codegen_action is CodegenAction.CALL_LOCAL_INPUT: + return BridgeDataAction.COPY_REPRESENTATION, STRING_INPUT_COPY_REASON + if decision.codegen_action is CodegenAction.COPY_IN_OUT: + return BridgeDataAction.COPY_REPRESENTATION, STRING_REPLACEMENT_COPY_REASON + if ( + optional_mode is OptionalMode.REQUIRED + and decision.python_barrier_action is PythonBarrierAction.STRING_STORAGE + and decision.native_barrier_action is NativeBarrierAction.PASS_STORAGE_ADDRESS + and decision.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + ): + return BridgeDataAction.COPY_REPRESENTATION, STRING_STORAGE_COPY_REASON + if ( + optional_mode is OptionalMode.REQUIRED + and decision.python_barrier_action is PythonBarrierAction.RAW_ADDRESS + and decision.native_barrier_action is NativeBarrierAction.PASS_RAW_ADDRESS + and decision.codegen_action is CodegenAction.IN_PLACE_ARGUMENT + ): + return BridgeDataAction.COPY_REPRESENTATION, RAW_STRING_ADDRESS_COPY_REASON + return BridgeDataAction.BLOCKED, None if optional_mode is OptionalMode.DESCRIPTOR: if value_kind == "pointer": return BridgeDataAction.ASSOCIATE_VIEW, None @@ -1217,6 +1753,13 @@ def _result_bridge_data_action( """Complete direct result transfer without widening unsupported lanes.""" if _is_first_lane_scalar_type(semantic_type): return BridgeDataAction.DIRECT_TRANSFER, None + if _is_phase6_ordinary_array_type(semantic_type): + return BridgeDataAction.COPY_REPRESENTATION, ORDINARY_ARRAY_RESULT_COPY_REASON + if _is_fixed_plan_string_result_type(semantic_type): + return ( + BridgeDataAction.COPY_REPRESENTATION, + FIXED_STRING_RESULT_COPY_REASON, + ) return BridgeDataAction.BLOCKED, None @@ -1226,16 +1769,27 @@ def _native_result_bridge_data_action( """Complete bridge data movement for one hidden native output slot.""" if _is_first_lane_scalar_type(semantic_type): return BridgeDataAction.DIRECT_TRANSFER, None + if _is_phase6_ordinary_array_type(semantic_type): + return BridgeDataAction.COPY_REPRESENTATION, ORDINARY_ARRAY_RESULT_COPY_REASON if semantic_type.name == "String" and _character_length(semantic_type) is not None: return ( BridgeDataAction.COPY_REPRESENTATION, - "copy fixed-length Fortran character output into C-owned null-terminated storage", + FIXED_STRING_RESULT_COPY_REASON, ) return BridgeDataAction.BLOCKED, None def _argument_handoff_mode(decision: OwnershipDecision) -> ArgumentHandoffMode: - """Return the completed scalar ABI shape consumed by both backends.""" + """Return the completed ABI shape consumed by both backends.""" + if decision.kind is ObjectKind.NUMPY_ARRAY: + return ArgumentHandoffMode.ARRAY_BUFFER + if decision.python_barrier_action in { + PythonBarrierAction.STRING_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: + return ArgumentHandoffMode.OPAQUE_ADDRESS + if decision.kind is ObjectKind.STRING: + return ArgumentHandoffMode.CHARACTER_BUFFER if decision.python_barrier_action in { PythonBarrierAction.SCALAR_STORAGE, PythonBarrierAction.RAW_ADDRESS, @@ -1246,6 +1800,116 @@ def _argument_handoff_mode(decision: OwnershipDecision) -> ArgumentHandoffMode: return ArgumentHandoffMode.TYPED_REFERENCE +# Ordinary-array handoff policy. +def _array_handoff_policy(semantic_type: models.SemanticType) -> ArrayHandoffPolicy | None: + """Copy structured ordinary-array facts into completed wrapper policy.""" + storage = semantic_type.storage + array = storage.array if storage is not None else None + if array is None: + return None + assumed_rank = array.category == "assumed_rank" + rank = None if assumed_rank else int(array.rank or semantic_type.rank or 0) + if rank is not None and rank <= 0: + return None + shape = tuple(str(item) for item in (array.shape or semantic_type.shape)) + axes = tuple(str(item) for item in array.axes) + return ArrayHandoffPolicy( + rank=rank, + shape=shape, + axes=axes, + order="ORDER_F" if assumed_rank and array.order is None else array.order, + contiguous=array.contiguous, + itemsize=_character_length(semantic_type) if semantic_type.name == "String" else None, + category=array.category, + extent_references=tuple(_array_extent_references(item) for item in shape), + ) + + +def _is_phase6_ordinary_array_type(semantic_type: models.SemanticType) -> bool: + """Return whether one type is an ordinary non-descriptor array buffer.""" + array_policy = _array_handoff_policy(semantic_type) + if array_policy is None: + return False + storage = semantic_type.storage + array = storage.array if storage is not None else None + return bool( + array is not None + and ( + semantic_type.name in _PLAN_PRIMITIVE_SCALAR_TYPES + or (semantic_type.name == "String" and array_policy.itemsize is not None) + ) + and (array_policy.rank is None or 1 <= array_policy.rank <= 15) + and (array_policy.rank is None or len(array_policy.shape) == array_policy.rank) + and (array_policy.rank is None or len(array_policy.axes) == array_policy.rank) + and not array.allocatable + and not array.pointer + ) + + +def _array_extent_references(expression: str) -> tuple[str, ...]: + """Return stable scalar names used by one declared extent expression.""" + if expression in {":", "::Strided", "...", "Flat"}: + return () + try: + tree = ast.parse(expression, mode="eval") + except SyntaxError: + return ("",) + if not _valid_array_extent_expression(tree): + return ("",) + return tuple(dict.fromkeys(node.id for node in ast.walk(tree) if isinstance(node, ast.Name))) + + +def _valid_array_extent_expression(tree: ast.AST) -> bool: + """Return whether an extent uses only integer arithmetic and scalar names.""" + allowed = ( + ast.Expression, + ast.BinOp, + ast.UnaryOp, + ast.Name, + ast.Load, + ast.Constant, + ast.Add, + ast.Sub, + ast.Mult, + ast.FloorDiv, + ast.Div, + ast.Mod, + ast.USub, + ast.UAdd, + ) + return all(isinstance(node, allowed) and _is_integer_constant(node) for node in ast.walk(tree)) + + +def _is_integer_constant(node: ast.AST) -> bool: + return not isinstance(node, ast.Constant) or (isinstance(node.value, int) and not isinstance(node.value, bool)) + + +def _array_extent_reference_blockers( + function: models.SemanticFunction, + arguments: list[ArgumentPolicy], + results: tuple[ResultPolicy, ...], +) -> tuple[str, ...]: + """Require every declared extent name to come from a visible scalar argument.""" + scalar_names = { + argument.name + for argument in function.arguments + if int(argument.semantic_type.rank or 0) == 0 + and (decision := _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA)) is not None + and decision.python_visible + } + blockers = [] + for owner in (*arguments, *results): + if owner.array is None: + continue + for axis, references in enumerate(owner.array.extent_references): + missing = tuple(name for name in references if name not in scalar_names) + if missing: + blockers.append( + f"array owner {owner.owner_path!r} extent axis {axis} has unavailable scalar references {missing}" + ) + return tuple(blockers) + + def _argument_result_position(function: models.SemanticFunction, python_position: int) -> int | None: """Return one visible argument's completed projected result position.""" for mapping in function.projection: diff --git a/x2py/wrapper_codegen/__init__.py b/x2py/wrapper_codegen/__init__.py index 2b313c897..da7428f2e 100644 --- a/x2py/wrapper_codegen/__init__.py +++ b/x2py/wrapper_codegen/__init__.py @@ -28,6 +28,7 @@ CodeExpression, FortranAssignment, FortranCall, + FortranCase, FortranDeclaration, FortranFunction, FortranIf, @@ -36,10 +37,12 @@ FortranModule, FortranParameter, FortranPointerAssignment, + FortranSelectCase, FortranUse, ) from .plan import ( ArgumentTransferPlan, + ArrayHandoffPlan, BindingArgumentPlan, BindingFunctionPlan, BindingLifecyclePlan, @@ -73,6 +76,7 @@ __all__ = ( "ArgumentTransferPlan", + "ArrayHandoffPlan", "BackendScalarType", "BindingArgumentPlan", "BindingFunctionPlan", @@ -113,6 +117,7 @@ "FortranAssignment", "FortranBridgeGenerator", "FortranCall", + "FortranCase", "FortranDeclaration", "FortranFunction", "FortranIf", @@ -121,6 +126,7 @@ "FortranModule", "FortranParameter", "FortranPointerAssignment", + "FortranSelectCase", "FortranSourcePrinter", "FortranUse", "FunctionPlan", diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index 289c3474c..81e0aaaeb 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -3,9 +3,11 @@ from __future__ import annotations from dataclasses import dataclass +import re from x2py.semantics.ownership import ( CodegenAction, + ObjectKind, PythonBarrierAction, SetterAction, ) @@ -57,8 +59,14 @@ class _CArgumentNames: object_name: str value_name: str + length_name: str nullable_name: str present_name: str + extent_names: tuple[str, ...] + upper_bound_names: tuple[str, ...] + stride_names: tuple[str, ...] + runtime_rank_name: str + itemsize_name: str @dataclass @@ -68,6 +76,7 @@ class _CFunctionContext: result_name: str | None python_result_name: str | None python_results: dict[str, str] + role_values: dict[str, str] class CBindingGenerator(ClassVisitor): @@ -85,7 +94,17 @@ def _require_function_supported(self, function: FunctionPlan) -> None: for argument in function.arguments: self._require_argument_supported(argument) for result in function.results: - PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) + match result.object_kind: + case ObjectKind.SCALAR: + PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) + case ObjectKind.STRING: + self._require_string_binding_result_supported(result) + case ObjectKind.NUMPY_ARRAY: + self._require_array_binding_result_supported(result) + case _: + raise ValueError( + f"Unsupported C result object kind for {result.owner_path!r}: {result.object_kind!r}" + ) for slot in function.native_call_slots: self._require_native_result_supported(function, slot) for action in function.writeback_actions: @@ -96,47 +115,167 @@ def _require_argument_supported(self, argument: ArgumentTransferPlan) -> None: if argument.binding.python_action not in { PythonBarrierAction.SCALAR_VALUE, PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.STRING_STORAGE, + PythonBarrierAction.STRING_VALUE, PythonBarrierAction.RAW_ADDRESS, + PythonBarrierAction.ARRAY_STORAGE, }: raise ValueError( f"Unsupported C argument action for {argument.owner_path!r}: {argument.binding.python_action!r}" ) if ( - argument.binding.python_action in {PythonBarrierAction.SCALAR_STORAGE, PythonBarrierAction.RAW_ADDRESS} + argument.binding.python_action + in { + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.STRING_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + } and argument.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS ): raise ValueError(f"Unsupported C address handoff for {argument.owner_path!r}") - PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + match argument.object_kind: + case ObjectKind.SCALAR: + self._require_scalar_argument_supported(argument) + case ObjectKind.STRING: + self._require_string_argument_supported(argument) + case ObjectKind.NUMPY_ARRAY: + self._require_array_argument_supported(argument) + case _: + raise ValueError( + f"Unsupported C argument object kind for {argument.owner_path!r}: {argument.object_kind!r}" + ) def _require_native_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: - """Reject one unsupported native result output.""" + """Dispatch one native result output to its family support check.""" if slot.source_kind != "result": return + match slot.object_kind: + case ObjectKind.SCALAR: + self._require_scalar_native_result_supported(slot) + case ObjectKind.STRING: + self._require_string_result_supported(function, slot) + case ObjectKind.NUMPY_ARRAY: + self._require_array_result_supported(function, slot) + case _: + raise ValueError( + f"Unsupported C native result object kind for {slot.owner_path!r}: {slot.object_kind!r}" + ) + + # Scalar support checks. + def _require_scalar_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one first-lane primitive scalar argument type.""" + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_scalar_native_result_supported(self, slot: NativeCallSlotPlan) -> None: + """Require one first-lane primitive scalar native result type.""" + if slot.semantic_type_name is None: + raise ValueError(f"Missing C result datatype for {slot.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + + # Ordinary-array support checks. + def _require_array_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one completed ordinary array-buffer handoff.""" + array = argument.array + if array is None or (array.rank is not None and not 1 <= array.rank <= 15): + raise ValueError(f"Unsupported C array rank for {argument.owner_path!r}") + if array.contiguous not in {True, False}: + raise ValueError(f"Unsupported C array layout for {argument.owner_path!r}") + if argument.bridge.handoff_mode is not ArgumentHandoffMode.ARRAY_BUFFER: + raise ValueError(f"Unsupported C array handoff for {argument.owner_path!r}") + if argument.bridge.data_action is not BridgeDataAction.ASSOCIATE_VIEW: + raise ValueError(f"Unsupported C array data action for {argument.owner_path!r}") + if argument.datatype_family is not DatatypeFamily.STRING: + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_array_binding_result_supported(self, result: ResultPlan) -> None: + """Require one fixed-shape ordinary array result consumer.""" + if result.array is None or result.array.rank is None: + raise ValueError(f"Unsupported C array result for {result.owner_path!r}") + if result.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported C array result data action for {result.owner_path!r}") + if result.datatype_family is DatatypeFamily.STRING: + if result.array.itemsize is None or result.array.itemsize <= 0: + raise ValueError(f"Unsupported C character array result for {result.owner_path!r}") + return + PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) + + def _require_array_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Require one hidden fixed-shape array output slot.""" + if slot.array is None or slot.array.rank is None: + raise ValueError(f"Unsupported C array output for {slot.owner_path!r}") + if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported C array output data action for {slot.owner_path!r}") + if not any(result.native_call_slot is slot for result in function.results): + raise ValueError(f"Unclaimed C array output for {slot.owner_path!r}") if slot.datatype_family is DatatypeFamily.STRING: - self._require_string_result_supported(function, slot) + if slot.array.itemsize is None or slot.array.itemsize <= 0: + raise ValueError(f"Unsupported C character array output for {slot.owner_path!r}") return if slot.semantic_type_name is None: - raise ValueError(f"Missing C result datatype for {slot.owner_path!r}") + raise ValueError(f"Missing C array output datatype for {slot.owner_path!r}") PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + # String support checks. + def _require_string_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require a completed string value, storage, or raw-address action.""" + if argument.datatype_family is not DatatypeFamily.STRING: + raise ValueError(f"Unsupported C string datatype for {argument.owner_path!r}") + if argument.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported C string data action for {argument.owner_path!r}") + action = argument.binding.python_action + if action is PythonBarrierAction.STRING_VALUE: + self._require_string_value_argument_supported(argument) + return + if action not in {PythonBarrierAction.STRING_STORAGE, PythonBarrierAction.RAW_ADDRESS}: + raise ValueError(f"Unsupported C string boundary for {argument.owner_path!r}: {action!r}") + self._require_string_address_argument_supported(argument) + + def _require_string_value_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one character-buffer value handoff.""" + if argument.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + raise ValueError(f"Unsupported C string handoff for {argument.owner_path!r}") + if argument.binding.codegen_action not in {CodegenAction.CALL_LOCAL_INPUT, CodegenAction.COPY_IN_OUT}: + raise ValueError(f"Unsupported C string codegen action for {argument.owner_path!r}") + + def _require_string_address_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one fixed storage/raw-address handoff.""" + if argument.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS: + raise ValueError(f"Unsupported C string address handoff for {argument.owner_path!r}") + if argument.binding.codegen_action is not CodegenAction.IN_PLACE_ARGUMENT: + raise ValueError(f"Unsupported C string address action for {argument.owner_path!r}") + if argument.character_length is None or argument.character_length <= 0: + raise ValueError(f"Unsupported C string address length for {argument.owner_path!r}") + def _require_string_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: - """Require fixed-length status-message storage for one string result.""" - policy = function.binding.status_error - if policy is None: - raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") - if policy.message_role != slot.symbolic_role: - raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") - if slot.character_length is None: + """Require one fixed string result slot or status-message slot.""" + if slot.character_length is None or slot.character_length <= 0: raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: raise ValueError(f"Unsupported C string bridge data action for {slot.owner_path!r}") + policy = function.binding.status_error + if policy is not None and policy.message_role == slot.symbolic_role: + return + if any(result.native_call_slot is slot for result in function.results): + return + raise ValueError(f"Unsupported C string output for {slot.owner_path!r}") + + def _require_string_binding_result_supported(self, result: ResultPlan) -> None: + """Require one fixed string result consumer with a justified bridge copy.""" + if result.character_length is None or result.character_length <= 0: + raise ValueError(f"Unsupported C string result for {result.owner_path!r}") + if result.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported C string result data action for {result.owner_path!r}") def _require_writeback_supported(self, action: LifecycleActionPlan) -> None: - """Require a scalar type for one binding-owned copy-out action.""" + """Require a supported scalar or string binding copy-out action.""" if action.phase is not WritebackPhase.COPY_OUT: return if action.binding is None: return + if action.binding.datatype_family is DatatypeFamily.STRING: + if action.binding.codegen_action is not CodegenAction.COPY_IN_OUT: + raise ValueError(f"Unsupported C string writeback for {action.owner_path!r}") + return PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: @@ -179,11 +318,22 @@ def _module_needs_allocator(self, plan: ModulePlan) -> bool: """Return whether emitted bridge-owned copies need the shared allocator.""" return any( variable.binding.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT for variable in self._variables(plan) - ) or any( - slot.datatype_family is DatatypeFamily.STRING - for function in self._functions(plan) - for slot in function.native_call_slots - if slot.source_kind == "result" + ) or any(self._function_needs_allocator(function) for function in self._functions(plan)) + + def _function_needs_allocator(self, function: FunctionPlan) -> bool: + """Return whether one binding/bridge function owns allocated string storage.""" + return ( + any(result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} for result in function.results) + or any( + argument.object_kind is ObjectKind.STRING + and argument.binding.codegen_action is CodegenAction.COPY_IN_OUT + for argument in function.arguments + ) + or any( + slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} + for slot in function.native_call_slots + if slot.source_kind == "result" + ) ) def _module_defines(self, needs_runtime: bool) -> tuple[CMacroDefinition, ...]: @@ -204,11 +354,28 @@ def _module_includes( CInclude("stdint.h"), CInclude("stdbool.h"), CInclude("complex.h"), + *((CInclude("string.h"),) if self._module_uses_memory_copy(plan) else ()), *((CInclude("stdlib.h"),) if needs_free else ()), *self._module_runtime_includes(needs_runtime), CInclude(f"{plan.binding.owner_path}_wrapper.h", system=False), ) + def _module_uses_string_values(self, plan: ModulePlan) -> bool: + """Return whether binding conversion needs C string helpers.""" + return any( + argument.binding.python_action is PythonBarrierAction.STRING_VALUE + for function in self._functions(plan) + for argument in function.arguments + ) + + def _module_uses_memory_copy(self, plan: ModulePlan) -> bool: + """Return whether binding conversion emits string or array byte copies.""" + return self._module_uses_string_values(plan) or any( + result.object_kind is ObjectKind.NUMPY_ARRAY + for function in self._functions(plan) + for result in function.results + ) + def _module_runtime_includes(self, required: bool) -> tuple[CInclude, ...]: """Return NumPy/runtime includes when generated nodes consume them.""" if not required: @@ -411,9 +578,7 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> CFunction: """Recursively assemble one complete CPython binding function.""" context = self._function_context(plan) argument_nodes = tuple( - node - for argument in sorted(plan.arguments, key=lambda item: item.python_position) - for node in self.visit(argument, context=context) + node for argument in self._binding_conversion_order(plan) for node in self.visit(argument, context=context) ) output_nodes = self._output_nodes(plan, context) return CFunction( @@ -432,6 +597,18 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> CFunction: ), ) + def _binding_conversion_order(self, plan: FunctionPlan) -> tuple[ArgumentTransferPlan, ...]: + """Convert non-owning inputs before allocating the sole replacement buffer.""" + return tuple( + sorted( + plan.arguments, + key=lambda argument: ( + argument.binding.codegen_action is CodegenAction.COPY_IN_OUT, + argument.python_position, + ), + ) + ) + def _visit_ArgumentTransferPlan( self, plan: ArgumentTransferPlan, @@ -461,7 +638,7 @@ def _lower_argument_required( self, plan: ArgumentTransferPlan, context: _CFunctionContext, - ) -> tuple[CDeclaration | CExpressionStatement, ...]: + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: """Dispatch one required argument from its completed Python action.""" action = plan.binding.python_action match action: @@ -469,10 +646,17 @@ def _lower_argument_required( return self._lower_argument_required_scalar_value(plan, context) case PythonBarrierAction.SCALAR_STORAGE: return self._lower_argument_required_scalar_storage(plan, context) + case PythonBarrierAction.STRING_STORAGE: + return self._lower_argument_required_string_storage(plan, context) + case PythonBarrierAction.STRING_VALUE: + return self._lower_argument_required_string_value(plan, context) case PythonBarrierAction.RAW_ADDRESS: return self._lower_argument_required_raw_address(plan, context) + case PythonBarrierAction.ARRAY_STORAGE: + return self._lower_argument_required_array_storage(plan, context) raise ValueError(f"Unsupported required C argument action for {plan.owner_path!r}: {action!r}") + # Scalar argument lowering. def _lower_argument_required_scalar_value( self, plan: ArgumentTransferPlan, @@ -500,6 +684,426 @@ def _lower_argument_required_scalar_value( CExpressionStatement(CodeExpression("if (PyErr_Occurred()) return NULL")), ) + # String argument lowering. + def _lower_argument_required_string_value( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Dispatch one completed string input-storage action.""" + action = plan.binding.codegen_action + if action is CodegenAction.CALL_LOCAL_INPUT: + return self._lower_argument_required_string_input(plan, context) + if action is CodegenAction.COPY_IN_OUT: + return self._lower_argument_required_string_replacement(plan, context) + raise ValueError(f"Unsupported required C string action for {plan.owner_path!r}: {action!r}") + + def _lower_argument_required_string_input( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Validate and borrow one read-only UTF-8 payload for the call.""" + names = context.arguments[plan.owner_path] + return ( + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(names.value_name, "const char *", CodeExpression("NULL")), + CDeclaration(names.length_name, "Py_ssize_t", CodeExpression("0")), + *self._required_string_validation_nodes(plan, names, names.value_name), + ) + + def _lower_argument_required_string_replacement( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Allocate and populate one mutable string call buffer.""" + names = context.arguments[plan.owner_path] + source_name = f"{names.value_name}_source" + return ( + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(source_name, "const char *", CodeExpression("NULL")), + CDeclaration(names.value_name, "char *", CodeExpression("NULL")), + CDeclaration(names.length_name, "Py_ssize_t", CodeExpression("0")), + *self._required_string_validation_nodes(plan, names, source_name), + *self._string_replacement_allocation_nodes(plan, names, source_name), + ) + + def _string_replacement_allocation_nodes( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + source_name: str, + ) -> tuple[CExpressionStatement | CIf, ...]: + """Allocate and copy one validated mutable string payload.""" + return ( + CExpressionStatement( + CodeExpression(f"{names.value_name} = (char *)x2py_malloc((size_t){names.length_name} + 1)") + ), + CIf( + CodeExpression(f"{names.value_name} == NULL"), + body=( + CExpressionStatement( + CodeExpression( + f'PyErr_SetString(PyExc_MemoryError, "Unable to allocate mutable string buffer ' + f'for argument {plan.binding.python_name}.")' + ) + ), + CReturn(CodeExpression("NULL")), + ), + ), + CExpressionStatement( + CodeExpression(f"memcpy({names.value_name}, {source_name}, (size_t){names.length_name})") + ), + CExpressionStatement(CodeExpression(f"{names.value_name}[{names.length_name}] = '\\0'")), + ) + + def _required_string_validation_nodes( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + payload_name: str, + ) -> tuple[CExpressionStatement, ...]: + """Return shared required-string type, UTF-8, NUL, and length checks.""" + nodes = [ + CExpressionStatement( + CodeExpression( + f"if (!PyUnicode_Check({names.object_name})) {{ " + f'PyErr_Format(PyExc_TypeError, "Expected an argument of type str for argument ' + f"{plan.binding.python_name}. Received \", " + f"Py_TYPE({names.object_name})->tp_name); return NULL; }}" + ) + ), + CExpressionStatement( + CodeExpression(f"{payload_name} = PyUnicode_AsUTF8AndSize({names.object_name}, &{names.length_name})") + ), + CExpressionStatement(CodeExpression(f"if ({payload_name} == NULL) return NULL")), + CExpressionStatement( + CodeExpression( + f"if ((Py_ssize_t)strlen({payload_name}) != {names.length_name}) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} cannot contain ' + 'embedded NUL"); return NULL; }' + ) + ), + ] + fixed_length = plan.character_length + if fixed_length is not None: + nodes.append( + CExpressionStatement( + CodeExpression( + f"if ({names.length_name} != {fixed_length}) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must encode to ' + f'exactly {fixed_length} bytes"); return NULL; }}' + ) + ) + ) + return tuple(nodes) + + # Ordinary-array argument lowering. + def _lower_argument_required_array_storage( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Validate and borrow one completed ordinary NumPy array buffer.""" + array_plan = plan.array + if array_plan is None: + raise ValueError(f"Array argument {plan.owner_path!r} is missing its handoff") + names = context.arguments[plan.owner_path] + array = f"(PyArrayObject *){names.object_name}" + nodes = [ + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(names.value_name, "void *", CodeExpression("NULL")), + *(CDeclaration(name, "int64_t", CodeExpression("0")) for name in names.extent_names), + *( + (CDeclaration(name, "int64_t", CodeExpression("0")) for name in names.upper_bound_names) + if array_plan.upper_bound_roles + else () + ), + *( + (CDeclaration(name, "int64_t", CodeExpression("1")) for name in names.stride_names) + if array_plan.stride_roles + else () + ), + *( + (CDeclaration(names.runtime_rank_name, "int64_t", CodeExpression("0")),) + if array_plan.runtime_rank_role is not None + else () + ), + *( + (CDeclaration(names.itemsize_name, "int64_t", CodeExpression("0")),) + if array_plan.itemsize_role is not None + else () + ), + self._array_type_and_rank_check(plan, names, array), + *self._array_access_checks(plan, array), + *self._array_layout_checks(plan, array), + *self._array_shape_checks(plan, context, array), + ] + nodes.extend(self._array_extraction_nodes(plan, names, array)) + return tuple(nodes) + + def _array_type_and_rank_check( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + array: str, + ) -> CExpressionStatement: + """Require the exact NumPy element type and completed rank shape.""" + handoff = plan.array + if handoff is None: + raise ValueError(f"Array argument {plan.owner_path!r} is missing its handoff") + if plan.datatype_family is DatatypeFamily.STRING: + numpy_type = "NPY_STRING" + python_type = f"numpy.bytes_[{handoff.itemsize}]" + else: + scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) + if scalar_type.numpy_type_macro is None: + raise ValueError(f"Unsupported array element type {plan.semantic_type_name!r}") + numpy_type = scalar_type.numpy_type_macro + python_type = scalar_type.python_type_name + rank_check = ( + f"PyArray_NDIM({array}) < 1 || PyArray_NDIM({array}) > 15" + if handoff.rank is None + else f"PyArray_NDIM({array}) != {handoff.rank}" + ) + return CExpressionStatement( + CodeExpression( + f"if (!PyArray_Check({names.object_name}) || PyArray_TYPE({array}) != {numpy_type} || " + f'{rank_check}) {{ PyErr_Format(PyExc_TypeError, "Expected a compatible numpy.ndarray of ' + f"type {python_type} for argument {plan.binding.python_name}. Received \", " + f"Py_TYPE({names.object_name})->tp_name); return NULL; }}" + ) + ) + + def _array_access_checks( + self, + plan: ArgumentTransferPlan, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Require native byte order, alignment, and planned writeability.""" + checks = [ + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISNOTSWAPPED({array})) {{ PyErr_SetString(PyExc_TypeError, " + f'"Argument {plan.binding.python_name} must use native byte order"); return NULL; }}' + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISALIGNED({array})) {{ PyErr_SetString(PyExc_TypeError, " + f'"Argument {plan.binding.python_name} must be aligned"); return NULL; }}' + ) + ), + ] + if plan.binding.writable: + checks.append( + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISWRITEABLE({array})) {{ PyErr_SetString(PyExc_TypeError, " + f'"Argument {plan.binding.python_name} must be writeable"); return NULL; }}' + ) + ) + ) + return tuple(checks) + + def _array_layout_checks( + self, + plan: ArgumentTransferPlan, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Dispatch dense or positive-strided layout validation.""" + handoff = plan.array + if handoff is None: + return () + if handoff.contiguous is False: + return self._positive_strided_array_checks(plan, array) + if handoff.order == "ORDER_C": + condition = f"!PyArray_IS_C_CONTIGUOUS({array})" + elif handoff.order == "ORDER_F" or (handoff.rank is not None and handoff.rank > 1): + condition = f"!PyArray_IS_F_CONTIGUOUS({array})" + else: + condition = f"!(PyArray_IS_C_CONTIGUOUS({array}) || PyArray_IS_F_CONTIGUOUS({array}))" + return ( + CExpressionStatement( + CodeExpression( + f"if ({condition}) {{ PyErr_SetString(PyExc_TypeError, " + f'"Argument {plan.binding.python_name} must satisfy its contiguous layout"); return NULL; }}' + ) + ), + ) + + def _positive_strided_array_checks( + self, + plan: ArgumentTransferPlan, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Require positive non-overlapping Fortran-oriented element strides.""" + handoff = plan.array + if handoff is None or handoff.rank is None: + raise ValueError(f"Strided array {plan.owner_path!r} requires a concrete rank") + checks = [] + for axis in range(handoff.rank): + stride = f"PyArray_STRIDE({array}, {axis})" + checks.append( + CExpressionStatement( + CodeExpression( + f"if (({stride} % PyArray_ITEMSIZE({array})) != 0 || " + f"(PyArray_SIZE({array}) > 0 && PyArray_DIM({array}, {axis}) > 1 && {stride} <= 0)) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must use positive ' + f'element strides"); return NULL; }}' + ) + ) + ) + if axis: + previous_stride = f"PyArray_STRIDE({array}, {axis - 1})" + previous_extent = f"PyArray_DIM({array}, {axis - 1})" + checks.append( + CExpressionStatement( + CodeExpression( + f"if (PyArray_SIZE({array}) > 0 && {previous_extent} > 0 && " + f"{stride} < {previous_stride} * {previous_extent}) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must use a ' + f'Fortran-oriented non-overlapping view"); return NULL; }}' + ) + ) + ) + return tuple(checks) + + def _array_shape_checks( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Validate every concrete declared extent against completed scalar roles.""" + handoff = plan.array + if handoff is None or handoff.rank is None: + return () + checks = [] + runtime_markers = {":", "::Strided", "Flat"} + for axis, expression in enumerate(handoff.shape): + if expression in runtime_markers: + continue + expected = self._array_extent_expression(handoff, axis, expression, context) + checks.append( + CExpressionStatement( + CodeExpression( + f"if (PyArray_DIM({array}, {axis}) != (npy_intp)({expected})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} has incompatible ' + f'shape at axis {axis}"); return NULL; }}' + ) + ) + ) + return tuple(checks) + + def _array_extent_expression( + self, + handoff, + axis: int, + expression: str, + context: _CFunctionContext, + ) -> str: + """Lower one validated extent expression through its planned role references.""" + lowered = expression + for role in handoff.extent_reference_roles[axis]: + try: + value_name = context.role_values[role] + except KeyError: + raise ValueError(f"Array extent role {role!r} has no binding value") from None + reference_name = role.rsplit(".", 1)[-1].split(":", 1)[0] + lowered = re.sub(rf"\b{re.escape(reference_name)}\b", value_name, lowered) + return lowered + + def _array_extraction_nodes( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Extract only the ABI fields named by the editable handoff plan.""" + handoff = plan.array + if handoff is None: + return () + nodes = [CExpressionStatement(CodeExpression(f"{names.value_name} = PyArray_DATA({array})"))] + if handoff.runtime_rank_role is not None: + nodes.append( + CExpressionStatement(CodeExpression(f"{names.runtime_rank_name} = (int64_t)PyArray_NDIM({array})")) + ) + if handoff.itemsize_role is not None: + nodes.extend( + ( + CExpressionStatement(CodeExpression(f"{names.itemsize_name} = (int64_t)PyArray_ITEMSIZE({array})")), + CExpressionStatement( + CodeExpression( + f"if ({names.itemsize_name} != {handoff.itemsize}) {{ PyErr_SetString(PyExc_TypeError, " + f'"Argument {plan.binding.python_name} must have NumPy bytes dtype itemsize ' + f'{handoff.itemsize}"); return NULL; }}' + ) + ), + ) + ) + active_rank = 15 if handoff.rank is None else handoff.rank + for axis in range(active_rank): + guard = f"if (PyArray_NDIM({array}) > {axis}) " if handoff.rank is None else "" + nodes.append( + CExpressionStatement( + CodeExpression(f"{guard}{names.extent_names[axis]} = (int64_t)PyArray_DIM({array}, {axis})") + ) + ) + if handoff.contiguous is False: + nodes.extend(self._strided_array_extraction_nodes(handoff.rank, names, array)) + return tuple(nodes) + + def _strided_array_extraction_nodes( + self, + rank: int | None, + names: _CArgumentNames, + array: str, + ) -> tuple[CExpressionStatement, ...]: + """Compute bridge base extents, slice bounds, and relative strides.""" + if rank is None: + raise ValueError("Assumed-rank strided arrays require a separate completed lane") + nodes = [] + base_product = "1" + for axis in range(rank): + absolute_stride = f"(PyArray_STRIDE({array}, {axis}) / PyArray_ITEMSIZE({array}))" + nodes.append( + CExpressionStatement( + CodeExpression( + f"{names.stride_names[axis]} = PyArray_SIZE({array}) == 0 ? 1 : " + f"{absolute_stride} / ({base_product})" + ) + ) + ) + nodes.append( + CExpressionStatement( + CodeExpression( + f"{names.upper_bound_names[axis]} = {names.extent_names[axis]} == 0 ? -1 : " + f"({names.extent_names[axis]} - 1) * {names.stride_names[axis]}" + ) + ) + ) + if axis + 1 < rank: + next_stride = f"(PyArray_STRIDE({array}, {axis + 1}) / PyArray_ITEMSIZE({array}))" + nodes.append( + CExpressionStatement( + CodeExpression( + f"{names.extent_names[axis]} = {next_stride} / ({base_product}); " + f"if ({names.extent_names[axis]} < 1) {names.extent_names[axis]} = 1" + ) + ) + ) + base_product = f"({base_product}) * {names.extent_names[axis]}" + else: + nodes.append( + CExpressionStatement( + CodeExpression(f"{names.extent_names[axis]} = {names.upper_bound_names[axis]} + 1") + ) + ) + return tuple(nodes) + + # Scalar storage and address lowering. def _lower_argument_required_scalar_storage( self, plan: ArgumentTransferPlan, @@ -552,6 +1156,61 @@ def _lower_argument_required_scalar_storage( nodes.append(CExpressionStatement(CodeExpression(f"{names.value_name} = PyArray_DATA({array})"))) return tuple(nodes) + # String storage lowering. + def _lower_argument_required_string_storage( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Validate and borrow one rank-zero fixed-width NumPy bytes buffer.""" + if plan.character_length is None or plan.character_length <= 0: + raise ValueError(f"String storage {plan.owner_path!r} is missing a fixed length") + names = context.arguments[plan.owner_path] + array = f"(PyArrayObject *){names.object_name}" + length = plan.character_length + return ( + CDeclaration(names.object_name, "PyObject *"), + CDeclaration(names.value_name, "void *", CodeExpression("NULL")), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_Check({names.object_name}) || PyArray_TYPE({array}) != NPY_STRING || " + f"PyArray_NDIM({array}) != 0) {{ " + f'PyErr_Format(PyExc_TypeError, "Expected a rank-zero numpy.ndarray with dtype S{length} ' + f"for argument {plan.binding.python_name}. Received \", " + f"Py_TYPE({names.object_name})->tp_name); return NULL; }}" + ) + ), + CExpressionStatement( + CodeExpression( + f"if (PyArray_ITEMSIZE({array}) != {length}) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must use itemsize ' + f'{length}"); return NULL; }}' + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISNOTSWAPPED({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must use native ' + 'byte order"); return NULL; }' + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISALIGNED({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must be aligned"); ' + "return NULL; }" + ) + ), + CExpressionStatement( + CodeExpression( + f"if (!PyArray_ISWRITEABLE({array})) {{ " + f'PyErr_SetString(PyExc_TypeError, "Argument {plan.binding.python_name} must be writeable"); ' + "return NULL; }" + ) + ), + CExpressionStatement(CodeExpression(f"{names.value_name} = PyArray_DATA({array})")), + ) + def _lower_argument_required_raw_address( self, plan: ArgumentTransferPlan, @@ -579,7 +1238,15 @@ def _lower_argument_nullable_value( plan: ArgumentTransferPlan, context: _CFunctionContext, ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: - """Return omitted-or-value conversion nodes for an optional scalar.""" + """Return omitted-or-value conversion nodes for an optional value.""" + if plan.object_kind is ObjectKind.NUMPY_ARRAY: + return self._lower_argument_nullable_array_storage(plan, context) + if plan.object_kind is ObjectKind.STRING: + return self._lower_argument_nullable_string_value(plan, context) + if plan.object_kind is not ObjectKind.SCALAR: + raise ValueError( + f"Unsupported optional C argument object kind for {plan.owner_path!r}: {plan.object_kind!r}" + ) scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) names = context.arguments[plan.owner_path] return ( @@ -597,6 +1264,61 @@ def _lower_argument_nullable_value( ), ) + # Optional ordinary-array lowering. + def _lower_argument_nullable_array_storage( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CIf, ...]: + """Preserve absent optional arrays or validate one present NumPy buffer.""" + names = context.arguments[plan.owner_path] + required_nodes = self._lower_argument_required_array_storage(plan, context) + declarations = tuple(node for node in required_nodes if isinstance(node, CDeclaration)) + body = tuple(node for node in required_nodes if not isinstance(node, CDeclaration)) + return ( + CDeclaration(names.object_name, "PyObject *", CodeExpression("Py_None")), + *(node for node in declarations if node.name != names.object_name), + CIf(CodeExpression(f"{names.object_name} != Py_None"), body=body), + ) + + # Optional string lowering. + def _lower_argument_nullable_string_value( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CIf, ...]: + """Convert a concrete optional string or preserve its absent state.""" + names = context.arguments[plan.owner_path] + declarations: tuple[CDeclaration, ...] = ( + CDeclaration(names.object_name, "PyObject *", CodeExpression("Py_None")), + CDeclaration(names.length_name, "Py_ssize_t", CodeExpression("0")), + ) + action = plan.binding.codegen_action + if action is CodegenAction.CALL_LOCAL_INPUT: + return ( + *declarations, + CDeclaration(names.value_name, "const char *", CodeExpression("NULL")), + CIf( + CodeExpression(f"{names.object_name} != Py_None"), + body=self._required_string_validation_nodes(plan, names, names.value_name), + ), + ) + if action is CodegenAction.COPY_IN_OUT: + source_name = f"{names.value_name}_source" + return ( + *declarations, + CDeclaration(source_name, "const char *", CodeExpression("NULL")), + CDeclaration(names.value_name, "char *", CodeExpression("NULL")), + CIf( + CodeExpression(f"{names.object_name} != Py_None"), + body=( + *self._required_string_validation_nodes(plan, names, source_name), + *self._string_replacement_allocation_nodes(plan, names, source_name), + ), + ), + ) + raise ValueError(f"Unsupported optional C string action for {plan.owner_path!r}: {action!r}") + def _lower_argument_descriptor( self, plan: ArgumentTransferPlan, @@ -659,23 +1381,140 @@ def _lower_result( failure_cleanup: tuple[str, ...], ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: """Dispatch one completed binding result action explicitly.""" - action = plan.binding.codegen_action - match action: - case CodegenAction.DIRECT_VALUE: - return self._lower_result_direct_value(plan, context, failure_cleanup) - case CodegenAction.HIDDEN_OUTPUT: - return self._lower_result_hidden_output(plan, context, failure_cleanup) - raise ValueError(f"Unsupported C result action for {plan.owner_path!r}: {action!r}") + match plan.object_kind: + case ObjectKind.NUMPY_ARRAY: + return self._lower_result_array_copy(plan, context, failure_cleanup) + case ObjectKind.STRING: + return self._lower_result_fixed_string(plan, context, failure_cleanup) + case ObjectKind.SCALAR: + if plan.binding.codegen_action is CodegenAction.DIRECT_VALUE: + return self._lower_result_direct_value(plan, context, failure_cleanup) + raise ValueError( + f"Unsupported C scalar result action for {plan.owner_path!r}: {plan.binding.codegen_action!r}" + ) + case _: + raise ValueError(f"Unsupported C result object kind for {plan.owner_path!r}: {plan.object_kind!r}") - def _lower_result_direct_value( + # Ordinary-array result lowering. + def _lower_result_array_copy( + self, + plan: ResultPlan, + context: _CFunctionContext, + failure_cleanup: tuple[str, ...], + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Copy one bridge-owned fixed-shape array into Python-owned NumPy storage.""" + handoff = plan.array + native_name = self._result_native_name(plan, context) + python_name = context.python_results.get(plan.owner_path) + if handoff is None or handoff.rank is None or python_name is None: + raise ValueError(f"Array result {plan.owner_path!r} has no fixed binding shape") + dimensions = tuple( + self._array_extent_expression(handoff, axis, expression, context) + for axis, expression in enumerate(handoff.shape) + ) + dims_name = f"{python_name}_dims" + fortran_order = 0 if handoff.order == "ORDER_C" or handoff.rank == 1 else 1 + decrefs = tuple(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in failure_cleanup) + return ( + CIf( + CodeExpression(f"{native_name} == NULL"), + body=( + CExpressionStatement( + CodeExpression( + 'PyErr_SetString(PyExc_MemoryError, "Unable to allocate copy-return output array.")' + ) + ), + *decrefs, + CReturn(CodeExpression("NULL")), + ), + ), + CDeclaration(f"{dims_name}[]", "npy_intp", CodeExpression(f"{{{', '.join(dimensions)}}}")), + CDeclaration( + python_name, + "PyObject *", + self._array_result_creation_expression(plan, handoff.rank, dims_name, fortran_order), + ), + CIf( + CodeExpression(f"{python_name} == NULL"), + body=( + CExpressionStatement(CodeExpression(f"free({native_name})")), + *decrefs, + CReturn(CodeExpression("NULL")), + ), + ), + CExpressionStatement( + CodeExpression( + f"memcpy(PyArray_DATA((PyArrayObject *){python_name}), {native_name}, " + f"(size_t)PyArray_NBYTES((PyArrayObject *){python_name}))" + ) + ), + CExpressionStatement(CodeExpression(f"free({native_name})")), + ) + + def _array_result_creation_expression( + self, + plan: ResultPlan, + rank: int, + dims_name: str, + fortran_order: int, + ) -> CodeExpression: + """Construct one numeric or fixed-width bytes NumPy result array.""" + if plan.datatype_family is DatatypeFamily.STRING: + handoff = plan.array + if handoff is None or handoff.itemsize is None or handoff.itemsize <= 0: + raise ValueError(f"Character array result {plan.owner_path!r} has no fixed itemsize") + flags = "NPY_ARRAY_F_CONTIGUOUS" if fortran_order else "0" + return CodeExpression( + f"(PyObject *)PyArray_New(&PyArray_Type, {rank}, {dims_name}, NPY_STRING, " + f"NULL, NULL, {handoff.itemsize}, {flags}, NULL)" + ) + scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) + if scalar_type.numpy_type_macro is None: + raise ValueError(f"Unsupported array result type {plan.semantic_type_name!r}") + return CodeExpression( + f"(PyObject *)PyArray_EMPTY({rank}, {dims_name}, {scalar_type.numpy_type_macro}, {fortran_order})" + ) + + # String result lowering. + def _lower_result_fixed_string( self, plan: ResultPlan, context: _CFunctionContext, failure_cleanup: tuple[str, ...], ) -> tuple[CExpressionStatement | CDeclaration | CIf, ...]: - return self._lower_result_value(plan, context, failure_cleanup) + """Consume one bridge-owned NUL-terminated fixed string copy.""" + native_name = self._result_native_name(plan, context) + python_name = context.python_results.get(plan.owner_path) + if python_name is None: + raise ValueError(f"String result {plan.owner_path!r} has no Python result role") + decrefs = tuple(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in failure_cleanup) + return ( + CIf( + CodeExpression(f"{native_name} == NULL"), + body=( + CExpressionStatement( + CodeExpression( + 'PyErr_SetString(PyExc_MemoryError, "Unable to allocate copy-return output string.")' + ) + ), + *decrefs, + CReturn(CodeExpression("NULL")), + ), + ), + CDeclaration( + python_name, + "PyObject *", + CodeExpression(f'Py_BuildValue("s", (const char *){native_name})'), + ), + CExpressionStatement(CodeExpression(f"free({native_name})")), + CIf( + CodeExpression(f"{python_name} == NULL"), + body=(*decrefs, CReturn(CodeExpression("NULL"))), + ), + ) - def _lower_result_hidden_output( + # Scalar result lowering. + def _lower_result_direct_value( self, plan: ResultPlan, context: _CFunctionContext, @@ -751,30 +1590,35 @@ def _binding_result_nodes( for result in ordered: nodes.extend(self.visit(result, context=context, failure_cleanup=tuple(converted))) converted.append(context.python_results[result.owner_path]) + nodes.extend(self._python_result_aggregation_nodes(tuple(converted), context)) + return tuple(nodes) + + def _python_result_aggregation_nodes( + self, + converted: tuple[str, ...], + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: + """Return one object directly or assemble ordered tuple ownership.""" if len(converted) == 1: - nodes.append(CReturn(CodeExpression(converted[0]))) - return tuple(nodes) + return (CReturn(CodeExpression(converted[0])),) aggregate = context.python_result_name if aggregate is None: - raise ValueError(f"{plan.owner_path!r} multiple results have no aggregate binding role") - nodes.extend( - ( - CDeclaration(aggregate, "PyObject *", CodeExpression(f"PyTuple_New({len(converted)})")), - CIf( - CodeExpression(f"{aggregate} == NULL"), - body=( - *(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in converted), - CReturn(CodeExpression("NULL")), - ), - ), - *( - CExpressionStatement(CodeExpression(f"PyTuple_SET_ITEM({aggregate}, {position}, {name})")) - for position, name in enumerate(converted) + raise ValueError("Multiple Python results have no aggregate binding role") + return ( + CDeclaration(aggregate, "PyObject *", CodeExpression(f"PyTuple_New({len(converted)})")), + CIf( + CodeExpression(f"{aggregate} == NULL"), + body=( + *(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in converted), + CReturn(CodeExpression("NULL")), ), - CReturn(CodeExpression(aggregate)), - ) + ), + *( + CExpressionStatement(CodeExpression(f"PyTuple_SET_ITEM({aggregate}, {position}, {name})")) + for position, name in enumerate(converted) + ), + CReturn(CodeExpression(aggregate)), ) - return tuple(nodes) def _bridge_call_statement(self, plan: FunctionPlan, context: _CFunctionContext) -> CExpressionStatement: """Return the mechanical bridge call selected by result storage.""" @@ -876,7 +1720,7 @@ def _writeback_nodes( plan: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return scalar replacement conversion after the native call.""" + """Return the sole replacement conversion after the native call.""" ordered = sorted( ( action @@ -885,11 +1729,37 @@ def _writeback_nodes( ), key=lambda item: item.binding.result_position, ) + if ordered and all(action.object_kind is ObjectKind.NUMPY_ARRAY for action in ordered): + return self._array_identity_writeback_nodes(plan, tuple(ordered), context) if len(ordered) != 1: - raise ValueError(f"{plan.owner_path!r} requires exactly one scalar writeback result") + raise ValueError(f"{plan.owner_path!r} requires exactly one writeback result") action = ordered[0] return self._lower_writeback(plan, action, context) + # Ordinary-array writeback lowering. + def _array_identity_writeback_nodes( + self, + plan: FunctionPlan, + actions: tuple[LifecycleActionPlan, ...], + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: + """Return original validated NumPy objects with owned references.""" + nodes = [] + converted = [] + for action in actions: + source = self._argument_for_role(plan, action.source_role) + names = context.arguments[source.owner_path] + python_name = context.python_results[action.owner_path] + nodes.extend( + ( + CDeclaration(python_name, "PyObject *", CodeExpression(names.object_name)), + CExpressionStatement(CodeExpression(f"Py_INCREF({python_name})")), + ) + ) + converted.append(python_name) + nodes.extend(self._python_result_aggregation_nodes(tuple(converted), context)) + return tuple(nodes) + def _lower_writeback( self, plan: FunctionPlan, @@ -911,8 +1781,77 @@ def _lower_writeback_copy_in_out( action: LifecycleActionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: + if action.binding.datatype_family is DatatypeFamily.STRING: + return self._lower_writeback_string(plan, action, context) return self._lower_writeback_value(plan, action, context) + # String writeback lowering. + def _lower_writeback_string( + self, + plan: FunctionPlan, + action: LifecycleActionPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: + """Dispatch string replacement conversion from completed presence mode.""" + source = self._argument_for_role(plan, action.source_role) + if source.binding.optional_mode is OptionalMode.REQUIRED: + return self._lower_writeback_required_string(source, context) + if source.binding.optional_mode is OptionalMode.NULLABLE_VALUE: + return self._lower_writeback_optional_string(source, context) + raise ValueError(f"Unsupported string writeback presence mode for {source.owner_path!r}") + + def _lower_writeback_required_string( + self, + source: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: + """Convert one required mutable buffer and release it exactly once.""" + names = context.arguments[source.owner_path] + python_result_name = context.python_result_name or "result_obj" + return ( + CDeclaration( + python_result_name, + "PyObject *", + CodeExpression(f'Py_BuildValue("s", (const char *){names.value_name})'), + ), + CExpressionStatement(CodeExpression(f"free({names.value_name})")), + CIf( + CodeExpression(f"{python_result_name} == NULL"), + body=(CReturn(CodeExpression("NULL")),), + ), + CReturn(CodeExpression(python_result_name)), + ) + + def _lower_writeback_optional_string( + self, + source: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CIf | CReturn, ...]: + """Return None for absence or convert and release one concrete replacement.""" + names = context.arguments[source.owner_path] + python_result_name = context.python_result_name or "result_obj" + return ( + CDeclaration(python_result_name, "PyObject *", CodeExpression("NULL")), + CIf( + CodeExpression(f"{names.value_name} == NULL"), + body=( + CExpressionStatement(CodeExpression("Py_INCREF(Py_None)")), + CExpressionStatement(CodeExpression(f"{python_result_name} = Py_None")), + ), + else_body=( + CExpressionStatement( + CodeExpression(f'{python_result_name} = Py_BuildValue("s", (const char *){names.value_name})') + ), + CExpressionStatement(CodeExpression(f"free({names.value_name})")), + CIf( + CodeExpression(f"{python_result_name} == NULL"), + body=(CReturn(CodeExpression("NULL")),), + ), + ), + ), + CReturn(CodeExpression(python_result_name)), + ) + def _lower_writeback_in_place_argument( self, plan: FunctionPlan, @@ -921,6 +1860,7 @@ def _lower_writeback_in_place_argument( ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: return self._lower_writeback_value(plan, action, context) + # Scalar writeback lowering. def _lower_writeback_value( self, plan: FunctionPlan, @@ -950,28 +1890,95 @@ def _argument_for_role(self, plan: FunctionPlan, role: str) -> ArgumentTransferP raise ValueError(f"{plan.owner_path!r} has no argument for lifecycle role {role!r}") def _function_context(self, plan: FunctionPlan) -> _CFunctionContext: - arguments = { - argument.owner_path: _CArgumentNames( - f"{argument.binding.python_name.lower()}_obj", - argument.binding.python_name.lower(), - f"{argument.binding.python_name.lower()}_nullable", - f"{argument.binding.python_name.lower()}_present", - ) - for argument in plan.arguments - } - native_outputs = { + arguments = self._argument_contexts(plan) + native_outputs = self._native_output_names(plan) + output_owners = self._output_owners(plan) + python_results = self._python_result_names(output_owners) + python_result = self._python_result_name(plan) + native_result = self._native_result_name(plan) + role_values = self._argument_role_values(plan, arguments) + return _CFunctionContext( + arguments, + native_outputs, + native_result, + python_result, + python_results, + role_values, + ) + + def _argument_contexts(self, plan: FunctionPlan) -> dict[str, _CArgumentNames]: + """Name the binding locals for every Python argument.""" + return {argument.owner_path: self._argument_context_names(argument) for argument in plan.arguments} + + def _native_output_names(self, plan: FunctionPlan) -> dict[str, str]: + """Name native hidden-output locals by their completed symbolic roles.""" + return { slot.symbolic_role: slot.native_name.lower() for slot in plan.native_call_slots if slot.source_kind == "result" } - ordered_results = tuple(sorted(plan.results, key=lambda item: item.result_position)) - python_results = { - result.owner_path: ("result_obj" if len(ordered_results) == 1 else f"result_{result.result_position}_obj") - for result in ordered_results + + def _output_owners(self, plan: FunctionPlan) -> tuple[tuple[str, int], ...]: + """Return ordered Python result owners from results or copy-out actions.""" + results = tuple(sorted(plan.results, key=lambda item: item.result_position)) + if results: + return tuple((result.owner_path, result.result_position) for result in results) + writebacks = self._ordered_output_writebacks(plan) + return tuple((action.owner_path, action.binding.result_position) for action in writebacks) + + def _ordered_output_writebacks(self, plan: FunctionPlan) -> tuple[LifecycleActionPlan, ...]: + """Return copy-out writebacks ordered by their completed result positions.""" + actions = ( + action + for action in plan.writeback_actions + if action.phase is WritebackPhase.COPY_OUT and action.binding is not None + ) + return tuple(sorted(actions, key=lambda action: action.binding.result_position)) + + def _python_result_names(self, output_owners: tuple[tuple[str, int], ...]) -> dict[str, str]: + """Name one Python result local per ordered output owner.""" + single_output = len(output_owners) == 1 + return { + owner_path: ("result_obj" if single_output else f"result_{position}_obj") + for owner_path, position in output_owners } - python_result = "result_obj" if ordered_results or plan.writeback_actions else None - native_result = "result" if self._direct_result(plan) is not None else None - return _CFunctionContext(arguments, native_outputs, native_result, python_result, python_results) + + def _python_result_name(self, plan: FunctionPlan) -> str | None: + """Return the aggregate Python result local only when output exists.""" + return "result_obj" if plan.results or plan.writeback_actions else None + + def _native_result_name(self, plan: FunctionPlan) -> str | None: + """Return the direct native result local only for native functions.""" + return "result" if self._direct_result(plan) is not None else None + + def _argument_role_values( + self, + plan: FunctionPlan, + arguments: dict[str, _CArgumentNames], + ) -> dict[str, str]: + """Map completed handoff roles to their binding value locals.""" + return {argument.binding.handoff_role: arguments[argument.owner_path].value_name for argument in plan.arguments} + + def _argument_context_names(self, argument: ArgumentTransferPlan) -> _CArgumentNames: + """Name one argument's binding locals from its public Python name.""" + name = argument.binding.python_name.lower() + rank = ( + 15 + if argument.array is not None and argument.array.rank is None + else (argument.array.rank if argument.array is not None else 0) + ) + return _CArgumentNames( + f"{name}_obj", + name, + f"{name}_length", + f"{name}_nullable", + f"{name}_present", + tuple(f"{name}_extent_{axis}" for axis in range(rank)), + tuple(f"{name}_upper_bound_{axis}" for axis in range(rank)), + tuple(f"{name}_stride_{axis}" for axis in range(rank)), + f"{name}_rank", + f"{name}_itemsize", + ) def _keyword_declaration(self, plan: FunctionPlan) -> CDeclaration: keywords = ", ".join( @@ -1000,6 +2007,8 @@ def _direct_result_declaration( result = self._direct_result(plan) if result is None or context.result_name is None: return () + if result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: + return (CDeclaration(context.result_name, "void *", CodeExpression("NULL")),) scalar_type = PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) return (CDeclaration(context.result_name, scalar_type.c_spelling),) @@ -1014,7 +2023,7 @@ def _native_output_declarations( if slot.source_kind != "result": continue name = context.native_outputs[slot.symbolic_role] - if slot.datatype_family is DatatypeFamily.STRING: + if slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: declarations.append(CDeclaration(name, "void *", CodeExpression("NULL"))) continue if slot.semantic_type_name is None: @@ -1027,8 +2036,7 @@ def _bridge_call(self, plan: FunctionPlan, context: _CFunctionContext) -> str: arguments = [] for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position): names = context.arguments[argument.owner_path] - value = self._bridge_call_argument(argument, names) - arguments.append(value) + arguments.extend(self._bridge_call_arguments(argument, names)) if argument.bridge.optional_mode is OptionalMode.DESCRIPTOR: arguments.append(names.present_name) arguments.extend( @@ -1038,15 +2046,57 @@ def _bridge_call(self, plan: FunctionPlan, context: _CFunctionContext) -> str: ) return f"{self._bridge_function_name(plan)}({', '.join(arguments)})" - def _bridge_call_argument(self, plan: ArgumentTransferPlan, names: _CArgumentNames) -> str: - """Return one binding-to-bridge C argument expression.""" + def _bridge_call_arguments(self, plan: ArgumentTransferPlan, names: _CArgumentNames) -> tuple[str, ...]: + """Return one binding-to-bridge C handoff, including helper ABI fields.""" + if plan.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: + return self._string_bridge_call_arguments(names) + if plan.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + return self._array_bridge_call_arguments(plan, names) + return self._scalar_bridge_call_arguments(plan, names) + + # Scalar bridge call arguments. + def _scalar_bridge_call_arguments( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + ) -> tuple[str, ...]: + """Return one scalar value, storage, address, or optional handoff.""" if plan.bridge.optional_mode is not OptionalMode.REQUIRED: - return names.nullable_name + return (names.nullable_name,) if plan.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS: - return names.value_name + return (names.value_name,) if plan.bridge.handoff_mode is ArgumentHandoffMode.TYPED_REFERENCE: - return f"&{names.value_name}" - return names.value_name + return (f"&{names.value_name}",) + return (names.value_name,) + + # String bridge call arguments. + def _string_bridge_call_arguments(self, names: _CArgumentNames) -> tuple[str, ...]: + """Return one scalar string pointer-and-length handoff.""" + return names.value_name, f"(int64_t){names.length_name}" + + # Ordinary-array bridge call arguments. + def _array_bridge_call_arguments( + self, + plan: ArgumentTransferPlan, + names: _CArgumentNames, + ) -> tuple[str, ...]: + """Return one completed ordinary-array C ABI field sequence.""" + handoff = plan.array + if handoff is None: + raise ValueError(f"Array argument {plan.owner_path!r} has no handoff spec") + arguments = [names.value_name] + if handoff.runtime_rank_role is not None: + arguments.append(names.runtime_rank_name) + if handoff.itemsize_role is not None: + arguments.append(names.itemsize_name) + arguments.extend(names.extent_names) + arguments.extend(self._selected_array_axis_names(names.upper_bound_names, handoff.upper_bound_roles)) + arguments.extend(self._selected_array_axis_names(names.stride_names, handoff.stride_roles)) + return tuple(arguments) + + def _selected_array_axis_names(self, names: tuple[str, ...], roles: tuple[str, ...]) -> tuple[str, ...]: + """Return array ABI local names only when the plan carries their roles.""" + return names if roles else () def _bridge_prototype(self, plan: FunctionPlan) -> CFunctionPrototype: argument_parameters = tuple( @@ -1070,6 +2120,8 @@ def _bridge_return_type(self, plan: FunctionPlan) -> str: result = self._direct_result(plan) if result is None: return "void" + if result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: + return "void *" return PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name).c_spelling def _direct_result(self, plan: FunctionPlan) -> ResultPlan | None: @@ -1079,13 +2131,18 @@ def _direct_result(self, plan: FunctionPlan) -> ResultPlan | None: def _bridge_argument_parameters(self, argument: ArgumentTransferPlan) -> tuple[CParameter, ...]: """Return the bridge ABI parameters for one Python argument.""" name = argument.bridge.native_name.lower() - scalar_type = self._bridge_argument_type(argument) + if argument.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: + return self._string_bridge_argument_parameters(argument, name) + if argument.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + return self._array_bridge_argument_parameters(argument, name) + scalar_type = self._scalar_bridge_argument_type(argument) if argument.bridge.optional_mode is OptionalMode.DESCRIPTOR: return (CParameter(name, scalar_type), CParameter(f"{name}_present", "void *")) return (CParameter(name, scalar_type),) - def _bridge_argument_type(self, argument: ArgumentTransferPlan) -> str: - """Return the C ABI type for one bridge input.""" + # Scalar bridge ABI parameters. + def _scalar_bridge_argument_type(self, argument: ArgumentTransferPlan) -> str: + """Return the C ABI type for one scalar bridge input.""" if argument.bridge.optional_mode is not OptionalMode.REQUIRED: return "void *" if argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS: @@ -1095,11 +2152,45 @@ def _bridge_argument_type(self, argument: ArgumentTransferPlan) -> str: return f"{scalar_type} *" return scalar_type + # String bridge ABI parameters. + def _string_bridge_argument_parameters( + self, + argument: ArgumentTransferPlan, + name: str, + ) -> tuple[CParameter, ...]: + """Return one scalar string pointer-and-length ABI pair.""" + pointer_type = "char *" if argument.bridge.codegen_action is CodegenAction.COPY_IN_OUT else "const char *" + return CParameter(name, pointer_type), CParameter(f"{name}_length", "int64_t") + + # Ordinary-array bridge ABI parameters. + def _array_bridge_argument_parameters( + self, + argument: ArgumentTransferPlan, + name: str, + ) -> tuple[CParameter, ...]: + """Return the completed ordinary-array bridge ABI parameters.""" + handoff = argument.array + if handoff is None: + raise ValueError(f"Array argument {argument.owner_path!r} has no handoff spec") + parameters = [CParameter(name, "void *")] + if handoff.runtime_rank_role is not None: + parameters.append(CParameter(f"{name}_rank", "int64_t")) + if handoff.itemsize_role is not None: + parameters.append(CParameter(f"{name}_itemsize", "int64_t")) + parameters.extend(self._array_bridge_axis_parameters(name, "extent", len(handoff.extent_roles))) + parameters.extend(self._array_bridge_axis_parameters(name, "upper_bound", len(handoff.upper_bound_roles))) + parameters.extend(self._array_bridge_axis_parameters(name, "stride", len(handoff.stride_roles))) + return tuple(parameters) + + def _array_bridge_axis_parameters(self, name: str, label: str, count: int) -> tuple[CParameter, ...]: + """Return one named int64 bridge parameter per ordinary-array axis.""" + return tuple(CParameter(f"{name}_{label}_{axis}", "int64_t") for axis in range(count)) + def _bridge_result_parameters(self, slot: NativeCallSlotPlan) -> tuple[CParameter, ...]: """Return the C ABI parameter for one native result slot.""" if slot.source_kind != "result": return () - if slot.datatype_family is DatatypeFamily.STRING: + if slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: return (CParameter(slot.native_name.lower(), "void **"),) if slot.semantic_type_name is None: raise ValueError(f"Missing bridge result datatype for {slot.owner_path!r}") @@ -1357,6 +2448,7 @@ def _module_literal(self, plan: ModuleVariablePlan, value: object) -> str: return self._lower_module_literal_complex(value) raise ValueError(f"Unsupported C module literal family for {plan.owner_path!r}: {family!r}") + # Scalar module-literal lowering. def _lower_module_literal_bool(self, value: object) -> str: return "true" if value else "false" diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index ab6680027..c549c6f18 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -2,7 +2,15 @@ from __future__ import annotations -from x2py.semantics.ownership import AssignmentMode, CodegenAction, NativeBarrierAction +import re + +from x2py.semantics.ownership import ( + AssignmentMode, + CodegenAction, + NativeBarrierAction, + ObjectKind, + PythonBarrierAction, +) from x2py.semantics.wrapper_policy import ( ArgumentHandoffMode, BridgeDataAction, @@ -13,6 +21,7 @@ CodeExpression, FortranAssignment, FortranCall, + FortranCase, FortranDeclaration, FortranFunction, FortranIf, @@ -21,6 +30,7 @@ FortranModule, FortranParameter, FortranPointerAssignment, + FortranSelectCase, FortranUse, ) from x2py.wrapper_codegen.plan import ( @@ -55,6 +65,18 @@ def _require_function_supported(self, function: FunctionPlan) -> None: raise ValueError(f"{function.owner_path!r} mixes optional scalar arguments with hidden literals") for argument in function.arguments: self._require_argument_supported(argument) + for result in function.results: + match result.object_kind: + case ObjectKind.SCALAR: + PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) + case ObjectKind.STRING: + self._require_string_plan_result_supported(result) + case ObjectKind.NUMPY_ARRAY: + self._require_array_plan_result_supported(result) + case _: + raise ValueError( + f"Unsupported Fortran result object kind for {result.owner_path!r}: {result.object_kind!r}" + ) for slot in function.native_call_slots: self._require_native_result_supported(function, slot) @@ -65,6 +87,7 @@ def _require_argument_supported(self, argument: ArgumentTransferPlan) -> None: NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, NativeBarrierAction.PASS_RAW_ADDRESS, NativeBarrierAction.PASS_STORAGE_ADDRESS, + NativeBarrierAction.PASS_ARRAY_BUFFER, } if argument.bridge.native_action not in supported: raise ValueError( @@ -77,30 +100,156 @@ def _require_argument_supported(self, argument: ArgumentTransferPlan) -> None: raise ValueError(f"Unsupported Fortran raw-address handoff for {argument.owner_path!r}") if argument.bridge.data_action is BridgeDataAction.BLOCKED: raise ValueError(f"Blocked Fortran bridge data action for {argument.owner_path!r}") - PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + match argument.object_kind: + case ObjectKind.SCALAR: + self._require_scalar_argument_supported(argument) + case ObjectKind.STRING: + self._require_string_argument_supported(argument) + case ObjectKind.NUMPY_ARRAY: + self._require_array_argument_supported(argument) + case _: + raise ValueError( + f"Unsupported Fortran argument object kind for {argument.owner_path!r}: {argument.object_kind!r}" + ) def _require_native_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: - """Reject one unsupported native result output.""" + """Dispatch one native result output to its family support check.""" if slot.source_kind != "result": return + match slot.object_kind: + case ObjectKind.SCALAR: + self._require_scalar_native_result_supported(slot) + case ObjectKind.STRING: + self._require_string_result_supported(function, slot) + case ObjectKind.NUMPY_ARRAY: + self._require_array_result_supported(function, slot) + case _: + raise ValueError( + f"Unsupported Fortran native result object kind for {slot.owner_path!r}: {slot.object_kind!r}" + ) + + # Scalar support checks. + def _require_scalar_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one first-lane primitive scalar argument type.""" + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_scalar_native_result_supported(self, slot: NativeCallSlotPlan) -> None: + """Require one first-lane primitive scalar native result type.""" + if slot.semantic_type_name is None: + raise ValueError(f"Missing Fortran result datatype for {slot.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + + # Ordinary-array support checks. + def _require_array_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one completed ordinary array view.""" + array = argument.array + if array is None or (array.rank is not None and not 1 <= array.rank <= 15): + raise ValueError(f"Unsupported Fortran array rank for {argument.owner_path!r}") + if array.contiguous not in {True, False}: + raise ValueError(f"Unsupported Fortran array layout for {argument.owner_path!r}") + if argument.bridge.handoff_mode is not ArgumentHandoffMode.ARRAY_BUFFER: + raise ValueError(f"Unsupported Fortran array handoff for {argument.owner_path!r}") + if argument.bridge.data_action is not BridgeDataAction.ASSOCIATE_VIEW: + raise ValueError(f"Unsupported Fortran array data action for {argument.owner_path!r}") + if argument.datatype_family is not DatatypeFamily.STRING: + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) + + def _require_array_plan_result_supported(self, result: ResultPlan) -> None: + """Require one fixed-shape ordinary array copy result.""" + array = result.array + if array is None or array.rank is None or not 1 <= array.rank <= 15: + raise ValueError(f"Unsupported Fortran array result rank for {result.owner_path!r}") + if array.order == "ORDER_C" and array.rank > 1: + raise ValueError(f"Unsupported Fortran array result order for {result.owner_path!r}") + if result.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported Fortran array result data action for {result.owner_path!r}") + if result.datatype_family is DatatypeFamily.STRING: + if array.itemsize is None or array.itemsize <= 0: + raise ValueError(f"Unsupported Fortran character array result for {result.owner_path!r}") + return + PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name) + + def _require_array_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Require one hidden fixed-shape ordinary array copy result.""" + self._require_array_result_shape_supported(slot) + self._require_array_result_action_supported(slot) + self._require_array_result_type_supported(function, slot) + + def _require_array_result_shape_supported(self, slot: NativeCallSlotPlan) -> None: + """Require one fixed-rank non-C-oriented array result shape.""" + array = slot.array + if array is None or array.rank is None or not 1 <= array.rank <= 15: + raise ValueError(f"Unsupported Fortran array output rank for {slot.owner_path!r}") + if array.order == "ORDER_C" and array.rank > 1: + raise ValueError(f"Unsupported Fortran array output order for {slot.owner_path!r}") + + def _require_array_result_action_supported(self, slot: NativeCallSlotPlan) -> None: + """Require the completed ordinary-array representation-copy action.""" + if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported Fortran array output data action for {slot.owner_path!r}") + + def _require_array_result_type_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: + """Require one primitive non-character result owned by the function plan.""" + if not any(result.native_call_slot is slot for result in function.results): + raise ValueError(f"Unsupported Fortran array output for {slot.owner_path!r}") if slot.datatype_family is DatatypeFamily.STRING: - self._require_string_result_supported(function, slot) + if slot.array is None or slot.array.itemsize is None or slot.array.itemsize <= 0: + raise ValueError(f"Unsupported Fortran character array output for {slot.owner_path!r}") return if slot.semantic_type_name is None: - raise ValueError(f"Missing Fortran result datatype for {slot.owner_path!r}") + raise ValueError(f"Missing Fortran array output datatype for {slot.owner_path!r}") PrimitiveScalarTypeRegistry.type_for(slot.semantic_type_name) + # String support checks. + def _require_string_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require a completed string value, storage, or raw-address contract.""" + if argument.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported Fortran string data action for {argument.owner_path!r}") + action = argument.binding.python_action + if action is PythonBarrierAction.STRING_VALUE: + self._require_string_value_argument_supported(argument) + return + if action not in {PythonBarrierAction.STRING_STORAGE, PythonBarrierAction.RAW_ADDRESS}: + raise ValueError(f"Unsupported Fortran string boundary for {argument.owner_path!r}: {action!r}") + self._require_string_address_argument_supported(argument) + + def _require_string_value_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one character-buffer value handoff.""" + if argument.bridge.native_action is not NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: + raise ValueError(f"Unsupported Fortran string action for {argument.owner_path!r}") + if argument.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + raise ValueError(f"Unsupported Fortran string handoff for {argument.owner_path!r}") + if argument.bridge.codegen_action not in {CodegenAction.CALL_LOCAL_INPUT, CodegenAction.COPY_IN_OUT}: + raise ValueError(f"Unsupported Fortran string codegen action for {argument.owner_path!r}") + + def _require_string_address_argument_supported(self, argument: ArgumentTransferPlan) -> None: + """Require one fixed storage/raw-address handoff.""" + if argument.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS: + raise ValueError(f"Unsupported Fortran string address handoff for {argument.owner_path!r}") + if argument.bridge.codegen_action is not CodegenAction.IN_PLACE_ARGUMENT: + raise ValueError(f"Unsupported Fortran string address action for {argument.owner_path!r}") + if argument.character_length is None or argument.character_length <= 0: + raise ValueError(f"Unsupported Fortran string address length for {argument.owner_path!r}") + def _require_string_result_supported(self, function: FunctionPlan, slot: NativeCallSlotPlan) -> None: - """Require one fixed-length status-message result.""" - policy = function.binding.status_error - if policy is None: - raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") - if policy.message_role != slot.symbolic_role: - raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") - if slot.character_length is None: + """Require one fixed string result slot or status-message slot.""" + if slot.character_length is None or slot.character_length <= 0: raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") if slot.bridge_data_action is not BridgeDataAction.COPY_REPRESENTATION: raise ValueError(f"Unsupported Fortran string bridge data action for {slot.owner_path!r}") + policy = function.binding.status_error + if policy is not None and policy.message_role == slot.symbolic_role: + return + if any(result.native_call_slot is slot for result in function.results): + return + raise ValueError(f"Unsupported Fortran string output for {slot.owner_path!r}") + + def _require_string_plan_result_supported(self, result: ResultPlan) -> None: + """Require one fixed string result with a justified representation copy.""" + if result.character_length is None or result.character_length <= 0: + raise ValueError(f"Unsupported Fortran string result for {result.owner_path!r}") + if result.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"Unsupported Fortran string result data action for {result.owner_path!r}") def _require_variable_supported(self, variable: ModuleVariablePlan) -> None: """Reject unsupported actions in one planned module variable.""" @@ -161,12 +310,22 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: declarations=( *self._optional_declarations(plan), *self._opaque_address_declarations(plan), + *self._array_declarations(plan), + *self._string_value_declarations(plan), + *self._string_address_declarations(plan), + *self._direct_result_declarations(plan), *self._native_output_declarations(plan), ), body=( *self._descriptor_initializers(plan), *self._opaque_address_initializers(plan), + *self._array_initializers(plan), + *self._string_value_initializers(plan), + *self._string_address_initializers(plan), *self._function_body(plan, result_name), + *self._string_value_finalizers(plan), + *self._string_address_finalizers(plan), + *self._direct_result_finalizers(plan), *self._native_output_finalizers(plan), ), is_subroutine=is_subroutine, @@ -181,10 +340,35 @@ def _lower_result( if result is None: return self._lower_result_none(plan) action = result.bridge.codegen_action - match action: - case CodegenAction.DIRECT_VALUE: + match result.object_kind: + case ObjectKind.NUMPY_ARRAY if action is CodegenAction.COPY_OUT: + return self._lower_result_array_copy(plan, result) + case ObjectKind.STRING if action is CodegenAction.COPY_OUT: + return self._lower_result_fixed_string(plan, result) + case ObjectKind.SCALAR if action is CodegenAction.DIRECT_VALUE: return self._lower_result_direct_value(plan, result) - raise ValueError(f"Unsupported Fortran result action for {plan.owner_path!r}: {action!r}") + case _: + raise ValueError( + f"Unsupported Fortran result selection for {plan.owner_path!r}: {result.object_kind!r}:{action!r}" + ) + + # String result lowering. + def _lower_result_fixed_string( + self, + _plan: FunctionPlan, + _result: ResultPlan, + ) -> tuple[str | None, str | None]: + """Return the C-pointer bridge shape for one copied fixed string.""" + return "result", "type(c_ptr)" + + # Ordinary-array result lowering. + def _lower_result_array_copy( + self, + _plan: FunctionPlan, + _result: ResultPlan, + ) -> tuple[str | None, str | None]: + """Return the C-pointer bridge shape for one copied ordinary array.""" + return "result", "type(c_ptr)" def _lower_result_none( self, @@ -193,6 +377,7 @@ def _lower_result_none( """Return the procedure shape of a native subroutine with no projection.""" return None, None + # Scalar result lowering. def _lower_result_direct_value( self, plan: FunctionPlan, @@ -324,6 +509,16 @@ def _visit_ArgumentTransferPlan(self, plan: ArgumentTransferPlan) -> tuple[Fortr def _lower_argument(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Dispatch one completed bridge optional mode explicitly.""" mode = plan.bridge.optional_mode + if plan.object_kind is ObjectKind.NUMPY_ARRAY: + if mode not in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE}: + raise ValueError(f"Unsupported Fortran array presence mode for {plan.owner_path!r}: {mode!r}") + return self._lower_argument_array_buffer(plan) + if plan.object_kind is ObjectKind.STRING and plan.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: + if mode not in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE}: + raise ValueError(f"Unsupported Fortran string presence mode for {plan.owner_path!r}: {mode!r}") + return self._lower_argument_string_value(plan) + if plan.object_kind not in {ObjectKind.SCALAR, ObjectKind.STRING}: + raise ValueError(f"Unsupported Fortran argument object kind for {plan.owner_path!r}: {plan.object_kind!r}") match mode: case OptionalMode.REQUIRED: return self._lower_argument_required(plan) @@ -343,8 +538,11 @@ def _lower_argument_required(self, plan: ArgumentTransferPlan) -> tuple[FortranP return self._lower_argument_required_typed_reference(plan) case ArgumentHandoffMode.OPAQUE_ADDRESS: return self._lower_argument_required_opaque_address(plan) + case ArgumentHandoffMode.CHARACTER_BUFFER: + return self._lower_argument_string_value(plan) raise ValueError(f"Unsupported Fortran argument handoff for {plan.owner_path!r}: {mode!r}") + # Scalar argument lowering. def _lower_argument_required_value(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Return one interoperable scalar value parameter.""" return (self._parameter(plan, ("value",)),) @@ -364,6 +562,54 @@ def _lower_argument_required_opaque_address( name = plan.bridge.native_name.lower() return (FortranParameter(f"bound_{name}", "type(c_ptr)", ("value",)),) + # String argument lowering. + def _lower_argument_string_value( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranParameter, ...]: + """Receive one C UTF-8 payload address and its runtime byte length.""" + name = plan.bridge.native_name.lower() + return ( + FortranParameter(f"bound_{name}", "type(c_ptr)", ("value",)), + FortranParameter(f"{name}_length", "integer(c_int64_t)", ("value",)), + ) + + # Ordinary-array argument lowering. + def _lower_argument_array_buffer( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranParameter, ...]: + """Receive exactly the ordinary-array ABI fields named by the plan.""" + array = plan.array + if array is None: + raise ValueError(f"Array argument {plan.owner_path!r} has no handoff spec") + name = plan.bridge.native_name.lower() + return ( + FortranParameter(f"bound_{name}", "type(c_ptr)", ("value",)), + *( + (FortranParameter(f"{name}_rank", "integer(c_int64_t)", ("value",)),) + if array.runtime_rank_role is not None + else () + ), + *( + (FortranParameter(f"{name}_itemsize", "integer(c_int64_t)", ("value",)),) + if array.itemsize_role is not None + else () + ), + *( + FortranParameter(f"{name}_extent_{axis}", "integer(c_int64_t)", ("value",)) + for axis in range(len(array.extent_roles)) + ), + *( + FortranParameter(f"{name}_upper_bound_{axis}", "integer(c_int64_t)", ("value",)) + for axis in range(len(array.upper_bound_roles)) + ), + *( + FortranParameter(f"{name}_stride_{axis}", "integer(c_int64_t)", ("value",)) + for axis in range(len(array.stride_roles)) + ), + ) + def _lower_argument_nullable_value(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Return one nullable C pointer parameter.""" name = plan.bridge.native_name.lower() @@ -385,15 +631,23 @@ def _function_body( self, plan: FunctionPlan, result_name: str | None, - ) -> tuple[FortranAssignment | FortranCall | FortranIf, ...]: + ) -> tuple[FortranAssignment | FortranCall | FortranIf | FortranSelectCase, ...]: + result_name = self._native_direct_result_name(plan, result_name) + assumed_rank = tuple( + argument + for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) + if argument.array is not None and argument.array.rank is None + ) + if assumed_rank: + return (self._assumed_rank_call_tree(plan, assumed_rank, 0, {}, result_name),) optional = tuple( argument for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) if argument.bridge.optional_mode is not OptionalMode.REQUIRED ) if not optional: - return (self._native_invocation(plan, frozenset(), result_name),) - return (self._optional_call_tree(plan, optional, 0, frozenset(), result_name),) + return (self._native_invocation(plan, frozenset(), result_name, {}),) + return (self._optional_call_tree(plan, optional, 0, frozenset(), result_name, {}),) def _optional_call_tree( self, @@ -402,19 +656,20 @@ def _optional_call_tree( index: int, present: frozenset[str], result_name: str | None, + replacements: dict[str, str], ) -> FortranAssignment | FortranCall | FortranIf: """Return an exhaustive native-call tree for optional presence states.""" if index == len(optional): - return self._native_invocation(plan, present, result_name) + return self._native_invocation(plan, present, result_name, replacements) argument = optional[index] present_roles = present | {argument.owner_path} return FortranIf( condition=CodeExpression(self._presence_condition(argument)), body=( *self._present_preparation(argument), - self._optional_call_tree(plan, optional, index + 1, present_roles, result_name), + self._optional_call_tree(plan, optional, index + 1, present_roles, result_name, replacements), ), - else_body=(self._optional_call_tree(plan, optional, index + 1, present, result_name),), + else_body=(self._optional_call_tree(plan, optional, index + 1, present, result_name, replacements),), ) def _native_invocation( @@ -422,8 +677,9 @@ def _native_invocation( plan: FunctionPlan, present: frozenset[str], result_name: str | None, + replacements: dict[str, str], ) -> FortranAssignment | FortranCall: - arguments = self._native_arguments(plan, present) + arguments = self._native_arguments(plan, present, replacements) native_name = self._native_function_name(plan) if plan.bridge.native_is_subroutine: return FortranCall(native_name, arguments) @@ -436,8 +692,9 @@ def _native_arguments( self, plan: FunctionPlan, present: frozenset[str], + replacements: dict[str, str], ) -> tuple[CodeExpression, ...]: - expressions = dict(self._visible_native_argument_entries(plan, present)) + expressions = dict(self._visible_native_argument_entries(plan, present, replacements)) expressions.update( (slot.native_position, CodeExpression(self._literal_expression(slot.literal_value))) for slot in plan.native_call_slots @@ -454,6 +711,7 @@ def _visible_native_argument_entries( self, plan: FunctionPlan, present: frozenset[str], + replacements: dict[str, str], ) -> tuple[tuple[int, CodeExpression], ...]: """Return native-position entries for present Python arguments.""" entries = [] @@ -461,12 +719,50 @@ def _visible_native_argument_entries( for argument in plan.arguments: if argument.bridge.optional_mode is not OptionalMode.REQUIRED and argument.owner_path not in present: continue - expression = self._native_argument_expression(argument) + expression = replacements.get(argument.owner_path, self._native_argument_expression(argument)) if has_optional: expression = f"{argument.bridge.native_name}={expression}" entries.append((argument.native_call_slot.native_position, CodeExpression(expression))) return tuple(entries) + def _assumed_rank_call_tree( + self, + plan: FunctionPlan, + arguments: tuple[ArgumentTransferPlan, ...], + index: int, + replacements: dict[str, str], + result_name: str | None, + ) -> FortranAssignment | FortranCall | FortranIf | FortranSelectCase: + """Dispatch each runtime-rank array through explicit one-to-fifteen branches.""" + if index == len(arguments): + optional = tuple( + argument + for argument in sorted(plan.arguments, key=lambda item: item.bridge.abi_position) + if argument.bridge.optional_mode is not OptionalMode.REQUIRED + ) + if optional: + return self._optional_call_tree(plan, optional, 0, frozenset(), result_name, replacements) + return self._native_invocation(plan, frozenset(), result_name, replacements) + argument = arguments[index] + name = argument.bridge.native_name.lower() + cases = [] + for rank in range(1, 16): + rank_name = f"{name}_rank_{rank}" + replacements[argument.owner_path] = rank_name + nested = self._assumed_rank_call_tree(plan, arguments, index + 1, replacements, result_name) + del replacements[argument.owner_path] + cases.append( + FortranCase( + rank, + ( + self._assumed_rank_pointer_initializer(argument, rank, rank_name), + nested, + ), + ) + ) + cases.append(FortranCase(None, ())) + return FortranSelectCase(CodeExpression(f"{name}_rank"), tuple(cases)) + def _hidden_native_result_entries( self, plan: FunctionPlan, @@ -484,6 +780,8 @@ def _hidden_native_result_entries( def _native_argument_expression(self, plan: ArgumentTransferPlan) -> str: name = plan.bridge.native_name.lower() + if plan.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + return self._array_native_argument_expression(plan) if plan.bridge.optional_mode is OptionalMode.DESCRIPTOR: return f"{name}_descriptor" return name @@ -512,6 +810,8 @@ def _prepare_present_associated_view( ) -> tuple[FortranPointerAssignment | FortranCall | FortranIf, ...]: """Associate a non-owning native view without copying payload data.""" name = plan.bridge.native_name.lower() + if plan.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + return (self._array_pointer_initializer(plan),) if plan.bridge.optional_mode is OptionalMode.NULLABLE_VALUE: return ( FortranCall( @@ -534,8 +834,10 @@ def _prepare_present_associated_view( def _prepare_present_representation_copy( self, plan: ArgumentTransferPlan, - ) -> tuple[FortranCall | FortranIf, ...]: + ) -> tuple[FortranCall | FortranAssignment | FortranIf, ...]: """Copy only when completed policy requires a different native representation.""" + if plan.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: + return self._string_value_initializer_nodes(plan) if plan.bridge.optional_mode is not OptionalMode.DESCRIPTOR: raise ValueError(f"Representation copy requires descriptor policy: {plan.owner_path!r}") if plan.native_call_slot.value_kind != "allocatable": @@ -562,6 +864,10 @@ def _optional_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration mode = argument.bridge.optional_mode if mode is OptionalMode.REQUIRED: continue + if argument.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: + continue + if argument.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + continue name = argument.bridge.native_name.lower() scalar_type = PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name) if mode is OptionalMode.NULLABLE_VALUE: @@ -587,6 +893,7 @@ def _opaque_address_declarations(self, plan: FunctionPlan) -> tuple[FortranDecla argument.bridge.optional_mode is OptionalMode.REQUIRED and argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS and argument.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + and argument.object_kind is ObjectKind.SCALAR ) ) @@ -605,9 +912,301 @@ def _opaque_address_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, argument.bridge.optional_mode is OptionalMode.REQUIRED and argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS and argument.bridge.data_action is BridgeDataAction.ASSOCIATE_VIEW + and argument.object_kind is ObjectKind.SCALAR ) ) + # Ordinary-array bridge storage. + def _array_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Declare typed pointer views for ordinary array buffers.""" + declarations = [] + for argument in plan.arguments: + if argument.bridge.handoff_mode is not ArgumentHandoffMode.ARRAY_BUFFER: + continue + array = argument.array + if array is None: + raise ValueError(f"Array argument {argument.owner_path!r} is missing its handoff") + if array.rank is None: + declarations.extend(self._assumed_rank_array_declarations(argument)) + continue + declarations.append( + FortranDeclaration( + self._array_pointer_name(argument), + self._array_element_fortran_type(argument), + ("pointer", self._array_dimension_attribute(array.rank)), + ) + ) + return tuple(declarations) + + def _array_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, ...]: + """Associate each completed ordinary array data/extent handoff.""" + initializers = [] + for argument in plan.arguments: + if argument.bridge.handoff_mode is not ArgumentHandoffMode.ARRAY_BUFFER: + continue + if argument.bridge.optional_mode is not OptionalMode.REQUIRED: + continue + if argument.array is not None and argument.array.rank is None: + continue + initializers.append(self._array_pointer_initializer(argument)) + return tuple(initializers) + + def _array_pointer_initializer(self, argument: ArgumentTransferPlan) -> FortranCall: + """Associate one fixed-rank array pointer using planned base extents.""" + array = argument.array + if array is None or array.rank is None: + raise ValueError(f"Array argument {argument.owner_path!r} requires a concrete rank") + name = argument.bridge.native_name.lower() + extents = [f"{name}_extent_{axis}" for axis in range(array.rank)] + if array.order == "ORDER_C": + extents.reverse() + return FortranCall( + "c_f_pointer", + ( + CodeExpression(f"bound_{name}"), + CodeExpression(self._array_pointer_name(argument)), + CodeExpression(f"[{', '.join(extents)}]"), + ), + ) + + def _assumed_rank_array_declarations( + self, + argument: ArgumentTransferPlan, + ) -> tuple[FortranDeclaration, ...]: + """Declare one readable typed pointer local for every supported runtime rank.""" + name = argument.bridge.native_name.lower() + element_type = self._array_element_fortran_type(argument) + return tuple( + FortranDeclaration( + f"{name}_rank_{rank}", + element_type, + ("pointer", self._array_dimension_attribute(rank)), + ) + for rank in range(1, 16) + ) + + def _assumed_rank_pointer_initializer( + self, + argument: ArgumentTransferPlan, + rank: int, + pointer_name: str, + ) -> FortranCall: + """Associate one runtime-rank branch with its planned extent prefix.""" + name = argument.bridge.native_name.lower() + extents = ", ".join(f"{name}_extent_{axis}" for axis in range(rank)) + return FortranCall( + "c_f_pointer", + ( + CodeExpression(f"bound_{name}"), + CodeExpression(pointer_name), + CodeExpression(f"[{extents}]"), + ), + ) + + def _array_pointer_name(self, argument: ArgumentTransferPlan) -> str: + """Name the bridge pointer, separating strided base storage visibly.""" + name = argument.bridge.native_name.lower() + return f"{name}_base" if argument.array is not None and argument.array.contiguous is False else name + + def _array_native_argument_expression(self, argument: ArgumentTransferPlan) -> str: + """Pass a dense pointer or the explicitly planned positive-stride slice.""" + array = argument.array + if array is None: + raise ValueError(f"Array argument {argument.owner_path!r} has no handoff spec") + name = argument.bridge.native_name.lower() + if array.rank is None: + return name + pointer_name = self._array_pointer_name(argument) + if array.contiguous is not False: + return pointer_name + slices = (f"1:{name}_upper_bound_{axis} + 1:{name}_stride_{axis}" for axis in range(array.rank)) + return f"{pointer_name}({', '.join(slices)})" + + def _array_element_fortran_type(self, argument: ArgumentTransferPlan) -> str: + """Return the completed primitive or fixed-width character element type.""" + array = argument.array + if argument.datatype_family is DatatypeFamily.STRING: + if array is None or array.itemsize is None or array.itemsize <= 0: + raise ValueError(f"Character array {argument.owner_path!r} has no fixed itemsize") + return f"character(kind=c_char, len={array.itemsize})" + return PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).fortran_spelling + + def _array_dimension_attribute(self, rank: int) -> str: + """Spell one explicit-rank deferred-shape pointer attribute.""" + return f"dimension({', '.join(':' for _ in range(rank))})" + + # String address bridge storage. + def _string_address_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Declare fixed helper-local character storage for address boundaries.""" + declarations = [] + for argument in self._string_address_arguments(plan): + name = argument.bridge.native_name.lower() + length = self._string_address_length(argument) + declarations.extend( + ( + FortranDeclaration( + f"{name}_bytes", + "character(kind=c_char)", + ("pointer", "dimension(:)"), + ), + FortranDeclaration(name, f"character(kind=c_char, len={length})"), + ) + ) + return tuple(declarations) + + def _string_address_initializers( + self, + plan: FunctionPlan, + ) -> tuple[FortranCall | FortranAssignment, ...]: + """Associate fixed-width bytes and materialize native character locals.""" + nodes = [] + for argument in self._string_address_arguments(plan): + name = argument.bridge.native_name.lower() + length = self._string_address_length(argument) + nodes.extend( + ( + FortranCall( + "c_f_pointer", + ( + CodeExpression(f"bound_{name}"), + CodeExpression(f"{name}_bytes"), + CodeExpression(f"[{length}]"), + ), + ), + FortranAssignment(name, CodeExpression(f"transfer({name}_bytes, {name})")), + ) + ) + return tuple(nodes) + + def _string_address_finalizers(self, plan: FunctionPlan) -> tuple[FortranAssignment, ...]: + """Copy every mutated fixed character byte back to caller storage.""" + nodes = [] + for argument in self._string_address_arguments(plan): + if not argument.mutates_native: + continue + name = argument.bridge.native_name.lower() + length = self._string_address_length(argument) + nodes.append( + FortranAssignment( + f"{name}_bytes(1:{length})", + CodeExpression(f"transfer({name}, {name}_bytes(1:{length}))"), + ) + ) + return tuple(nodes) + + def _string_address_arguments(self, plan: FunctionPlan) -> tuple[ArgumentTransferPlan, ...]: + """Return address-shaped strings selected by completed plan facts.""" + return tuple( + argument + for argument in plan.arguments + if argument.object_kind is ObjectKind.STRING + and argument.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + and argument.bridge.data_action is BridgeDataAction.COPY_REPRESENTATION + ) + + def _string_address_length(self, plan: ArgumentTransferPlan) -> int: + """Return the fixed extent already completed in the shared plan.""" + if plan.character_length is None or plan.character_length <= 0: + raise ValueError(f"String address {plan.owner_path!r} is missing a fixed character length") + return plan.character_length + + # String value bridge storage. + def _string_value_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Return bridge-local character storage for string-value inputs.""" + declarations = [] + for argument in plan.arguments: + if argument.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + continue + name = argument.bridge.native_name.lower() + declarations.extend( + ( + FortranDeclaration( + f"{name}_bytes", + "character(kind=c_char)", + ("pointer", "dimension(:)"), + ), + FortranDeclaration(name, f"character(kind=c_char, len={name}_length)"), + ) + ) + return tuple(declarations) + + def _string_value_initializers( + self, + plan: FunctionPlan, + ) -> tuple[FortranCall | FortranAssignment, ...]: + """Associate and copy C bytes only for completed representation-copy inputs.""" + nodes = [] + for argument in plan.arguments: + if argument.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + continue + if argument.bridge.optional_mode is not OptionalMode.REQUIRED: + continue + if argument.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + raise ValueError(f"String input {argument.owner_path!r} is missing representation-copy policy") + nodes.extend(self._string_value_initializer_nodes(argument)) + return tuple(nodes) + + def _string_value_initializer_nodes( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranCall | FortranAssignment, ...]: + """Associate and materialize one present string payload.""" + name = plan.bridge.native_name.lower() + extent = f"{name}_length + 1" if plan.bridge.codegen_action is CodegenAction.COPY_IN_OUT else f"{name}_length" + source = ( + f"{name}_bytes(1:{name}_length)" + if plan.bridge.codegen_action is CodegenAction.COPY_IN_OUT + else f"{name}_bytes" + ) + return ( + FortranCall( + "c_f_pointer", + ( + CodeExpression(f"bound_{name}"), + CodeExpression(f"{name}_bytes"), + CodeExpression(f"[{extent}]"), + ), + ), + FortranAssignment(name, CodeExpression(f"transfer({source}, {name})")), + ) + + def _string_value_finalizers( + self, + plan: FunctionPlan, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Dispatch completed post-call string copyback actions.""" + nodes = [] + for argument in plan.arguments: + if argument.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + continue + action = argument.bridge.codegen_action + if action is CodegenAction.CALL_LOCAL_INPUT: + continue + if action is CodegenAction.COPY_IN_OUT: + copyback = self._lower_argument_string_copyback(argument) + if argument.bridge.optional_mode is OptionalMode.NULLABLE_VALUE: + name = argument.bridge.native_name.lower() + nodes.append(FortranIf(CodeExpression(f"c_associated(bound_{name})"), body=copyback)) + else: + nodes.extend(copyback) + continue + raise ValueError(f"Unsupported Fortran string finalizer for {argument.owner_path!r}: {action!r}") + return tuple(nodes) + + def _lower_argument_string_copyback( + self, + plan: ArgumentTransferPlan, + ) -> tuple[FortranAssignment, ...]: + """Copy one complete native character value back to binding storage.""" + name = plan.bridge.native_name.lower() + return ( + FortranAssignment( + f"{name}_bytes(1:{name}_length)", + CodeExpression(f"transfer({name}, {name}_bytes(1:{name}_length))"), + ), + FortranAssignment(f"{name}_bytes({name}_length + 1)", CodeExpression("c_null_char")), + ) + def _descriptor_initializers(self, plan: FunctionPlan) -> tuple[FortranCall, ...]: return tuple( FortranCall("nullify", (CodeExpression(f"{argument.bridge.native_name.lower()}_descriptor"),)) @@ -624,7 +1223,7 @@ def _native_output_parameters(self, plan: FunctionPlan) -> tuple[FortranParamete for slot in sorted(plan.native_call_slots, key=lambda item: item.native_position): if slot.source_kind != "result": continue - if slot.datatype_family is DatatypeFamily.STRING: + if slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: parameters.append(FortranParameter(slot.native_name.lower(), "type(c_ptr)")) continue if slot.semantic_type_name is None: @@ -642,19 +1241,84 @@ def _native_output_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclar if slot.bridge_data_action is BridgeDataAction.DIRECT_TRANSFER: continue if slot.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION: - declarations.extend(self._representation_copy_output_declarations(slot)) + declarations.extend(self._representation_copy_output_declarations(plan, slot)) continue raise ValueError( f"Unsupported native-output bridge data action for {slot.owner_path!r}: {slot.bridge_data_action!r}" ) return tuple(declarations) + def _direct_result_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Declare helper-local native storage for one copied direct result.""" + result = self._direct_result(plan) + if result is None: + return () + if result.object_kind is ObjectKind.NUMPY_ARRAY: + return self._direct_array_result_declarations(plan, result) + if result.object_kind is not ObjectKind.STRING: + return () + length = self._string_result_length(result) + return ( + FortranDeclaration("result_value", f"character(kind=c_char, len={length})"), + FortranDeclaration( + "result_copy", + "character(kind=c_char)", + ("pointer", "dimension(:)"), + ), + ) + + # Ordinary-array result storage. + def _direct_array_result_declarations( + self, + plan: FunctionPlan, + result: ResultPlan, + ) -> tuple[FortranDeclaration, ...]: + """Declare typed native and contiguous-copy storage for one array result.""" + shape = self._array_result_shape(plan, result) + element_type = self._array_result_element_type(result) + copy_type = "character(kind=c_char)" if result.datatype_family is DatatypeFamily.STRING else element_type + return ( + FortranDeclaration("result_value", element_type, (f"dimension({', '.join(shape)})",)), + FortranDeclaration("result_copy", copy_type, ("pointer", "dimension(:)")), + ) + + def _direct_result_finalizers( + self, + plan: FunctionPlan, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Copy one direct result into bridge-owned C storage.""" + result = self._direct_result(plan) + if result is None: + return () + if result.object_kind is ObjectKind.NUMPY_ARRAY: + if result.array is None: + raise ValueError(f"Array result {result.owner_path!r} has no shape plan") + return self._fixed_array_copy_nodes( + result.array.order, + result.array.rank, + itemsize=self._array_result_itemsize(result), + target_name="result", + value_name="result_value", + copy_name="result_copy", + ) + if result.object_kind is not ObjectKind.STRING: + return () + return self._fixed_string_copy_nodes( + length=self._string_result_length(result), + target_name="result", + value_name="result_value", + copy_name="result_copy", + ) + def _representation_copy_output_declarations( self, + plan: FunctionPlan, slot: NativeCallSlotPlan, ) -> tuple[FortranDeclaration, ...]: """Declare storage only for one justified representation-copy output.""" - if slot.datatype_family is not DatatypeFamily.STRING: + if slot.object_kind is ObjectKind.NUMPY_ARRAY: + return self._array_copy_output_declarations(plan, slot) + if slot.object_kind is not ObjectKind.STRING: raise ValueError(f"Unsupported representation-copy output for {slot.owner_path!r}") length = self._string_output_length(slot) value_name = self._native_output_value_name(slot) @@ -667,6 +1331,23 @@ def _representation_copy_output_declarations( ), ) + def _array_copy_output_declarations( + self, + plan: FunctionPlan, + slot: NativeCallSlotPlan, + ) -> tuple[FortranDeclaration, ...]: + """Declare typed native and contiguous-copy storage for one hidden array.""" + shape = self._array_output_shape(plan, slot) + if slot.semantic_type_name is None: + raise ValueError(f"Missing array output datatype for {slot.owner_path!r}") + element_type = self._array_result_element_type(slot) + copy_type = "character(kind=c_char)" if slot.datatype_family is DatatypeFamily.STRING else element_type + name = slot.native_name.lower() + return ( + FortranDeclaration(f"{name}_value", element_type, (f"dimension({', '.join(shape)})",)), + FortranDeclaration(f"{name}_copy", copy_type, ("pointer", "dimension(:)")), + ) + def _native_output_finalizers( self, plan: FunctionPlan, @@ -677,7 +1358,7 @@ def _native_output_finalizers( if slot.source_kind != "result" or slot.bridge_data_action is BridgeDataAction.DIRECT_TRANSFER: continue if slot.bridge_data_action is BridgeDataAction.COPY_REPRESENTATION: - nodes.extend(self._lower_native_output_representation_copy(slot)) + nodes.extend(self._lower_native_output_representation_copy(plan, slot)) continue raise ValueError( f"Unsupported native-output bridge data action for {slot.owner_path!r}: {slot.bridge_data_action!r}" @@ -686,24 +1367,159 @@ def _native_output_finalizers( def _lower_native_output_representation_copy( self, + plan: FunctionPlan, slot: NativeCallSlotPlan, ) -> tuple[FortranAssignment | FortranIf, ...]: """Copy one native output only through the explicit policy permission.""" - if slot.datatype_family is not DatatypeFamily.STRING: + if slot.object_kind is ObjectKind.NUMPY_ARRAY: + if slot.array is None: + raise ValueError(f"Array output {slot.owner_path!r} has no shape plan") + name = slot.native_name.lower() + return self._fixed_array_copy_nodes( + slot.array.order, + slot.array.rank, + itemsize=self._array_result_itemsize(slot), + target_name=name, + value_name=f"{name}_value", + copy_name=f"{name}_copy", + ) + if slot.object_kind is not ObjectKind.STRING: raise ValueError(f"Unsupported representation-copy output for {slot.owner_path!r}") name = slot.native_name.lower() value_name = self._native_output_value_name(slot) copy_name = f"{name}_copy" length = self._string_output_length(slot) + return self._fixed_string_copy_nodes( + length=length, + target_name=name, + value_name=value_name, + copy_name=copy_name, + ) + + def _fixed_array_copy_nodes( + self, + order: str | None, + rank: int | None, + *, + itemsize: int | None, + target_name: str, + value_name: str, + copy_name: str, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Allocate and fill one detached contiguous ordinary-array copy.""" + if rank is None or rank <= 0: + raise ValueError(f"Array copy {value_name!r} requires a fixed positive rank") + if order == "ORDER_C" and rank > 1: + raise ValueError(f"Array copy {value_name!r} requires Fortran element order") + if itemsize is not None: + return self._fixed_character_array_copy_nodes( + itemsize, + target_name=target_name, + value_name=value_name, + copy_name=copy_name, + ) + return ( + FortranAssignment( + target_name, + CodeExpression(f"c_malloc(max(1_c_size_t, c_sizeof({value_name})))"), + ), + FortranIf( + CodeExpression(f"c_associated({target_name})"), + body=( + FortranCall( + "c_f_pointer", + ( + CodeExpression(target_name), + CodeExpression(copy_name), + CodeExpression(f"[size({value_name})]"), + ), + ), + FortranAssignment( + copy_name, + CodeExpression(f"reshape({value_name}, [size({value_name})])"), + ), + ), + ), + ) + + def _fixed_character_array_copy_nodes( + self, + itemsize: int, + *, + target_name: str, + value_name: str, + copy_name: str, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Allocate and copy one fixed-width character array as raw bytes.""" + if itemsize <= 0: + raise ValueError(f"Character array copy {value_name!r} requires a fixed positive itemsize") + byte_count = f"{itemsize} * size({value_name})" + return ( + FortranAssignment( + target_name, + CodeExpression(f"c_malloc(max(1_c_size_t, {itemsize}_c_size_t * size({value_name}, kind=c_size_t)))"), + ), + FortranIf( + CodeExpression(f"c_associated({target_name})"), + body=( + FortranCall( + "c_f_pointer", + ( + CodeExpression(target_name), + CodeExpression(copy_name), + CodeExpression(f"[{byte_count}]"), + ), + ), + FortranAssignment( + copy_name, + CodeExpression(f"transfer({value_name}, {copy_name}, {byte_count})"), + ), + ), + ), + ) + + def _array_result_element_type(self, plan: ResultPlan | NativeCallSlotPlan) -> str: + """Return the completed numeric or fixed-width character element type.""" + if plan.datatype_family is DatatypeFamily.STRING: + itemsize = self._array_result_itemsize(plan) + if itemsize is None: + raise ValueError(f"Character array result {plan.owner_path!r} has no itemsize") + return f"character(kind=c_char, len={itemsize})" + if plan.semantic_type_name is None: + raise ValueError(f"Array result {plan.owner_path!r} has no element type") + return PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name).fortran_spelling + + def _array_result_itemsize(self, plan: ResultPlan | NativeCallSlotPlan) -> int | None: + """Return a character-array itemsize after object-kind dispatch.""" + if plan.datatype_family is not DatatypeFamily.STRING: + return None + if plan.array is None or plan.array.itemsize is None or plan.array.itemsize <= 0: + raise ValueError(f"Character array result {plan.owner_path!r} has no fixed itemsize") + return plan.array.itemsize + + # String result storage. + def _fixed_string_copy_nodes( + self, + *, + length: int, + target_name: str, + value_name: str, + copy_name: str, + ) -> tuple[FortranAssignment | FortranIf, ...]: + """Allocate and fill one justified NUL-terminated fixed string copy.""" c_length = length + 1 return ( - FortranAssignment(name, CodeExpression(f"c_malloc({c_length}_c_size_t)")), + FortranAssignment(target_name, CodeExpression(f"c_malloc({c_length}_c_size_t)")), FortranIf( - CodeExpression(f"c_associated({name})"), + CodeExpression(f"c_associated({target_name})"), body=( FortranCall( "c_f_pointer", - (CodeExpression(name), CodeExpression(copy_name), CodeExpression(f"[{c_length}]")), + ( + CodeExpression(target_name), + CodeExpression(copy_name), + CodeExpression(f"[{c_length}]"), + ), ), FortranAssignment( f"{copy_name}(1:{length})", @@ -717,17 +1533,30 @@ def _lower_native_output_representation_copy( def _native_output_value_name(self, slot: NativeCallSlotPlan) -> str: """Return the native-call expression selected for one output slot.""" name = slot.native_name.lower() - return f"{name}_value" if slot.datatype_family is DatatypeFamily.STRING else name + return f"{name}_value" if slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} else name def _string_output_length(self, slot: NativeCallSlotPlan) -> int: if slot.character_length is None or slot.character_length <= 0: raise ValueError(f"String output {slot.owner_path!r} is missing a fixed character length") return slot.character_length + def _string_result_length(self, result: ResultPlan) -> int: + if result.character_length is None or result.character_length <= 0: + raise ValueError(f"String result {result.owner_path!r} is missing a fixed character length") + return result.character_length + + def _native_direct_result_name(self, plan: FunctionPlan, result_name: str | None) -> str | None: + result = self._direct_result(plan) + if result is not None and result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: + return "result_value" + return result_name + def _bridge_result_type(self, plan: FunctionPlan, result: ResultPlan | None = None) -> str: result = result or self._direct_result(plan) if result is None: raise ValueError(f"{plan.owner_path!r} native function has no result plan") + if result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY}: + return "type(c_ptr)" return PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name).fortran_spelling def _direct_result(self, plan: FunctionPlan) -> ResultPlan | None: @@ -758,16 +1587,7 @@ def _external_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...] def _allocator_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...]: """Return the allocator interface required by detached bridge copies.""" - needs_snapshot = any( - variable.bridge.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT for variable in self._variables(plan) - ) - needs_string_output = any( - slot.datatype_family is DatatypeFamily.STRING - for function in self._functions(plan) - for slot in function.native_call_slots - if slot.source_kind == "result" - ) - if not needs_snapshot and not needs_string_output: + if not self._needs_allocator_interface(plan): return () procedure = FortranInterfaceProcedure( name="c_malloc", @@ -779,19 +1599,45 @@ def _allocator_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ... ) return (FortranInterface((procedure,)),) + def _needs_allocator_interface(self, plan: ModulePlan) -> bool: + """Return whether module snapshots or function copies allocate storage.""" + return self._needs_snapshot_allocator(plan) or self._needs_function_copy_allocator(plan) + + def _needs_snapshot_allocator(self, plan: ModulePlan) -> bool: + """Return whether a module-variable getter produces a detached snapshot.""" + return any( + variable.bridge.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT for variable in self._variables(plan) + ) + + def _needs_function_copy_allocator(self, plan: ModulePlan) -> bool: + """Return whether any function copies an array or string result.""" + return any(self._function_needs_copy_allocator(function) for function in self._functions(plan)) + + def _function_needs_copy_allocator(self, function: FunctionPlan) -> bool: + """Return whether one function owns a result-copy allocation.""" + return self._result_plans_need_allocator(function) or self._native_result_slots_need_allocator(function) + + def _result_plans_need_allocator(self, function: FunctionPlan) -> bool: + """Return whether one Python result is an array or string copy.""" + return any(result.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} for result in function.results) + + def _native_result_slots_need_allocator(self, function: FunctionPlan) -> bool: + """Return whether one hidden native result is an array or string copy.""" + return any( + slot.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} + for slot in function.native_call_slots + if slot.source_kind == "result" + ) + def _external_interface_procedure(self, plan: FunctionPlan) -> FortranInterfaceProcedure: parameters = tuple( - FortranParameter( - argument.bridge.native_name.lower(), - PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).fortran_spelling, - ("optional",) if argument.bridge.optional_mode is not OptionalMode.REQUIRED else (), - ) + self._external_interface_parameter(argument) for argument in sorted(plan.arguments, key=lambda item: item.native_position) ) imports = tuple(dict.fromkeys(self._iso_symbol(argument.semantic_type_name) for argument in plan.arguments)) result_name = None if plan.bridge.native_is_subroutine else "native_result" direct_result = self._direct_result(plan) - result_type = self._bridge_result_type(plan, direct_result) if result_name is not None else None + result_type = self._native_result_type(plan, direct_result) if result_name is not None else None if result_type is not None and direct_result is not None: imports = tuple(dict.fromkeys((*imports, self._iso_symbol(direct_result.semantic_type_name)))) return FortranInterfaceProcedure( @@ -803,6 +1649,74 @@ def _external_interface_procedure(self, plan: FunctionPlan) -> FortranInterfaceP is_subroutine=plan.bridge.native_is_subroutine, ) + def _native_result_type(self, plan: FunctionPlan, result: ResultPlan | None) -> str: + """Return the native procedure result type inside an external interface.""" + if result is None: + raise ValueError("External native function is missing its direct result plan") + if result.object_kind is ObjectKind.NUMPY_ARRAY: + shape = self._array_result_shape(plan, result) + return f"{self._array_result_element_type(result)}, dimension({', '.join(shape)})" + if result.object_kind is ObjectKind.STRING: + return f"character(kind=c_char, len={self._string_result_length(result)})" + return PrimitiveScalarTypeRegistry.type_for(result.semantic_type_name).fortran_spelling + + def _external_interface_parameter(self, argument: ArgumentTransferPlan) -> FortranParameter: + """Return the native external dummy declaration for one planned argument.""" + attributes = ("optional",) if argument.bridge.optional_mode is not OptionalMode.REQUIRED else () + if argument.object_kind is ObjectKind.NUMPY_ARRAY: + array = argument.array + if array is None: + raise ValueError(f"Array argument {argument.owner_path!r} has no shape plan") + element_type = self._array_element_fortran_type(argument) + dimension = ".." if array.rank is None else ", ".join(":" for _ in range(array.rank)) + return FortranParameter( + argument.bridge.native_name.lower(), + element_type, + (*attributes, f"dimension({dimension})"), + ) + if argument.object_kind is ObjectKind.STRING: + length = argument.native_call_slot.character_length + length_text = "*" if length is None else str(length) + return FortranParameter( + argument.bridge.native_name.lower(), + f"character(kind=c_char, len={length_text})", + attributes, + ) + return FortranParameter( + argument.bridge.native_name.lower(), + PrimitiveScalarTypeRegistry.type_for(argument.semantic_type_name).fortran_spelling, + attributes, + ) + + # Ordinary-array result-shape lowering. + def _array_result_shape(self, plan: FunctionPlan, result: ResultPlan) -> tuple[str, ...]: + """Lower one result shape through the plan's native scalar roles.""" + if result.array is None: + raise ValueError(f"Array result {result.owner_path!r} has no shape plan") + return self._array_result_shape_from_roles(result.array, plan.arguments) + + def _array_output_shape(self, plan: FunctionPlan, slot: NativeCallSlotPlan) -> tuple[str, ...]: + """Lower one hidden-output shape through the plan's native scalar roles.""" + if slot.array is None: + raise ValueError(f"Array output {slot.owner_path!r} has no shape plan") + return self._array_result_shape_from_roles(slot.array, plan.arguments) + + def _array_result_shape_from_roles(self, array, arguments) -> tuple[str, ...]: + """Replace validated shape references with their native dummy names.""" + lowered_shape = [] + role_names = {argument.binding.handoff_role: argument.bridge.native_name.lower() for argument in arguments} + for axis, expression in enumerate(array.shape): + lowered = expression + for role in array.extent_reference_roles[axis]: + native_name = role_names.get(role) + if native_name is None: + reference_name = role.rsplit(".", 1)[-1].split(":", 1)[0] + native_name = reference_name + reference_name = role.rsplit(".", 1)[-1].split(":", 1)[0] + lowered = re.sub(rf"\b{re.escape(reference_name)}\b", native_name, lowered) + lowered_shape.append(lowered) + return tuple(lowered_shape) + def _has_optional_arguments(self, plan: FunctionPlan) -> bool: return any(argument.bridge.optional_mode is not OptionalMode.REQUIRED for argument in plan.arguments) @@ -845,6 +1759,7 @@ def _iso_symbol(self, semantic_type_name: str) -> str: "Float64": "c_double", "Complex64": "c_float_complex", "Complex128": "c_double_complex", + "String": "c_char", } return symbols[semantic_type_name] @@ -867,4 +1782,5 @@ def _iso_c_symbols(self) -> tuple[str, ...]: "c_ptr", "c_null_ptr", "c_size_t", + "c_sizeof", ) diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index e8cfb6014..732977aa7 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -13,16 +13,27 @@ from x2py.semantics.ownership import ( AssignmentMode, CodegenAction, + DestructionPolicy, NativeBarrierAction, + ObjectKind, + OwnershipOwner, PythonBarrierAction, SetterAction, + StorageMode, + TransferMode, ) from x2py.semantics.wrapper_policy import ( ArgumentHandoffMode, BridgeDataAction, + FIXED_STRING_RESULT_COPY_REASON, + ORDINARY_ARRAY_RESULT_COPY_REASON, ModuleGetterAction, OptionalMode, PythonExceptionKind, + RAW_STRING_ADDRESS_COPY_REASON, + STRING_INPUT_COPY_REASON, + STRING_REPLACEMENT_COPY_REASON, + STRING_STORAGE_COPY_REASON, WritebackPhase, ) from x2py.wrapper_codegen.c.binding import CBindingGenerator @@ -283,31 +294,34 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost *self._duplicate_role_diagnostics(plan), *self._available_role_diagnostics(plan), *self._function_output_diagnostics(plan), + *self._string_result_aggregation_diagnostics(plan), *self._status_error_diagnostics(plan), ] slots = {slot.native_position: slot for slot in plan.native_call_slots} for slot in plan.native_call_slots: diagnostics.extend(self._native_slot_diagnostics(slot)) for argument in plan.arguments: - diagnostics.extend(self._argument_diagnostics(argument, slots)) + diagnostics.extend(self._argument_diagnostics(argument, slots, plan.available_roles)) for result in plan.results: diagnostics.extend(self._result_diagnostics(result, slots, plan.available_roles)) for action in (*plan.writeback_actions, *plan.cleanup_actions, *plan.release_actions): diagnostics.extend(self._lifecycle_diagnostics(action, plan.available_roles)) diagnostics.extend(self._writeback_phase_diagnostics(plan)) + diagnostics.extend(self._string_writeback_diagnostics(plan)) return tuple(diagnostics) def _argument_diagnostics( self, plan: ArgumentTransferPlan, function_slots: dict[int, NativeCallSlotPlan], + available_roles: tuple[str, ...], ) -> tuple[WrapperPlanDiagnostic, ...]: """Return binding-to-bridge handoff and slot diagnostics.""" diagnostics = [ *self._argument_policy_consistency_diagnostics(plan), *self._argument_slot_consistency_diagnostics(plan, function_slots), *self._optional_argument_diagnostics(plan), - *self._scalar_boundary_diagnostics(plan), + *self._argument_family_diagnostics(plan, available_roles), *self._argument_data_action_diagnostics(plan), *self._bridge_data_diagnostics( plan.owner_path, @@ -317,6 +331,60 @@ def _argument_diagnostics( ] return tuple(diagnostics) + def _argument_family_diagnostics( + self, + plan: ArgumentTransferPlan, + available_roles: tuple[str, ...], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Dispatch one argument from its completed object-kind decision.""" + match plan.object_kind: + case ObjectKind.SCALAR: + diagnostics = list(self._scalar_boundary_diagnostics(plan)) + if plan.array is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "unexpected-scalar-array-handoff", None)) + if plan.datatype_family is DatatypeFamily.STRING: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-scalar-datatype-family", + plan.datatype_family.value, + ) + ) + return tuple(diagnostics) + case ObjectKind.STRING: + diagnostics = list(self._string_boundary_diagnostics(plan)) + if plan.array is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "unexpected-string-array-handoff", None)) + return tuple(diagnostics) + case ObjectKind.NUMPY_ARRAY: + return ( + *self._array_boundary_diagnostics(plan), + *self._array_extent_reference_diagnostics(plan, available_roles), + ) + case _: + return ( + self._diagnostic( + plan.owner_path, + "unsupported-argument-object-kind", + plan.object_kind.value, + ), + ) + + def _array_extent_reference_diagnostics( + self, + plan: ArgumentTransferPlan, + available_roles: tuple[str, ...], + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require every planned extent dependency to name a function role.""" + if plan.array is None: + return () + return tuple( + self._diagnostic(plan.owner_path, "unavailable-array-extent-reference", role) + for axis_roles in plan.array.extent_reference_roles + for role in axis_roles + if role not in available_roles + ) + def _argument_policy_consistency_diagnostics( self, plan: ArgumentTransferPlan, @@ -328,6 +396,14 @@ def _argument_policy_consistency_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-bridge-handoff", role)) if plan.native_call_slot.symbolic_role != role: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-native-handoff", role)) + if plan.bridge.length_handoff_role != plan.binding.length_handoff_role: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-length-handoff", + plan.binding.length_handoff_role, + ) + ) if plan.bridge.native_action is not plan.native_call_slot.native_action: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-native-action", plan.bridge.native_action.value) @@ -340,6 +416,9 @@ def _argument_policy_consistency_diagnostics( plan.native_call_slot.bridge_data_action.value, ) ) + if plan.array is not plan.native_call_slot.array: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-array-handoff", plan.array)) + diagnostics.extend(self._argument_completed_fact_diagnostics(plan)) if plan.bridge.copy_reason != plan.native_call_slot.bridge_copy_reason: diagnostics.append( self._diagnostic( @@ -350,6 +429,52 @@ def _argument_policy_consistency_diagnostics( ) return tuple(diagnostics) + def _argument_completed_fact_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return projected action, length, mutability, and nullability drift.""" + diagnostics = [] + if plan.binding.codegen_action is not plan.bridge.codegen_action: + diagnostics.append( + self._diagnostic( + plan.owner_path, "inconsistent-argument-codegen-action", plan.bridge.codegen_action.value + ) + ) + if plan.binding.codegen_action is not plan.native_call_slot.codegen_action: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-native-slot-codegen-action", + plan.native_call_slot.codegen_action.value, + ) + ) + if plan.character_length != plan.native_call_slot.character_length: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-argument-character-length", + plan.native_call_slot.character_length, + ) + ) + if plan.binding.writable != plan.mutates_native: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-argument-mutability", plan.binding.writable) + ) + if plan.binding.nullable != plan.nullable: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-argument-nullability", plan.binding.nullable) + ) + if plan.native_call_slot.object_kind is not plan.object_kind: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-argument-object-kind", + plan.native_call_slot.object_kind, + ) + ) + return tuple(diagnostics) + def _argument_slot_consistency_diagnostics( self, plan: ArgumentTransferPlan, @@ -378,7 +503,14 @@ def _argument_data_action_diagnostics( plan: ArgumentTransferPlan, ) -> tuple[WrapperPlanDiagnostic, ...]: """Require the completed data action to match the selected scalar path.""" - if plan.bridge.optional_mode is OptionalMode.DESCRIPTOR: + if plan.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: + expected = BridgeDataAction.ASSOCIATE_VIEW + elif plan.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER or ( + plan.datatype_family is DatatypeFamily.STRING + and plan.bridge.handoff_mode is ArgumentHandoffMode.OPAQUE_ADDRESS + ): + expected = BridgeDataAction.COPY_REPRESENTATION + elif plan.bridge.optional_mode is OptionalMode.DESCRIPTOR: expected = ( BridgeDataAction.COPY_REPRESENTATION if plan.native_call_slot.value_kind == "allocatable" @@ -401,12 +533,19 @@ def _argument_data_action_diagnostics( ), ) + # Scalar argument validation. def _scalar_boundary_diagnostics( self, plan: ArgumentTransferPlan, ) -> tuple[WrapperPlanDiagnostic, ...]: """Return completed Python/native scalar boundary consistency diagnostics.""" action = plan.binding.python_action + if action not in { + PythonBarrierAction.SCALAR_VALUE, + PythonBarrierAction.SCALAR_STORAGE, + PythonBarrierAction.RAW_ADDRESS, + }: + return (self._diagnostic(plan.owner_path, "invalid-scalar-python-action", action.value),) expected = { PythonBarrierAction.SCALAR_STORAGE: NativeBarrierAction.PASS_STORAGE_ADDRESS, PythonBarrierAction.RAW_ADDRESS: NativeBarrierAction.PASS_RAW_ADDRESS, @@ -432,6 +571,492 @@ def _scalar_boundary_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "optional-scalar-address-boundary", action.value)) return tuple(diagnostics) + # Ordinary-array argument validation. + def _array_boundary_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate one ordinary-array transfer selected by object kind.""" + array = plan.array + if array is None: + return (self._diagnostic(plan.owner_path, "missing-array-handoff", None),) + diagnostics = [ + *self._array_ownership_diagnostics(plan), + *self._array_action_diagnostics(plan), + *self._array_scope_diagnostics(plan), + *self._array_shape_diagnostics(plan), + ] + return tuple(diagnostics) + + def _array_ownership_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate caller-owned ordinary-array lifetime facts.""" + diagnostics = [] + expected = ( + ("object-kind", plan.object_kind, ObjectKind.NUMPY_ARRAY), + ("owner", plan.ownership_owner, OwnershipOwner.CALLER), + ("storage", plan.storage_mode, StorageMode.STACK), + ("boundary-storage", plan.boundary_storage_mode, StorageMode.STACK), + ) + diagnostics.extend( + self._diagnostic(plan.owner_path, f"invalid-array-{name}", actual.value) + for name, actual, required in expected + if actual is not required + ) + expected_transfer = TransferMode.IN_PLACE if plan.mutates_native else TransferMode.CALL_LOCAL + expected_destruction = DestructionPolicy.CALLER if plan.mutates_native else DestructionPolicy.NONE + if plan.transfer_mode is not expected_transfer: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-transfer", plan.transfer_mode.value)) + if plan.destruction_policy is not expected_destruction: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-destruction", plan.destruction_policy.value) + ) + return tuple(diagnostics) + + def _array_action_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate completed binding and bridge array actions.""" + diagnostics = [] + if plan.binding.python_action is not PythonBarrierAction.ARRAY_STORAGE: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-python-action", plan.binding.python_action.value) + ) + if plan.bridge.native_action is not NativeBarrierAction.PASS_ARRAY_BUFFER: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-native-action", plan.bridge.native_action.value) + ) + if plan.bridge.handoff_mode is not ArgumentHandoffMode.ARRAY_BUFFER: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-handoff-mode", plan.bridge.handoff_mode.value) + ) + if plan.bridge.data_action is not BridgeDataAction.ASSOCIATE_VIEW: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-data-action", plan.bridge.data_action.value) + ) + if plan.binding.codegen_action not in { + CodegenAction.CALL_LOCAL_INPUT, + CodegenAction.IN_PLACE_ARGUMENT, + CodegenAction.IDENTITY_OUTPUT, + }: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-codegen-action", plan.binding.codegen_action.value) + ) + return tuple(diagnostics) + + def _array_scope_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Keep ordinary arrays on the non-descriptor storage boundary.""" + diagnostics = [] + if plan.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE}: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-optional-mode", plan.binding.optional_mode.value) + ) + if plan.nullable or plan.binding.descriptor_boundary: + diagnostics.append(self._diagnostic(plan.owner_path, "descriptor-backed-ordinary-array", plan.nullable)) + if plan.projects_result and plan.result_position is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-array-result-position", None)) + return tuple(diagnostics) + + def _array_shape_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate concrete or assumed-rank array layout and ABI roles.""" + array = plan.array + if array is None: + return () + diagnostics = [] + if array.rank is None: + diagnostics.extend(self._assumed_rank_array_diagnostics(plan)) + else: + diagnostics.extend(self._concrete_rank_array_diagnostics(plan)) + diagnostics.extend(self._array_layout_role_diagnostics(plan)) + diagnostics.extend(self._array_handoff_role_diagnostics(plan)) + diagnostics.extend(self._array_itemsize_diagnostics(plan)) + return tuple(diagnostics) + + def _array_handoff_role_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate ordinary-array data, extent, and shape-reference roles.""" + array = plan.array + if array is None: + return () + diagnostics = [] + if array.data_role != plan.binding.handoff_role: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-array-data-role", array.data_role)) + if any(not role for role in array.extent_roles): + diagnostics.append(self._diagnostic(plan.owner_path, "missing-array-extent-role", array.extent_roles)) + if len(array.extent_reference_roles) != len(array.shape): + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-extent-reference-count", array.extent_reference_roles) + ) + return tuple(diagnostics) + + def _array_itemsize_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate fixed-width character itemsize fields only on character arrays.""" + array = plan.array + if array is None: + return () + if plan.datatype_family is DatatypeFamily.STRING: + if array.itemsize is None or array.itemsize <= 0 or array.itemsize_role is None: + return (self._diagnostic(plan.owner_path, "invalid-array-itemsize", array.itemsize),) + return () + if array.itemsize is not None or array.itemsize_role is not None: + return (self._diagnostic(plan.owner_path, "unexpected-array-itemsize", array.itemsize),) + return () + + def _concrete_rank_array_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate fixed-rank shape and layout facts.""" + array = plan.array + if array is None or array.rank is None: + return () + diagnostics = [] + if not 1 <= array.rank <= 15: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-rank", array.rank)) + if len(array.shape) != array.rank or len(array.axes) != array.rank: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-array-rank", array.rank)) + if len(array.extent_roles) != array.rank or array.runtime_rank_role is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-rank-roles", array.extent_roles)) + return tuple(diagnostics) + + def _assumed_rank_array_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the one-through-fifteen runtime-rank ABI.""" + array = plan.array + if array is None or array.rank is not None: + return () + diagnostics = [] + if array.category != "assumed_rank" or array.shape != ("...",): + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-assumed-rank-array", array.shape)) + if len(array.extent_roles) != 15 or array.runtime_rank_role is None: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-assumed-rank-roles", array.extent_roles)) + return tuple(diagnostics) + + def _array_layout_role_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate dense versus stride-aware ABI fields.""" + array = plan.array + if array is None: + return () + return ( + *self._array_order_diagnostics(plan), + *self._array_axis_mode_diagnostics(plan), + *self._array_stride_role_diagnostics(plan), + ) + + def _array_order_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the completed ordinary-array order marker.""" + array = plan.array + if array is not None and array.order not in {None, "ORDER_F", "ORDER_C"}: + return (self._diagnostic(plan.owner_path, "invalid-array-order", array.order),) + return () + + def _array_axis_mode_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate dense versus stride-aware axis markers.""" + array = plan.array + if array is None: + return () + if array.order not in {None, "ORDER_F", "ORDER_C"}: + return () + if array.contiguous not in {True, False}: + return (self._diagnostic(plan.owner_path, "invalid-array-contiguity", array.contiguous),) + if array.contiguous is True and any(axis != "dense" for axis in array.axes): + return (self._diagnostic(plan.owner_path, "invalid-array-axis-modes", array.axes),) + if array.contiguous is False and "strided" not in array.axes: + return (self._diagnostic(plan.owner_path, "invalid-array-axis-modes", array.axes),) + return () + + def _array_stride_role_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate stride metadata presence or absence from completed contiguity.""" + array = plan.array + if array is None: + return () + if array.contiguous is False: + return ( + *self._required_array_stride_role_diagnostics(plan), + *self._array_stride_role_count_diagnostics(plan), + ) + if array.upper_bound_roles or array.stride_roles: + return (self._diagnostic(plan.owner_path, "unexpected-dense-array-stride-roles", None),) + return () + + def _required_array_stride_role_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require positive-stride metadata and a Fortran-oriented layout.""" + array = plan.array + if array is not None and (array.order == "ORDER_C" or not array.upper_bound_roles or not array.stride_roles): + return (self._diagnostic(plan.owner_path, "invalid-strided-array-layout", array.order),) + return () + + def _array_stride_role_count_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require one upper bound and element stride per array extent.""" + array = plan.array + if array is None: + return () + diagnostics = [] + if len(array.upper_bound_roles) != len(array.extent_roles): + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-upper-bound-roles", None)) + if len(array.stride_roles) != len(array.extent_roles): + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-stride-roles", None)) + return tuple(diagnostics) + + # String argument validation. + def _string_boundary_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Dispatch completed string-value, storage, or raw-address validation.""" + if plan.datatype_family is not DatatypeFamily.STRING: + return ( + self._diagnostic( + plan.owner_path, + "invalid-string-datatype-family", + plan.datatype_family.value, + ), + ) + action = plan.binding.python_action + if action is PythonBarrierAction.STRING_VALUE: + return (*self._string_value_action_diagnostics(plan), *self._string_length_diagnostics(plan)) + if action is PythonBarrierAction.STRING_STORAGE: + return self._string_address_diagnostics( + plan, + native_action=NativeBarrierAction.PASS_STORAGE_ADDRESS, + storage_mode=StorageMode.ALIAS, + copy_reason=STRING_STORAGE_COPY_REASON, + label="storage", + ) + if action is PythonBarrierAction.RAW_ADDRESS: + return self._string_address_diagnostics( + plan, + native_action=NativeBarrierAction.PASS_RAW_ADDRESS, + storage_mode=StorageMode.STACK, + copy_reason=RAW_STRING_ADDRESS_COPY_REASON, + label="raw-address", + ) + return (self._diagnostic(plan.owner_path, "invalid-string-python-action", action.value),) + + def _string_value_action_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return action, handoff, and presence diagnostics for one string input.""" + diagnostics = [] + if plan.binding.python_action is not PythonBarrierAction.STRING_VALUE: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-python-action", plan.binding.python_action.value) + ) + if plan.bridge.native_action is not NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-native-action", plan.bridge.native_action.value) + ) + if plan.bridge.handoff_mode is not ArgumentHandoffMode.CHARACTER_BUFFER: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-handoff", plan.bridge.handoff_mode.value) + ) + if plan.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-data-action", plan.bridge.data_action.value) + ) + if plan.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.NULLABLE_VALUE}: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-string-optional-mode", + plan.binding.optional_mode.value, + ) + ) + diagnostics.extend(self._string_codegen_diagnostics(plan)) + return tuple(diagnostics) + + def _string_address_diagnostics( + self, + plan: ArgumentTransferPlan, + *, + native_action: NativeBarrierAction, + storage_mode: StorageMode, + copy_reason: str, + label: str, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate one fixed string storage/raw address plan.""" + diagnostics = list( + self._string_address_ownership_diagnostics( + plan, + storage_mode=storage_mode, + label=label, + ) + ) + if plan.bridge.native_action is not native_action: + diagnostics.append( + self._diagnostic( + plan.owner_path, f"invalid-string-{label}-native-action", plan.bridge.native_action.value + ) + ) + if plan.bridge.handoff_mode is not ArgumentHandoffMode.OPAQUE_ADDRESS: + diagnostics.append( + self._diagnostic(plan.owner_path, f"invalid-string-{label}-handoff", plan.bridge.handoff_mode.value) + ) + if plan.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + diagnostics.append( + self._diagnostic(plan.owner_path, f"invalid-string-{label}-data-action", plan.bridge.data_action.value) + ) + if plan.bridge.copy_reason != copy_reason: + diagnostics.append( + self._diagnostic(plan.owner_path, f"invalid-string-{label}-copy-reason", plan.bridge.copy_reason) + ) + diagnostics.extend(self._string_address_length_diagnostics(plan, label)) + return tuple(diagnostics) + + def _string_address_ownership_diagnostics( + self, + plan: ArgumentTransferPlan, + *, + storage_mode: StorageMode, + label: str, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate completed caller-owned in-place string address facts.""" + expected = ( + ("object-kind", plan.object_kind, ObjectKind.STRING), + ("owner", plan.ownership_owner, OwnershipOwner.CALLER), + ("transfer", plan.transfer_mode, TransferMode.IN_PLACE), + ("destruction", plan.destruction_policy, DestructionPolicy.CALLER), + ("storage", plan.storage_mode, storage_mode), + ("boundary-storage", plan.boundary_storage_mode, storage_mode), + ) + diagnostics = [ + self._diagnostic(plan.owner_path, f"invalid-string-{label}-{name}", actual.value) + for name, actual, required in expected + if actual is not required + ] + if plan.binding.codegen_action is not CodegenAction.IN_PLACE_ARGUMENT: + diagnostics.append( + self._diagnostic(plan.owner_path, f"invalid-string-{label}-action", plan.binding.codegen_action.value) + ) + if not plan.mutates_native: + diagnostics.append(self._diagnostic(plan.owner_path, f"string-{label}-without-mutation", False)) + if plan.projects_result: + diagnostics.append( + self._diagnostic(plan.owner_path, f"string-{label}-projects-result", plan.result_position) + ) + if plan.nullable or plan.binding.optional_mode is not OptionalMode.REQUIRED: + diagnostics.append( + self._diagnostic(plan.owner_path, f"optional-string-{label}", plan.binding.optional_mode.value) + ) + return tuple(diagnostics) + + def _string_address_length_diagnostics( + self, + plan: ArgumentTransferPlan, + label: str, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require one fixed plan length and prohibit a runtime length ABI role.""" + diagnostics = [] + if plan.character_length is None or plan.character_length <= 0: + diagnostics.append( + self._diagnostic(plan.owner_path, f"invalid-string-{label}-length", plan.character_length) + ) + if plan.binding.length_handoff_role is not None: + diagnostics.append( + self._diagnostic( + plan.owner_path, + f"unexpected-string-{label}-length-handoff", + plan.binding.length_handoff_role, + ) + ) + return tuple(diagnostics) + + def _string_codegen_diagnostics(self, plan: ArgumentTransferPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Return string action, copy-reason, and replacement diagnostics.""" + diagnostics = [] + action = plan.binding.codegen_action + if action not in {CodegenAction.CALL_LOCAL_INPUT, CodegenAction.COPY_IN_OUT}: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-string-codegen-action", action.value)) + expected_reason = ( + STRING_REPLACEMENT_COPY_REASON if action is CodegenAction.COPY_IN_OUT else STRING_INPUT_COPY_REASON + ) + if plan.bridge.copy_reason != expected_reason: + diagnostics.append(self._diagnostic(plan.owner_path, "invalid-string-copy-reason", plan.bridge.copy_reason)) + if action is CodegenAction.COPY_IN_OUT: + diagnostics.extend(self._string_replacement_diagnostics(plan)) + elif plan.projects_result: + diagnostics.append( + self._diagnostic(plan.owner_path, "call-local-string-projects-result", plan.result_position) + ) + return tuple(diagnostics) + + def _string_replacement_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate completed ownership and projection for one string replacement.""" + expected = ( + ("object-kind", plan.object_kind, ObjectKind.STRING), + ("owner", plan.ownership_owner, OwnershipOwner.PYTHON), + ("transfer", plan.transfer_mode, TransferMode.COPY_RETURN), + ("destruction", plan.destruction_policy, DestructionPolicy.PYTHON_REFCOUNT), + ("storage", plan.storage_mode, StorageMode.STACK), + ("boundary-storage", plan.boundary_storage_mode, StorageMode.STACK), + ) + diagnostics = [ + self._diagnostic(plan.owner_path, f"invalid-string-replacement-{name}", actual.value) + for name, actual, required in expected + if actual is not required + ] + if not plan.mutates_native: + diagnostics.append(self._diagnostic(plan.owner_path, "string-replacement-without-mutation", False)) + if not plan.projects_result or plan.result_position is None: + diagnostics.append( + self._diagnostic(plan.owner_path, "string-replacement-without-result", plan.result_position) + ) + if plan.nullable: + diagnostics.append(self._diagnostic(plan.owner_path, "nullable-string-replacement", True)) + return tuple(diagnostics) + + def _string_length_diagnostics( + self, + plan: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return payload-length handoff and fixed-length diagnostics.""" + diagnostics = [] + length_role = plan.binding.length_handoff_role + if length_role is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-string-length-handoff", None)) + if plan.native_call_slot.character_length is not None and plan.native_call_slot.character_length <= 0: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-string-character-length", + plan.native_call_slot.character_length, + ) + ) + if plan.character_length is not None and plan.character_length <= 0: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-argument-character-length", plan.character_length) + ) + return tuple(diagnostics) + def _optional_argument_diagnostics( self, plan: ArgumentTransferPlan, @@ -472,6 +1097,8 @@ def _optional_native_diagnostics( NativeBarrierAction.PASS_VALUE, NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, NativeBarrierAction.PASS_STORAGE_ADDRESS, + NativeBarrierAction.PASS_ARRAY_BUFFER, + NativeBarrierAction.PASS_NATIVE_DESCRIPTOR, }: diagnostics.append( self._diagnostic(plan.owner_path, "invalid-optional-native-action", plan.bridge.native_action.value) @@ -511,6 +1138,154 @@ def _result_diagnostics( diagnostics.extend(self._hidden_result_diagnostics(plan, function_slots)) else: diagnostics.append(self._diagnostic(plan.owner_path, "unknown-result-source", plan.source_kind)) + diagnostics.extend(self._result_family_diagnostics(plan)) + return tuple(diagnostics) + + def _result_family_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Dispatch one result from its completed object-kind decision.""" + match plan.object_kind: + case ObjectKind.SCALAR: + diagnostics = list(self._nonstring_result_length_diagnostics(plan)) + if plan.array is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "unexpected-scalar-result-array", None)) + if plan.datatype_family is DatatypeFamily.STRING: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-scalar-result-datatype-family", + plan.datatype_family.value, + ) + ) + return tuple(diagnostics) + case ObjectKind.STRING: + if plan.array is not None: + return (self._diagnostic(plan.owner_path, "unexpected-string-result-array", None),) + return self._string_result_diagnostics(plan) + case ObjectKind.NUMPY_ARRAY: + return self._array_result_diagnostics(plan) + case _: + return ( + self._diagnostic( + plan.owner_path, + "unsupported-result-object-kind", + plan.object_kind.value, + ), + ) + + # String result validation. + def _string_result_aggregation_diagnostics( + self, + plan: FunctionPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Keep fixed string allocation cleanup single-result in Phase 5B.""" + if any(result.object_kind is ObjectKind.STRING for result in plan.results) and len(plan.results) != 1: + return (self._diagnostic(plan.owner_path, "mixed-string-result-aggregation", len(plan.results)),) + return () + + def _string_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the fixed string result contract before either backend.""" + if plan.datatype_family is not DatatypeFamily.STRING: + return ( + self._diagnostic( + plan.owner_path, + "invalid-string-result-datatype-family", + plan.datatype_family.value, + ), + ) + diagnostics = [] + if plan.character_length is None or plan.character_length <= 0: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-result-character-length", plan.character_length) + ) + diagnostics.extend(self._string_result_ownership_diagnostics(plan)) + if plan.binding.python_action is not PythonBarrierAction.NONE: + diagnostics.append( + self._diagnostic( + plan.owner_path, "invalid-string-result-python-action", plan.binding.python_action.value + ) + ) + if plan.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-result-data-action", plan.bridge.data_action.value) + ) + if plan.bridge.copy_reason != FIXED_STRING_RESULT_COPY_REASON: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-string-result-copy-reason", plan.bridge.copy_reason) + ) + if plan.source_kind == "direct_return": + diagnostics.extend(self._direct_string_result_diagnostics(plan)) + elif plan.source_kind == "hidden_output": + diagnostics.extend(self._hidden_string_result_diagnostics(plan)) + return tuple(diagnostics) + + def _string_result_ownership_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate completed fixed-string ownership projected into the plan.""" + expected = ( + ("object-kind", plan.object_kind, ObjectKind.STRING), + ("owner", plan.ownership_owner, OwnershipOwner.PYTHON), + ("transfer", plan.transfer_mode, TransferMode.COPY_RETURN), + ("destruction", plan.destruction_policy, DestructionPolicy.PYTHON_REFCOUNT), + ("storage", plan.storage_mode, StorageMode.STACK), + ("boundary-storage", plan.boundary_storage_mode, StorageMode.STACK), + ) + diagnostics = [ + self._diagnostic(plan.owner_path, f"invalid-string-result-{name}", actual.value) + for name, actual, required in expected + if actual is not required + ] + if plan.nullable: + diagnostics.append(self._diagnostic(plan.owner_path, "nullable-fixed-string-result", plan.nullable)) + return tuple(diagnostics) + + def _nonstring_result_length_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + if plan.character_length is None: + return () + return ( + self._diagnostic( + plan.owner_path, + "nonstring-result-character-length", + plan.character_length, + ), + ) + + def _direct_string_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + diagnostics = [] + if plan.binding.codegen_action is not CodegenAction.COPY_OUT: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-direct-string-result-action", + plan.binding.codegen_action.value, + ) + ) + if plan.bridge.native_action is not NativeBarrierAction.NONE: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-direct-string-native-action", + plan.bridge.native_action.value, + ) + ) + return tuple(diagnostics) + + def _hidden_string_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + diagnostics = [] + if plan.binding.codegen_action is not CodegenAction.COPY_OUT: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-hidden-string-result-action", + plan.binding.codegen_action.value, + ) + ) + if plan.bridge.native_action is not NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "invalid-hidden-string-native-action", + plan.bridge.native_action.value, + ) + ) return tuple(diagnostics) def _result_role_diagnostics( @@ -526,7 +1301,12 @@ def _direct_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagn diagnostics = [] if plan.native_call_slot is not None or plan.bridge.abi_position is not None: diagnostics.append(self._diagnostic(plan.owner_path, "direct-result-has-native-slot", plan.source_kind)) - if plan.bridge.data_action is not BridgeDataAction.DIRECT_TRANSFER: + expected_data_action = ( + BridgeDataAction.COPY_REPRESENTATION + if plan.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} + else BridgeDataAction.DIRECT_TRANSFER + ) + if plan.bridge.data_action is not expected_data_action: diagnostics.append( self._diagnostic(plan.owner_path, "invalid-direct-result-data-action", plan.bridge.data_action.value) ) @@ -592,6 +1372,144 @@ def _hidden_result_policy_consistency_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-result-copy-reason", slot.bridge_copy_reason) ) + if slot.character_length != plan.character_length: + diagnostics.append( + self._diagnostic(plan.owner_path, "inconsistent-result-character-length", slot.character_length) + ) + if slot.array is not plan.array: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-result-array-handoff", slot.array)) + if slot.object_kind is not plan.object_kind: + diagnostics.append( + self._diagnostic( + plan.owner_path, + "inconsistent-result-object-kind", + slot.object_kind, + ) + ) + return tuple(diagnostics) + + # Ordinary-array result validation. + def _array_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate one fixed-shape ordinary array producer and copy consumer.""" + array = plan.array + if array is None: + return (self._diagnostic(plan.owner_path, "missing-array-result-handoff", None),) + diagnostics = [ + *self._array_result_ownership_diagnostics(plan), + *self._array_result_shape_diagnostics(plan), + *self._array_result_itemsize_diagnostics(plan), + *self._array_result_copy_diagnostics(plan), + ] + if plan.nullable: + diagnostics.append(self._diagnostic(plan.owner_path, "nullable-ordinary-array-result", plan.nullable)) + diagnostics.extend(self._array_result_source_diagnostics(plan)) + return tuple(diagnostics) + + def _array_result_ownership_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate ordinary array result ownership and storage decisions.""" + expected = ( + ("object-kind", plan.object_kind, ObjectKind.NUMPY_ARRAY), + ("owner", plan.ownership_owner, OwnershipOwner.PYTHON), + ("transfer", plan.transfer_mode, TransferMode.COPY_RETURN), + ("destruction", plan.destruction_policy, DestructionPolicy.PYTHON_REFCOUNT), + ("storage", plan.storage_mode, StorageMode.STACK), + ("boundary-storage", plan.boundary_storage_mode, StorageMode.STACK), + ) + return tuple( + self._diagnostic(plan.owner_path, f"invalid-array-result-{name}", actual.value) + for name, actual, required in expected + if actual is not required + ) + + def _array_result_itemsize_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate fixed-width character itemsize only inside the array family.""" + array = plan.array + if array is None: + return () + if plan.datatype_family is DatatypeFamily.STRING: + if ( + array.itemsize is None + or array.itemsize <= 0 + or array.itemsize_role is None + or plan.character_length != array.itemsize + ): + return (self._diagnostic(plan.owner_path, "invalid-array-result-itemsize", array.itemsize),) + return () + if array.itemsize is not None or array.itemsize_role is not None or plan.character_length is not None: + return (self._diagnostic(plan.owner_path, "unexpected-array-result-itemsize", array.itemsize),) + return () + + def _array_result_shape_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate a fully resolved fixed-rank ordinary array result shape.""" + return ( + *self._array_result_rank_diagnostics(plan), + *self._array_result_shape_count_diagnostics(plan), + *self._array_result_extent_diagnostics(plan), + *self._array_result_order_diagnostics(plan), + ) + + def _array_result_rank_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require a supported concrete ordinary array result rank.""" + array = plan.array + if array is not None and (array.rank is None or not 1 <= array.rank <= 15): + return (self._diagnostic(plan.owner_path, "invalid-array-result-rank", array.rank),) + return () + + def _array_result_shape_count_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require one shape and extent role per result axis.""" + array = plan.array + if ( + array is not None + and array.rank is not None + and (len(array.shape) != array.rank or len(array.extent_roles) != array.rank) + ): + return (self._diagnostic(plan.owner_path, "inconsistent-array-result-shape", array.shape),) + return () + + def _array_result_extent_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject unresolved ordinary array result extent spellings.""" + array = plan.array + if array is not None and any(shape in {":", "::Strided", "...", "Flat"} for shape in array.shape): + return (self._diagnostic(plan.owner_path, "unresolved-array-result-shape", array.shape),) + return () + + def _array_result_order_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject multidimensional C-oriented native result copies.""" + array = plan.array + if array is not None and array.order == "ORDER_C" and array.rank is not None and array.rank > 1: + return (self._diagnostic(plan.owner_path, "invalid-array-result-order", array.order),) + return () + + def _array_result_copy_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the explicit ordinary array representation-copy decision.""" + diagnostics = [] + if plan.bridge.data_action is not BridgeDataAction.COPY_REPRESENTATION: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-result-data-action", plan.bridge.data_action.value) + ) + if plan.bridge.copy_reason != ORDINARY_ARRAY_RESULT_COPY_REASON: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-result-copy-reason", plan.bridge.copy_reason) + ) + return tuple(diagnostics) + + def _array_result_source_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate direct versus hidden ordinary array producer actions.""" + if plan.source_kind == "direct_return": + expected_action = CodegenAction.COPY_OUT + expected_native = NativeBarrierAction.NONE + else: + expected_action = CodegenAction.COPY_OUT + expected_native = NativeBarrierAction.PASS_ARRAY_BUFFER + diagnostics = [] + if plan.binding.codegen_action is not expected_action: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-result-action", plan.binding.codegen_action.value) + ) + if plan.bridge.native_action is not expected_native: + diagnostics.append( + self._diagnostic(plan.owner_path, "invalid-array-result-native-action", plan.bridge.native_action.value) + ) return tuple(diagnostics) def _native_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPlanDiagnostic, ...]: @@ -607,6 +1525,8 @@ def _native_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPla diagnostics.append(self._diagnostic(plan.owner_path, "unknown-native-slot-source", plan.source_kind)) if plan.source_kind == "literal": diagnostics.extend(self._literal_slot_diagnostics(plan)) + elif plan.object_kind is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-native-slot-object-kind", None)) if plan.source_kind == "result": diagnostics.extend(self._result_slot_diagnostics(plan)) return tuple(diagnostics) @@ -637,6 +1557,8 @@ def _literal_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPl diagnostics.append(self._diagnostic(plan.owner_path, "missing-literal-value", plan.native_position)) if plan.python_position is not None: diagnostics.append(self._diagnostic(plan.owner_path, "literal-python-position", plan.python_position)) + if plan.object_kind is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "literal-object-kind", plan.object_kind.value)) if plan.bridge_data_action is not BridgeDataAction.DIRECT_TRANSFER: diagnostics.append( self._diagnostic(plan.owner_path, "invalid-literal-data-action", plan.bridge_data_action.value) @@ -652,13 +1574,13 @@ def _result_slot_diagnostics(self, plan: NativeCallSlotPlan) -> tuple[WrapperPla diagnostics.append(self._diagnostic(plan.owner_path, "result-python-position", plan.python_position)) if plan.semantic_type_name is None or plan.datatype_family is None: diagnostics.append(self._diagnostic(plan.owner_path, "missing-result-datatype", plan.native_position)) - if plan.datatype_family is DatatypeFamily.STRING and plan.character_length is None: + if plan.object_kind is ObjectKind.STRING and plan.character_length is None: diagnostics.append( self._diagnostic(plan.owner_path, "missing-result-character-length", plan.native_position) ) expected = ( BridgeDataAction.COPY_REPRESENTATION - if plan.datatype_family is DatatypeFamily.STRING + if plan.object_kind in {ObjectKind.STRING, ObjectKind.NUMPY_ARRAY} else BridgeDataAction.DIRECT_TRANSFER ) if plan.bridge_data_action is not expected: @@ -724,6 +1646,7 @@ def _binding_lifecycle_diagnostics( binding = plan.binding if binding.source_role != plan.source_role: diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-lifecycle-role", phase_name)) + diagnostics.extend(self._binding_lifecycle_fact_diagnostics(plan, phase_name)) if binding.codegen_action not in {CodegenAction.COPY_IN_OUT, CodegenAction.IN_PLACE_ARGUMENT}: diagnostics.append( self._diagnostic(plan.owner_path, "invalid-writeback-action", binding.codegen_action.value) @@ -734,14 +1657,87 @@ def _binding_lifecycle_diagnostics( diagnostics.append(self._diagnostic(plan.owner_path, "unexpected-python-writeback-target", phase_name)) return tuple(diagnostics) + def _binding_lifecycle_fact_diagnostics( + self, + plan: LifecycleActionPlan, + phase_name: str, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return shared-versus-binding lifecycle fact drift.""" + diagnostics = [] + binding = plan.binding + if binding.codegen_action is not plan.codegen_action: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-lifecycle-action", phase_name)) + if binding.semantic_type_name != plan.semantic_type_name: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-lifecycle-type", phase_name)) + if binding.datatype_family is not plan.datatype_family: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-lifecycle-family", phase_name)) + if binding.result_position != plan.result_position: + diagnostics.append(self._diagnostic(plan.owner_path, "inconsistent-lifecycle-position", phase_name)) + return tuple(diagnostics) + + # String lifecycle validation. + def _string_writeback_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate the complete string replacement lifecycle and exclusions.""" + replacements = tuple( + argument + for argument in plan.arguments + if argument.object_kind is ObjectKind.STRING + and argument.binding.codegen_action is CodegenAction.COPY_IN_OUT + ) + diagnostics = [] + if replacements and plan.binding.status_error is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "string-writeback-with-status-error", plan.owner_path)) + for argument in replacements: + diagnostics.extend(self._one_string_writeback_diagnostics(plan, argument)) + return tuple(diagnostics) + + def _one_string_writeback_diagnostics( + self, + plan: FunctionPlan, + argument: ArgumentTransferPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return lifecycle coverage and fact drift for one replacement.""" + actions = tuple( + action for action in plan.writeback_actions if action.source_role == argument.binding.handoff_role + ) + diagnostics = [] + if {action.phase for action in actions} != set(WritebackPhase): + diagnostics.append( + self._diagnostic(plan.owner_path, "incomplete-string-writeback-lifecycle", argument.owner_path) + ) + for action in actions: + diagnostics.extend(self._one_string_writeback_action_diagnostics(plan, argument, action)) + return tuple(diagnostics) + + def _one_string_writeback_action_diagnostics( + self, + plan: FunctionPlan, + argument: ArgumentTransferPlan, + action: LifecycleActionPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Return one lifecycle action's string replacement consistency.""" + if ( + action.codegen_action is CodegenAction.COPY_IN_OUT + and action.object_kind is ObjectKind.STRING + and action.semantic_type_name == "String" + and action.datatype_family is DatatypeFamily.STRING + and action.result_position == argument.result_position + ): + return () + return (self._diagnostic(plan.owner_path, "inconsistent-string-writeback-lifecycle", action.phase.value),) + def _writeback_phase_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Require one complete ordered phase set for every writeback handoff.""" diagnostics = [] grouped: dict[str, list[LifecycleActionPlan]] = {} for action in plan.writeback_actions: grouped.setdefault(action.source_role, []).append(action) - expected = set(WritebackPhase) for source_role, actions in grouped.items(): + expected = ( + {WritebackPhase.COPY_OUT} + if all(action.object_kind is ObjectKind.NUMPY_ARRAY for action in actions) + else set(WritebackPhase) + ) phases = [action.phase for action in actions] counts = Counter(phases) for phase, occurrences in counts.items(): @@ -852,7 +1848,7 @@ def _status_role_diagnostics( status = result_slots.get(policy.status_role) if status is None: return (self._diagnostic(plan.owner_path, "missing-status-result-role", policy.status_role),) - if status.datatype_family is not DatatypeFamily.INTEGER: + if status.object_kind is not ObjectKind.SCALAR or status.datatype_family is not DatatypeFamily.INTEGER: return (self._diagnostic(plan.owner_path, "incompatible-status-result-role", policy.status_role),) if status.semantic_type_name != "Int32": return (self._diagnostic(plan.owner_path, "incompatible-status-result-role", policy.status_role),) @@ -870,7 +1866,7 @@ def _message_role_diagnostics( message = result_slots.get(policy.message_role) if message is None: return (self._diagnostic(plan.owner_path, "missing-message-result-role", policy.message_role),) - if message.datatype_family is not DatatypeFamily.STRING: + if message.object_kind is not ObjectKind.STRING or message.datatype_family is not DatatypeFamily.STRING: return (self._diagnostic(plan.owner_path, "incompatible-message-result-role", policy.message_role),) if message.character_length is None: return (self._diagnostic(plan.owner_path, "incompatible-message-result-role", policy.message_role),) diff --git a/x2py/wrapper_codegen/nodes.py b/x2py/wrapper_codegen/nodes.py index c8f9cae0c..edbbf7b5b 100644 --- a/x2py/wrapper_codegen/nodes.py +++ b/x2py/wrapper_codegen/nodes.py @@ -247,6 +247,22 @@ class FortranIf(StageRecord): else_body: tuple[FortranAssignment | FortranPointerAssignment | FortranCall | FortranIf, ...] = () +@dataclass +class FortranCase(StageRecord): + """One explicit or default branch in a Fortran select-case block.""" + + value: int | None + body: tuple[FortranAssignment | FortranPointerAssignment | FortranCall | FortranIf | FortranSelectCase, ...] = () + + +@dataclass +class FortranSelectCase(StageRecord): + """Fortran select-case dispatch used by planned runtime-rank arrays.""" + + expression: CodeExpression + cases: tuple[FortranCase, ...] = () + + @dataclass class FortranInterfaceProcedure(StageRecord): """One native procedure signature inside an explicit interface.""" @@ -277,7 +293,10 @@ class FortranFunction(StageRecord): result_type: str | None = None bind_name: str | None = None declarations: tuple[FortranDeclaration, ...] = () - body: tuple[FortranAssignment | FortranPointerAssignment | FortranCall | FortranIf, ...] = () + body: tuple[ + FortranAssignment | FortranPointerAssignment | FortranCall | FortranIf | FortranSelectCase, + ..., + ] = () is_subroutine: bool = False diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py index f01080aa5..702b5e91e 100644 --- a/x2py/wrapper_codegen/plan.py +++ b/x2py/wrapper_codegen/plan.py @@ -9,9 +9,14 @@ from x2py.semantics.ownership import ( AssignmentMode, CodegenAction, + DestructionPolicy, NativeBarrierAction, + ObjectKind, + OwnershipOwner, PythonBarrierAction, SetterAction, + StorageMode, + TransferMode, ) from x2py.semantics.wrapper_policy import ( ArgumentHandoffMode, @@ -34,6 +39,26 @@ class DatatypeFamily(Enum): STRING = "string" +@dataclass +class ArrayHandoffPlan(StageRecord): + """Editable ordinary-array storage, layout, and ABI roles.""" + + rank: int | None + shape: tuple[str, ...] + axes: tuple[str, ...] + order: str | None + contiguous: bool | None + itemsize: int | None + category: str | None + data_role: str + extent_roles: tuple[str, ...] + extent_reference_roles: tuple[tuple[str, ...], ...] = () + upper_bound_roles: tuple[str, ...] = () + stride_roles: tuple[str, ...] = () + runtime_rank_role: str | None = None + itemsize_role: str | None = None + + @dataclass class BindingStatusErrorPlan(StageRecord): """Binding-owned post-call native status projection.""" @@ -119,11 +144,13 @@ class BindingArgumentPlan(StageRecord): python_name: str python_action: PythonBarrierAction + codegen_action: CodegenAction handoff_role: str optional_mode: OptionalMode nullable: bool writable: bool descriptor_boundary: bool + length_handoff_role: str | None = None @dataclass @@ -132,6 +159,7 @@ class BridgeArgumentPlan(StageRecord): native_name: str native_action: NativeBarrierAction + codegen_action: CodegenAction handoff_mode: ArgumentHandoffMode data_action: BridgeDataAction copy_reason: str | None @@ -139,6 +167,7 @@ class BridgeArgumentPlan(StageRecord): handoff_role: str optional_mode: OptionalMode presence_role: str | None + length_handoff_role: str | None = None @dataclass @@ -184,7 +213,7 @@ class BridgeLifecyclePlan(StageRecord): @dataclass class NativeCallSlotPlan(StageRecord): - """One native-call slot copied from completed policy.""" + """One ordered ABI slot referenced by its owning transfer when applicable.""" owner_path: str native_position: int @@ -198,23 +227,38 @@ class NativeCallSlotPlan(StageRecord): codegen_action: CodegenAction bridge_data_action: BridgeDataAction bridge_copy_reason: str | None + object_kind: ObjectKind | None literal_type: str | None = None literal_value: Any = None result_position: int | None = None semantic_type_name: str | None = None datatype_family: DatatypeFamily | None = None character_length: int | None = None + array: ArrayHandoffPlan | None = None @dataclass class ArgumentTransferPlan(StageRecord): - """One shared Python-to-native transfer with explicit backend views.""" + """One shared Python-to-native transfer, including its native-call slot.""" owner_path: str python_position: int native_position: int semantic_type_name: str datatype_family: DatatypeFamily + character_length: int | None + object_kind: ObjectKind + ownership_owner: OwnershipOwner + transfer_mode: TransferMode + destruction_policy: DestructionPolicy + storage_mode: StorageMode + boundary_storage_mode: StorageMode + nullable: bool + mutates_native: bool + projects_result: bool + python_visible: bool + result_position: int | None + array: ArrayHandoffPlan | None binding: BindingArgumentPlan bridge: BridgeArgumentPlan native_call_slot: NativeCallSlotPlan @@ -222,13 +266,22 @@ class ArgumentTransferPlan(StageRecord): @dataclass class ResultPlan(StageRecord): - """One shared native-to-Python result with explicit backend views.""" + """One native-to-Python transfer and its hidden ABI slot when applicable.""" owner_path: str semantic_type_name: str datatype_family: DatatypeFamily source_kind: str result_position: int + character_length: int | None + object_kind: ObjectKind + ownership_owner: OwnershipOwner + transfer_mode: TransferMode + destruction_policy: DestructionPolicy + storage_mode: StorageMode + boundary_storage_mode: StorageMode + nullable: bool + array: ArrayHandoffPlan | None binding: BindingResultPlan bridge: BridgeResultPlan native_call_slot: NativeCallSlotPlan | None = None @@ -236,18 +289,23 @@ class ResultPlan(StageRecord): @dataclass class LifecycleActionPlan(StageRecord): - """One ordered lifecycle action with explicit backend ownership.""" + """One transfer-owned action kept in function-wide execution order.""" owner_path: str phase: WritebackPhase source_role: str + codegen_action: CodegenAction + semantic_type_name: str + datatype_family: DatatypeFamily + object_kind: ObjectKind + result_position: int binding: BindingLifecyclePlan | None = None bridge: BridgeLifecyclePlan | None = None @dataclass class FunctionPlan(StageRecord): - """Wrapper plan for one semantic function owner.""" + """Stable orchestration plus ordered ABI and lifecycle transfer indexes.""" owner_path: str symbol_name: str diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py index 97d33dab2..c2efa4cd2 100644 --- a/x2py/wrapper_codegen/planner.py +++ b/x2py/wrapper_codegen/planner.py @@ -6,6 +6,8 @@ from x2py.semantics import models from x2py.semantics.wrapper_policy import ( + ArgumentHandoffMode, + ArrayHandoffPolicy, ModuleGetterAction, ModuleVariablePolicy, ArgumentPolicy, @@ -21,6 +23,7 @@ from x2py.semantics.wrapper_exports import PythonExportPolicy from x2py.semantics.ownership import SetterAction from x2py.wrapper_codegen.plan import ( + ArrayHandoffPlan, ArgumentTransferPlan, BindingArgumentPlan, BindingFunctionPlan, @@ -250,31 +253,51 @@ def _visit_ArgumentPolicy( ) -> ArgumentTransferPlan: """Return one transfer whose backend views share one handoff role.""" role = self._value_role(policy.owner_path) + length_role = ( + f"{policy.owner_path}:length" if policy.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER else None + ) return ArgumentTransferPlan( owner_path=policy.owner_path, python_position=policy.python_position, native_position=policy.native_position, semantic_type_name=policy.semantic_type_name, datatype_family=self._datatype_family(policy.semantic_type_name), + character_length=policy.character_length, + object_kind=policy.ownership.kind, + ownership_owner=policy.ownership.owner, + transfer_mode=policy.ownership.transfer, + destruction_policy=policy.ownership.destruction, + storage_mode=policy.storage_mode, + boundary_storage_mode=policy.boundary_storage_mode, + nullable=policy.nullable, + mutates_native=policy.ownership.mutates_native, + projects_result=policy.projects_result, + python_visible=policy.python_visible, + result_position=policy.result_position, + array=native_slot.array, binding=BindingArgumentPlan( - policy.python_name, - policy.python_barrier_action, - role, - policy.optional_mode, - policy.nullable, - policy.writable, - policy.descriptor_boundary, + python_name=policy.python_name, + python_action=policy.python_barrier_action, + codegen_action=policy.codegen_action, + handoff_role=role, + optional_mode=policy.optional_mode, + nullable=policy.nullable, + writable=policy.writable, + descriptor_boundary=policy.descriptor_boundary, + length_handoff_role=length_role, ), bridge=BridgeArgumentPlan( - policy.native_name, - policy.native_barrier_action, - policy.handoff_mode, - policy.bridge_data_action, - policy.bridge_copy_reason, - native_slot.native_position, - role, - policy.optional_mode, - f"{policy.owner_path}:present" if policy.descriptor_boundary else None, + native_name=policy.native_name, + native_action=policy.native_barrier_action, + codegen_action=policy.codegen_action, + handoff_mode=policy.handoff_mode, + data_action=policy.bridge_data_action, + copy_reason=policy.bridge_copy_reason, + abi_position=native_slot.native_position, + handoff_role=role, + optional_mode=policy.optional_mode, + presence_role=f"{policy.owner_path}:present" if policy.descriptor_boundary else None, + length_handoff_role=length_role, ), native_call_slot=native_slot, ) @@ -283,7 +306,7 @@ def _visit_LifecyclePolicy( self, policy: LifecyclePolicy, ) -> LifecycleActionPlan: - """Return one binding-owned writeback plan.""" + """Return one transfer-owned action for function-wide ordering.""" family = self._datatype_family(policy.semantic_type_name) binding = None bridge = None @@ -304,6 +327,11 @@ def _visit_LifecyclePolicy( owner_path=policy.owner_path, phase=policy.phase, source_role=policy.source_role, + codegen_action=policy.codegen_action, + semantic_type_name=policy.semantic_type_name, + datatype_family=family, + object_kind=policy.object_kind, + result_position=policy.result_position, binding=binding, bridge=bridge, ) @@ -324,6 +352,15 @@ def _visit_ResultPolicy( datatype_family=self._datatype_family(policy.semantic_type_name), source_kind=policy.source_kind, result_position=policy.result_position, + character_length=policy.character_length, + object_kind=policy.ownership.kind, + ownership_owner=policy.ownership.owner, + transfer_mode=policy.ownership.transfer, + destruction_policy=policy.ownership.destruction, + storage_mode=policy.storage_mode, + boundary_storage_mode=policy.boundary_storage_mode, + nullable=policy.ownership.nullable, + array=(native_slot.array if native_slot is not None else self._array_plan(policy.array, policy.owner_path)), binding=BindingResultPlan( policy.codegen_action, policy.python_barrier_action, @@ -342,7 +379,7 @@ def _visit_ResultPolicy( ) def _native_slot_plan(self, slot: NativeCallSlotPolicy, role: str) -> NativeCallSlotPlan: - """Return one native-call slot without selecting backend behavior.""" + """Return one shared ABI slot without selecting backend behavior.""" return NativeCallSlotPlan( owner_path=slot.owner_path, native_position=slot.native_position, @@ -356,14 +393,76 @@ def _native_slot_plan(self, slot: NativeCallSlotPolicy, role: str) -> NativeCall codegen_action=slot.codegen_action, bridge_data_action=slot.bridge_data_action, bridge_copy_reason=slot.bridge_copy_reason, + object_kind=slot.object_kind, literal_type=slot.literal_type, literal_value=slot.literal_value, result_position=slot.result_position, semantic_type_name=slot.semantic_type_name, datatype_family=(self._datatype_family(slot.semantic_type_name) if slot.semantic_type_name else None), character_length=slot.character_length, + array=self._array_plan(slot.array, slot.owner_path), + ) + + # Ordinary-array planning. + def _array_plan( + self, + policy: ArrayHandoffPolicy | None, + owner_path: str, + ) -> ArrayHandoffPlan | None: + """Mechanically add ABI role names to completed array facts.""" + if policy is None: + return None + abi_rank = self._array_abi_rank(policy) + return ArrayHandoffPlan( + rank=policy.rank, + shape=policy.shape, + axes=policy.axes, + order=policy.order, + contiguous=policy.contiguous, + itemsize=policy.itemsize, + category=policy.category, + data_role=self._value_role(owner_path), + extent_roles=tuple(f"{owner_path}:extent:{axis}" for axis in range(abi_rank)), + extent_reference_roles=self._array_extent_reference_roles(owner_path, policy.extent_references), + upper_bound_roles=self._array_layout_roles(owner_path, abi_rank, policy.contiguous, "upper-bound"), + stride_roles=self._array_layout_roles(owner_path, abi_rank, policy.contiguous, "stride"), + runtime_rank_role=self._array_runtime_rank_role(policy, owner_path), + itemsize_role=self._array_itemsize_role(policy, owner_path), ) + def _array_abi_rank(self, policy: ArrayHandoffPolicy) -> int: + """Return the concrete ABI field count for fixed or assumed rank.""" + return 15 if policy.rank is None else policy.rank + + def _array_runtime_rank_role(self, policy: ArrayHandoffPolicy, owner_path: str) -> str | None: + """Name the runtime-rank role only for assumed-rank arrays.""" + return f"{owner_path}:rank" if policy.rank is None else None + + def _array_itemsize_role(self, policy: ArrayHandoffPolicy, owner_path: str) -> str | None: + """Name the itemsize role only for fixed-width character arrays.""" + return f"{owner_path}:itemsize" if policy.itemsize is not None else None + + def _array_layout_roles( + self, + owner_path: str, + rank: int, + contiguous: bool | None, + label: str, + ) -> tuple[str, ...]: + """Name one ABI role per axis only for stride-aware layouts.""" + if contiguous is not False: + return () + return tuple(f"{owner_path}:{label}:{axis}" for axis in range(rank)) + + def _array_extent_reference_roles( + self, + owner_path: str, + references: tuple[tuple[str, ...], ...], + ) -> tuple[tuple[str, ...], ...]: + """Resolve completed extent names to existing argument handoff roles.""" + function_path = owner_path.rsplit(".", 1)[0] + return tuple(tuple(f"{function_path}.{name}:value" for name in axis) for axis in references) + def _status_error_plan( self, policy: NativeStatusErrorPolicy | None, diff --git a/x2py/wrapper_codegen/source_printers.py b/x2py/wrapper_codegen/source_printers.py index 87ff882b5..4f262a213 100644 --- a/x2py/wrapper_codegen/source_printers.py +++ b/x2py/wrapper_codegen/source_printers.py @@ -31,6 +31,7 @@ FortranModule, FortranParameter, FortranPointerAssignment, + FortranSelectCase, FortranUse, ) from x2py.stage_values import StageRecord @@ -261,6 +262,8 @@ def _indented(self, text: str) -> str: class FortranSourcePrinter(ClassVisitor): """Print isolated Fortran source nodes.""" + _LINE_LIMIT = 124 + def doprint(self, node: object) -> str: """Render one isolated Fortran backend node.""" if isinstance(node, StageRecord): @@ -321,7 +324,10 @@ def _visit_FortranPointerAssignment(self, node: FortranPointerAssignment) -> str def _visit_FortranCall(self, node: FortranCall) -> str: """Render one Fortran call statement.""" - return f"call {node.function_name}({', '.join(argument.text for argument in node.arguments)})" + return self._continued_call( + f"call {node.function_name}(", + tuple(argument.text for argument in node.arguments), + ) def _visit_FortranIf(self, node: FortranIf) -> str: """Render one Fortran conditional statement.""" @@ -333,6 +339,16 @@ def _visit_FortranIf(self, node: FortranIf) -> str: lines.append("end if") return "\n".join(lines) + def _visit_FortranSelectCase(self, node: FortranSelectCase) -> str: + """Render runtime-rank dispatch without hiding branch structure.""" + lines = [f"select case ({node.expression.text})"] + for case in node.cases: + selector = "default" if case.value is None else f"({case.value})" + lines.append(f"case {selector}") + lines.extend(self._indented(self.visit(statement)) for statement in case.body) + lines.append("end select") + return "\n".join(lines) + def _visit_FortranInterface(self, node: FortranInterface) -> str: """Render one explicit interface block.""" lines = ["interface"] @@ -342,11 +358,16 @@ def _visit_FortranInterface(self, node: FortranInterface) -> str: def _visit_FortranInterfaceProcedure(self, node: FortranInterfaceProcedure) -> str: """Render one native procedure declaration inside an interface.""" - arguments = ", ".join(parameter.name for parameter in node.parameters) kind = "subroutine" if node.is_subroutine else "function" suffix = f" result({node.result_name})" if node.result_name is not None else "" binding = f' bind(c, name="{node.bind_name}")' if node.bind_name is not None else "" - lines = [f"{kind} {node.name}({arguments}){binding}{suffix}"] + lines = [ + self._continued_call( + f"{kind} {node.name}(", + tuple(parameter.name for parameter in node.parameters), + suffix=f"){binding}{suffix}", + ) + ] if node.imports: lines.append(self._indented(f"import :: {', '.join(node.imports)}")) lines.extend(self._indented(self.visit(parameter)) for parameter in node.parameters) @@ -357,11 +378,109 @@ def _visit_FortranInterfaceProcedure(self, node: FortranInterfaceProcedure) -> s def _function_signature(self, node: FortranFunction) -> str: """Render a Fortran function signature.""" - args = ", ".join(parameter.name for parameter in node.parameters) suffix = f" result({node.result_name})" if node.result_name is not None else "" bind = f' bind(c, name="{node.bind_name}")' if node.bind_name is not None else "" kind = "subroutine" if node.is_subroutine else "function" - return f"{kind} {node.name}({args}){suffix}{bind}" + return self._continued_call( + f"{kind} {node.name}(", + tuple(parameter.name for parameter in node.parameters), + suffix=f"){suffix}{bind}", + ) + + def _continued_call( + self, + prefix: str, + arguments: tuple[str, ...], + *, + suffix: str = ")", + ) -> str: + """Wrap one comma-separated Fortran argument list with continuations.""" + rendered = f"{prefix}{', '.join(arguments)}{suffix}" + if len(rendered) <= self._LINE_LIMIT: + return rendered + + lines = [f"{prefix}&"] + last_argument_index = len(arguments) - 1 + for argument_index, argument in enumerate(arguments): + lines.extend( + self._continued_argument_lines( + argument, + last_argument=argument_index == last_argument_index, + suffix=suffix, + ) + ) + return "\n".join(lines) + + def _continued_argument_lines( + self, + argument: str, + *, + last_argument: bool, + suffix: str, + ) -> tuple[str, ...]: + """Wrap one outer-call argument without interpreting semantic policy.""" + array_items = self._array_constructor_items(argument) + if array_items is not None: + return self._continued_array_constructor_lines(array_items, last_argument, suffix) + parenthesized_items = self._parenthesized_items(argument) + if parenthesized_items is not None and len(f" & {argument}") > self._LINE_LIMIT: + return self._continued_parenthesized_lines(parenthesized_items, last_argument, suffix) + ending = suffix if last_argument else ", &" + return (f" & {argument}{ending}",) + + def _continued_array_constructor_lines( + self, + items: tuple[str, ...], + last_argument: bool, + suffix: str, + ) -> tuple[str, ...]: + """Wrap one simple Fortran array constructor item by item.""" + lines = [] + last_item_index = len(items) - 1 + for item_index, item in enumerate(items): + opening = "[" if item_index == 0 else "" + closing = "]" if item_index == last_item_index else "" + ending = self._continued_item_ending(item_index == last_item_index, last_argument, suffix) + lines.append(f" & {opening}{item}{closing}{ending}") + return tuple(lines) + + def _continued_parenthesized_lines( + self, + expression: tuple[str, tuple[str, ...]], + last_argument: bool, + suffix: str, + ) -> tuple[str, ...]: + """Wrap one nested array section or other simple parenthesized value.""" + name, items = expression + lines = [f" & {name}(&"] + last_item_index = len(items) - 1 + for item_index, item in enumerate(items): + closing = ")" if item_index == last_item_index else "" + ending = self._continued_item_ending(item_index == last_item_index, last_argument, suffix) + lines.append(f" & {item}{closing}{ending}") + return tuple(lines) + + def _continued_item_ending(self, last_item: bool, last_argument: bool, suffix: str) -> str: + """Return the continuation or outer-call ending for one nested item.""" + if not last_item: + return ", &" + return suffix if last_argument else ", &" + + def _array_constructor_items(self, expression: str) -> tuple[str, ...] | None: + """Return simple array-constructor items that need their own lines.""" + if not (expression.startswith("[") and expression.endswith("]") and "," in expression): + return None + return tuple(item.strip() for item in expression[1:-1].split(",")) + + def _parenthesized_items(self, expression: str) -> tuple[str, tuple[str, ...]] | None: + """Return simple parenthesized items that need continuation lines.""" + opening = expression.find("(") + if opening < 1 or not expression.endswith(")"): + return None + items = tuple(item.strip() for item in expression[opening + 1 : -1].split(",")) + if len(items) < 2: + return None + return expression[:opening], items def _declaration(self, type_name: str, name: str, attributes: tuple[str, ...]) -> str: """Render a Fortran declaration.""" diff --git a/x2py/wrapper_codegen/support.py b/x2py/wrapper_codegen/support.py index 8ed18c9e2..b6dab1b6f 100644 --- a/x2py/wrapper_codegen/support.py +++ b/x2py/wrapper_codegen/support.py @@ -3,11 +3,8 @@ from __future__ import annotations from x2py.semantics import models -from x2py.semantics.wrapper_policy import ( - ModuleVariablePolicy, - FunctionWrapperPolicy, -) -from x2py.semantics.ownership import PythonBarrierAction +from x2py.semantics.ownership import ObjectKind, PythonBarrierAction +from x2py.semantics.wrapper_policy import FunctionWrapperPolicy, ModuleVariablePolicy from x2py.wrapper_codegen.plan import WrapperPlanSupportBlocker, WrapperPlanSupportReport from x2py.wrapper_codegen.visitor import ClassVisitor @@ -60,10 +57,30 @@ def _visit_SemanticFunction(self, function: models.SemanticFunction) -> WrapperP reason="missing completed function wrapper policy", ) return WrapperPlanSupportReport(owner_path=function.name, blockers=(blocker,)) + capability_blockers = self._function_capability_blockers(function, policy) + blockers = ( + *capability_blockers, + *(WrapperPlanSupportBlocker(policy.owner_path, reason) for reason in policy.blockers), + ) return WrapperPlanSupportReport( owner_path=policy.owner_path, - covered_lanes=self._function_lanes(policy), - blockers=tuple(WrapperPlanSupportBlocker(policy.owner_path, reason) for reason in policy.blockers), + covered_lanes=() if blockers else self._function_lanes(policy), + blockers=blockers, + ) + + def _function_capability_blockers( + self, + function: models.SemanticFunction, + policy: FunctionWrapperPolicy, + ) -> tuple[WrapperPlanSupportBlocker, ...]: + """Keep source ABI optimizations on legacy until direct lowering owns them.""" + if not function.metadata.get("fortran_bind_c"): + return () + return ( + WrapperPlanSupportBlocker( + owner_path=policy.owner_path, + reason="existing bind(C) direct-symbol calls are not implemented by wrapper-plan lowering", + ), ) def _module_blockers(self, module: models.SemanticModule) -> tuple[WrapperPlanSupportBlocker, ...]: @@ -93,34 +110,138 @@ def _function_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: def _argument_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: """Return input-related lanes selected by completed arguments.""" - actions = {argument.python_barrier_action for argument in policy.arguments} + return ( + *self._scalar_argument_lanes(policy), + *self._string_argument_lanes(policy), + *self._array_argument_lanes(policy), + ) + + def _output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return result and writeback lanes selected by completed output policy.""" + lanes = [ + *self._scalar_output_lanes(policy), + *self._string_output_lanes(policy), + *self._array_output_lanes(policy), + ] + if not policy.results and not policy.writeback_actions: + lanes.append("void-calls") + return tuple(lanes) + + # Scalar lane classification. + def _scalar_argument_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return scalar input, address, optional, and descriptor lanes.""" + arguments = tuple(argument for argument in policy.arguments if argument.ownership.kind is ObjectKind.SCALAR) + actions = {argument.python_barrier_action for argument in arguments} lanes = [] if PythonBarrierAction.SCALAR_VALUE in actions: lanes.append("scalar-inputs") if PythonBarrierAction.SCALAR_STORAGE in actions: lanes.append("scalar-storage-inputs") - if PythonBarrierAction.RAW_ADDRESS in actions: + if self._has_scalar_raw_address(policy): lanes.append("scalar-raw-address-inputs") - if any(argument.optional for argument in policy.arguments): + if self._has_scalar_optional(policy): lanes.append("scalar-optional-inputs") - if any(argument.descriptor_boundary for argument in policy.arguments): + if any(argument.descriptor_boundary for argument in arguments): lanes.append("scalar-descriptor-inputs") return tuple(lanes) - def _output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: - """Return result and writeback lanes selected by completed output policy.""" + def _has_scalar_raw_address(self, policy: FunctionWrapperPolicy) -> bool: + """Return whether one scalar argument crosses as a raw address.""" + return any( + argument.ownership.kind is ObjectKind.SCALAR + and argument.python_barrier_action is PythonBarrierAction.RAW_ADDRESS + for argument in policy.arguments + ) + + def _has_scalar_optional(self, policy: FunctionWrapperPolicy) -> bool: + """Return whether one non-string scalar argument is optional.""" + return any(argument.optional and argument.ownership.kind is ObjectKind.SCALAR for argument in policy.arguments) + + def _scalar_output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return scalar writeback and result-source lanes.""" lanes = [] - if policy.writeback_actions: + if any(action.object_kind is ObjectKind.SCALAR for action in policy.writeback_actions): lanes.append("scalar-writebacks") - source_kinds = {result.source_kind for result in policy.results} + results = tuple(result for result in policy.results if result.ownership.kind is ObjectKind.SCALAR) + lanes.extend(self._result_source_lanes(results, prefix="scalar")) + if len(results) > 1: + lanes.append("scalar-multiple-results") + return tuple(lanes) + + # String lane classification. + def _string_argument_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return scalar-string input, address, and optional lanes.""" + actions = { + argument.python_barrier_action + for argument in policy.arguments + if argument.ownership.kind is ObjectKind.STRING + } + lanes = [] + if PythonBarrierAction.STRING_STORAGE in actions: + lanes.append("string-storage-inputs") + if PythonBarrierAction.STRING_VALUE in actions: + lanes.append("string-value-inputs") + if self._has_string_raw_address(policy): + lanes.append("string-raw-address-inputs") + if self._has_string_optional(policy): + lanes.append("string-optional-inputs") + return tuple(lanes) + + def _has_string_raw_address(self, policy: FunctionWrapperPolicy) -> bool: + """Return whether one scalar string crosses as a raw address.""" + return any( + argument.ownership.kind is ObjectKind.STRING + and argument.python_barrier_action is PythonBarrierAction.RAW_ADDRESS + for argument in policy.arguments + ) + + def _has_string_optional(self, policy: FunctionWrapperPolicy) -> bool: + """Return whether one scalar string argument is optional.""" + return any(argument.optional and argument.ownership.kind is ObjectKind.STRING for argument in policy.arguments) + + def _string_output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return scalar-string writeback and result-source lanes.""" + lanes = [] + if any(action.object_kind is ObjectKind.STRING for action in policy.writeback_actions): + lanes.append("string-writebacks") + results = tuple(result for result in policy.results if result.ownership.kind is ObjectKind.STRING) + lanes.extend(self._result_source_lanes(results, prefix="fixed-string")) + return tuple(lanes) + + # Ordinary-array lane classification. + def _array_argument_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return ordinary-array buffer, address, and optional lanes.""" + arguments = tuple( + argument for argument in policy.arguments if argument.ownership.kind is ObjectKind.NUMPY_ARRAY + ) + lanes = [] + if any(argument.python_barrier_action is PythonBarrierAction.ARRAY_STORAGE for argument in arguments): + lanes.append("array-buffer-inputs") + if any(argument.python_barrier_action is PythonBarrierAction.RAW_ADDRESS for argument in arguments): + lanes.append("array-raw-address-inputs") + if any(argument.optional for argument in arguments): + lanes.append("array-optional-inputs") + return tuple(lanes) + + def _array_output_lanes(self, policy: FunctionWrapperPolicy) -> tuple[str, ...]: + """Return ordinary-array writeback and result-source lanes.""" + lanes = [] + if any(action.object_kind is ObjectKind.NUMPY_ARRAY for action in policy.writeback_actions): + lanes.append("array-writebacks") + results = tuple(result for result in policy.results if result.ownership.kind is ObjectKind.NUMPY_ARRAY) + lanes.extend(self._result_source_lanes(results, prefix="array")) + if results and len(policy.results) > 1: + lanes.append("array-multiple-results") + return tuple(lanes) + + def _result_source_lanes(self, results: tuple, *, prefix: str) -> tuple[str, ...]: + """Return direct and hidden lane labels for one result family.""" + source_kinds = {result.source_kind for result in results} + lanes = [] if "direct_return" in source_kinds: - lanes.append("scalar-direct-results") + lanes.append(f"{prefix}-direct-results") if "hidden_output" in source_kinds: - lanes.append("scalar-hidden-outputs") - if len(policy.results) > 1: - lanes.append("scalar-multiple-results") - if not policy.results and not policy.writeback_actions: - lanes.append("void-calls") + lanes.append(f"{prefix}-hidden-outputs") return tuple(lanes) def _module_lanes( From 85be1ae4f97b73856c66bfa80e4ad59b5a49280a Mon Sep 17 00:00:00 2001 From: said Date: Tue, 14 Jul 2026 18:39:10 +0100 Subject: [PATCH 10/30] implement alloctable and pointer arrays --- .../roadmap/native-array-handle-checklist.md | 16 +- .../wrapper-plan-migration-checklist.md | 1141 +++++- docs/user/guide/allocatables.md | 37 +- docs/user/guide/arrays.md | 25 + docs/user/guide/data-types.md | 6 +- .../guide/editing-semantic-pyi-contracts.md | 4 + docs/user/guide/fortran-wrapper.md | 12 + docs/user/reference/semantic-pyi-format.md | 43 +- tests/docs/test_structure.py | 8 + .../test_wrapper_plan_route_selection.py | 62 +- .../runtime/handles/test_array_actual_abi.py | 6 +- .../handles/test_factories_and_lifecycle.py | 49 + .../conversion/c/test_types_and_constants.py | 2 + ...an_conversion_procedures_and_interfaces.py | 42 + .../fortran/test_types_and_storage.py | 2 + .../conversion/pyi/test_types_and_values.py | 28 + tests/semantics/policy/test_wrapper_policy.py | 111 +- tests/wrapper/CHECKLIST_COVERAGE.md | 12 +- .../fortran/arrays/test_array_results.py | 45 +- .../fortran/derived_types/test_pointers.py | 82 + .../test_native_order_contracts.py | 157 + .../function_calls/test_optional_arguments.py | 99 + .../test_allocatable_replacement.py | 70 + .../scalars/test_scalar_boundary_plan.py | 65 + .../fortran/scalars/test_verified_baseline.py | 32 +- .../strings/test_character_arguments.py | 169 + .../wrapper_codegen/test_phase0b_contracts.py | 2 + .../wrapper_codegen/test_phase0d_plan_core.py | 1 + ...st_phase3_scalar_presence_and_writeback.py | 32 + .../test_phase5c_fixed_string_writeback.py | 2 +- .../test_phase6a_array_buffers.py | 17 +- .../test_phase6b_dense_array_shapes.py | 158 +- .../test_phase6c_strided_arrays.py | 3 +- .../test_phase6e_array_results.py | 20 +- ...ase6f_optional_assumed_character_arrays.py | 13 +- .../test_phase6g_raw_array_addresses.py | 182 + .../test_phase7_native_array_handles.py | 268 ++ x2py/codegen/bindings/c_to_python.py | 12 +- x2py/codegen/bridges/fortran_to_c.py | 26 +- x2py/codegen/printers/fcode.py | 2 +- x2py/codegen/printers/pyi_printer.py | 2 + x2py/contracts/__init__.py | 2 + x2py/pipeline/build.py | 47 +- x2py/pipeline/wrapper_artifacts.py | 1 + x2py/runtime/handles.py | 81 +- x2py/semantics/fortran2ir.py | 41 + x2py/semantics/models.py | 2 + x2py/semantics/ownership.py | 61 +- x2py/semantics/policy_completion.py | 9 +- x2py/semantics/pyi2ir.py | 21 + x2py/semantics/wrapper_policy.py | 1122 +++++- x2py/wrapper_codegen/__init__.py | 14 + x2py/wrapper_codegen/c/binding.py | 3248 +++++++++++++++-- x2py/wrapper_codegen/fortran/bridge.py | 1268 ++++++- x2py/wrapper_codegen/generator.py | 1085 +++++- x2py/wrapper_codegen/nodes.py | 43 +- x2py/wrapper_codegen/plan.py | 117 +- x2py/wrapper_codegen/planner.py | 664 +++- .../wrapper_codegen/primitive_scalar_types.py | 9 + x2py/wrapper_codegen/source_printers.py | 24 + x2py/wrapper_codegen/support.py | 137 +- 61 files changed, 10399 insertions(+), 662 deletions(-) create mode 100644 tests/wrapper_codegen/test_phase6g_raw_array_addresses.py create mode 100644 tests/wrapper_codegen/test_phase7_native_array_handles.py diff --git a/docs/maintainer/roadmap/native-array-handle-checklist.md b/docs/maintainer/roadmap/native-array-handle-checklist.md index 28fac8be1..d51bedbc1 100644 --- a/docs/maintainer/roadmap/native-array-handle-checklist.md +++ b/docs/maintainer/roadmap/native-array-handle-checklist.md @@ -652,11 +652,17 @@ specialize operation bodies by descriptor kind. - [x] Generate owned allocatable function-result handles when policy supports stable owner storage. - [x] Use wrapper-owned standard C descriptor storage for allocatable results: - allocate persistent rank-specific `CFI_CDESC_T(rank)` storage, establish it - with allocatable attribute, and allocate its payload with `CFI_allocate`. -- [x] Copy the bridge-local native allocatable result or hidden output into the - persistent CFI allocation before releasing the bridge-local storage. Do not - return or copy a compiler-private Fortran descriptor record. + allocate persistent rank-specific `CFI_CDESC_T(rank)` storage and establish + it with allocatable attribute. Numeric function results populate local + allocatable storage whose allocation is transferred with `move_alloc`; + generated shape-changing operations use `CFI_allocate`. +- [x] Assign a numeric direct allocatable function result once into a + bridge-local allocatable, then `move_alloc` that allocation into the + allocatable `intent(out)` dummy backed by persistent CFI storage. Do not + generate a collector, an `allocated(...)` guard, or a second intrinsic + assignment. The native function must return an allocated, defined result; an + unallocated nonpointer result is a nonconforming native procedure and remains + the user's responsibility. - [x] Return a native pointer to owner storage for owned allocatable handles. - [x] Generate destroy routines called by the Python handle finalizer for owned allocatable handles. diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 5e96c06b7..d86fecbbf 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -70,10 +70,12 @@ ModulePlan binding: BindingArgumentPlan bridge: BridgeArgumentPlan native_call_slot: NativeCallSlotPlan + transformations: TransformationPlan ... results: ResultPlan ... binding: BindingResultPlan bridge: BridgeResultPlan native_call_slot: NativeCallSlotPlan | None + transformations: TransformationPlan ... lifecycle: LifecycleActionPlan ... binding: BindingLifecyclePlan | None bridge: BridgeLifecyclePlan | None @@ -118,6 +120,14 @@ ABI whose handoff plan carries data, rank, extents, strides, and itemsize. native descriptors and handles introduced in Phase 7. Neither backend may use one action as a fallback for the other. +`NativeBarrierAction.PASS_RAW_ADDRESS` is the third, deliberately narrower +array transport: one caller-supplied opaque address plus separately completed +pointee rank, shape, element type, and orientation facts. It does not authorize +NumPy extraction, a packed array-buffer ABI, or a native descriptor. Scalar, +fixed-string, and array raw addresses reuse this action and +`ArgumentHandoffMode.OPAQUE_ADDRESS`; their object kind then selects the named +bridge association method. Do not add datatype-specific raw-address actions. + The binding input and bridge input may have different representations: a C binding commonly receives `PyObject *`, produces a C scalar or address, and the bridge then consumes a value or pointer according to its ABI slot. The plan @@ -133,6 +143,54 @@ An owner may have only one active backend side when completed policy places the behavior entirely in one backend, but that ownership must be explicit in the plan rather than inferred during lowering. +### Transformation Layer Ownership + +Every representation transformation has one explicit +`TransformationPlan.layer`: `BINDING` or `BRIDGE`. The record also names its +phase (`COPY_IN`, `NATIVE_MUTATION`, `COPY_OUT`, or `CLEANUP`), typed action, +source representation, target representation, and reason. These records are +subordinate to the `ArgumentTransferPlan` or `ResultPlan` that owns the value; +they are not a parallel datatype-policy hierarchy. + +Use the binding layer for transformations involving Python objects or NumPy +semantics: dtype/layout conversion, Python encoding/decoding, reference and +identity handling, copy-back into caller objects, and Python-owned temporary +cleanup. Use the bridge layer for transformations wholly between the ABI and a +native-language representation: Fortran character representation, native +descriptor/result materialization, derived native layout, or native-only +allocation and copy. + +One logical conversion and its inverse/cleanup must stay at one layer. If a +workflow genuinely needs both layers, policy completion records two distinct +transformations separated by a named intermediate ABI representation. A +backend consumes only transformations assigned to it and fails validation if +asked to lower an action owned by the other backend. Method location, datatype, +`intent`, and available local storage never select the transformation layer. +For `COPY_F`, copy-in, conditional copy-out, and cleanup are all binding-owned; +the bridge has no `COPY_F` transformation and reuses its ordinary ORDER_F +association path. + +### Editable Signature And Native Intent Boundary + +The semantic `.pyi` signature is authoritative for the Python-facing call +shape. The source converter may use native `intent` to propose the initial +generated signature, but that proposal is not backend policy. A user may +reorder visible Python arguments, keep a native output dummy as caller-supplied +storage, project it into a Python result, or introduce hidden bridge storage. +After the semantic contract is constructed or edited, completed native-call +slots must account for every required native position exactly once; stored +source `intent` must not silently override that mapping by hiding, exposing, +reordering, allocating, or projecting a Python value. + +Bridge dummies and backend-local variables use the most permissive declaration +that is compatible with the selected ABI, normally no `intent` or an internal +`intent(inout)`-equivalent writable local. The called native procedure enforces +its actual `intent(in)`, `intent(out)`, or `intent(inout)` contract. Required +interoperability attributes such as `value`, optional presence fields, standard +descriptor attributes, and true bridge-output parameters remain explicit ABI +facts; they are not permission for a backend to reconstruct the user-facing +signature from native `intent`. + The plan includes every fact required for mechanical lowering: - owner path and plan-node kind; @@ -528,10 +586,10 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 78 | +| `wrapper-plan` | 94 | | `dual-route` | 5 | -| `legacy` | 130 | -| `not-applicable` | 95 | +| `legacy` | 123 | +| `not-applicable` | 96 | | `deferred-real-library` | 2 | #### Recorded Route Progression @@ -557,6 +615,10 @@ blockers. | Phase 5D fixed string storage/raw addresses | 72 | 0 | 134 | 95 | 2 | 303 | | Phase 5 production route reconciliation | 76 | 0 | 130 | 95 | 2 | 303 | | Phase 6 ordinary arrays | 78 | 5 | 130 | 95 | 2 | 310 | +| Phase 6G raw array addresses | 80 | 5 | 130 | 95 | 2 | 312 | +| Phase 6 `COPY_F` representation copy | 81 | 5 | 130 | 95 | 2 | 313 | +| Phase 7 native handles/descriptors | 88 | 5 | 129 | 96 | 2 | 320 | +| Phase 7 production route reconciliation | 94 | 5 | 123 | 96 | 2 | 320 | Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating @@ -575,8 +637,9 @@ already covered by the new generator. | --- | --- | --- | --- | | `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | | `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | source/generated-.pyi parity | ordinary arrays; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | production plan route in source/generated-.pyi parity modes | fixed/runtime-shape ordinary array results; owned allocatable descriptor results; namespace preservation | `wrapper-plan` | | `tests/wrapper/fortran/arrays/test_array_results.py::test_ordinary_array_results_match_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape ordinary array results; ranks one through fifteen; Fortran order; zero-sized results; allocation/copy/release failure paths | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes` | reduced owned-result contract with deliberate legacy rollback comparison | allocated and zero-sized wrapper-owned `CFI_CDESC_T` function-result handles; extraction and release | `wrapper-plan` | | `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | | `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | | `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing assumed-rank native unit | runtime ranks one through fifteen; mutable storage; rank validation; native-handle actuals deferred to Phase 7 | `dual-route` | @@ -609,7 +672,7 @@ already covered by the new generator. | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_requires_a_native_link_input` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | non-generating: manifest serialization unit | completed native-array build requirements and local standard-descriptor headers | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_a_missing_native_artifact` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_address_contracts_before_codegen[*]` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | @@ -646,9 +709,12 @@ already covered by the new generator. | `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes` | reduced module-only contract with deliberate legacy rollback comparison | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/snapshots; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; native-call projections; arrays and derived types deferred to later lanes | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; raw array addresses completed by Phase 6G; derived result remains Phase 8 | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entries over existing vector/matrix native routines | raw numeric addresses; visible scalar-storage extents; rank one/two; default C and explicit Fortran orientation; mutation; integer-only conversion | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_copy_f_preserves_logical_axes_through_binding_owned_temporary` | reduced edited semantic `.pyi` entries over the existing matrix native routine | explicit C-to-Fortran representation copy; native-input and inout calls; projected original identity; binding-owned copyback and cleanup | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fnative_call_examples_f90` native unit | fixed mutable rank-zero NumPy bytes storage; raw fixed-string addresses; in-place mutation; rank/dtype/itemsize/writability/type validation | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `not-applicable` | @@ -673,14 +739,19 @@ already covered by the new generator. | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | production plan route with deliberate legacy rollback comparison | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes` | reduced optional descriptor contract with deliberate legacy rollback comparison | omitted/`None` absence; present unallocated/unassociated and allocated/associated handle states; kind/dtype validation | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `dual-route` | | `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::*` | source/generated-.pyi parity or parametrized route | module variables/state; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_replacement_has_no_native_memory_errors[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes` | reduced owned-result plus projected-descriptor contract with deliberate legacy rollback comparison | direct persistent descriptor mutation; allocation/reallocation/deallocation; same-handle result identity | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | source/generated-.pyi parity with one mixed generation unit | derived class/field handles and parent retention remain Phase 8/9 blockers | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | production plan route in source/generated-.pyi parity modes | rank-zero allocatable/pointer arguments, writeback, results, and copied nullable module values | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction[*]` | production plan route in source/generated-.pyi parity modes | detached read-only snapshots of plain allocatable module arrays; capsule-owned lifetime | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | scalar calls with internal common-block storage | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/snapshots | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | @@ -727,7 +798,10 @@ already covered by the new generator. | `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | edited semantic `.pyi` contract | strings; fixed/assumed inputs; arrays; mutable string storage | `legacy` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | production plan route from source/generated-.pyi parity | fixed-form strings; fixed/assumed inputs; fixed results | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity | strings; fixed/assumed inputs; fixed/deferred results; arrays; writeback | `legacy` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes` | reduced scalar descriptor result contract with deliberate legacy rollback comparison | runtime length; nullable copy-out; UTF-8 data; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes` | reduced descriptor-result and projected-descriptor contract with deliberate legacy rollback comparison | hidden/direct owned deferred-character arrays; runtime `S3`/`S4`/`S5` width; projected identity; nullable rank-zero result | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | fixed-width `NPY_STRING` array itemsize; rank/dtype/zero-size validation; native-handle actuals deferred to Phase 7 | `dual-route` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fstrings_f90` native unit | raw fixed-width character array address; literal shape; element length; integer-only conversion | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | required fixed/assumed scalar string inputs; default/kind-1/`c_char`; UTF-8 length and NUL validation; scalar results | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_string_results_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | direct fixed string results; trailing blanks; default/`c_char`; allocation failure | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy[*]` | production plan route from source/generated-.pyi parity | strings; fixed/assumed input/output; optional presence; Unicode/NUL handling | `wrapper-plan` | @@ -1603,6 +1677,8 @@ The ordinary-array lane borrows or copies NumPy data buffers; it never creates or consumes a persistent native descriptor handle. `Allocatable[T[...]]`, `Pointer[T[...]]`, rank-zero allocatable/pointer scalars, and a native handle used as the actual value for an ordinary array dummy all remain in Phase 7. +Caller-supplied `Addr(T[...])` storage is the distinct Phase 6G follow-up and +must complete before Phase 7. Derived-type arrays remain in Phase 8, fields in Phases 8 and 9, and callback arrays in Phase 10. The full BLAS/LAPACK generation unit remains deferred until final cutover even when individual ordinary-array shapes become supported. @@ -1689,8 +1765,39 @@ bridge association order follows the completed layout. Any expression that cannot be represented by available roles remains blocked rather than being recomputed in a backend. +Order is an exact-storage selector, not an implicit conversion selector. +`ORDER_F` preserves logical axes over Fortran-contiguous storage. `ORDER_C` +passes the original C-contiguous address and reverses bridge extents, so native +Fortran observes the transposed storage view. Preserving the same logical axes +while accepting the opposite layout uses explicit `COPY_F` metadata, never an +inference from order. The owning `ArgumentTransferPlan` records C source order, +F native order, copy-in, conditional copy-out, original-object projection, and +temporary cleanup. The binding performs both copy directions and owns the +NumPy temporary. The bridge receives the temporary through the unchanged +ORDER_F association path and performs neither half of this representation +conversion. + +The initial `COPY_F` lane includes required, concrete-rank, dense numeric +ndarray arguments. It excludes `Flat`, assumed-rank, strided, optional and +character arrays, native descriptor arguments, and handle actuals until each +has separate policy and parity evidence. + +`Flat` is one axis marker and never collapses a multidimensional plan: +`T[:, Flat]` remains rank two in Fortran order, while +`Annotated[T[Flat, :], ORDER_C]` is its C-order orientation. The bridge reverses +only the C-order association extents. For an external assumed-size interface, +an explicit prefix such as `T[3, Flat]` may lower to `a(3, *)`; a runtime-only +prefix uses the standards-valid sequence-associated `a(*)` declaration while +the bridge retains every runtime extent and the completed logical rank. + - [x] Complete declared-shape evaluation, flat-storage orientation, multidimensional dense handoff, validation, parity, and ledger evidence. +- [x] Complete explicit C-to-Fortran representation copies through `COPY_F`, + including native-input and inout calls through the same binding-owned copy + lifecycle, projected original identity, temporary cleanup, direct bridge + reuse, validation, and compiled parity. Native `intent` remains owned by the + called procedure and is not duplicated in the semantic `.pyi` or bridge + temporary. ### Phase 6C — Positive-Strided Ordinary Views @@ -1743,7 +1850,7 @@ NumPy dtype construction and bridge byte-count calculation. - [x] Complete optional presence, assumed-rank dispatch, character itemsize, validation, parity, and ledger evidence. -### Phase 6 Completion +### Phase 6A-F Ordinary-Buffer Completion - [x] Expand the phase under the mandatory expansion gate from live semantic array contracts, legacy binding/bridge lowering, public docs, and focused @@ -1752,36 +1859,1000 @@ NumPy dtype construction and bridge byte-count calculation. order, itemsize, writeability, result, and lifecycle role. - [x] Validate every completed array policy and handoff role before either backend emits source. -- [x] Finish Phase 6 only when every ordinary-array matrix row is migrated or - remains blocked solely by an explicitly later descriptor, derived, field, - callback, or deferred-real-library lane. +- [x] Finish Phases 6A-F only when every ordinary-array buffer matrix row is + migrated or remains blocked solely by an explicitly later descriptor, + derived, field, callback, or deferred-real-library lane. + +### Phase 6G — Raw Array Addresses — Complete + +Implementation status: complete. Required raw array addresses now use the +shared completed policy, `ArgumentTransferPlan`, native slot, centralized +validation, and named binding/bridge lowering paths. The dependency-closed +numeric and fixed-character runtime rows have passed compiled legacy/direct +parity and moved to `wrapper-plan`. + +Scope: required Python-visible type-level raw-address array arguments such as +`Addr(Float64[n])`. The caller supplies one Python integer address, x2py +forwards it as one opaque C address, and the bridge associates a typed native +array view using rank, shape, element type, and orientation facts completed +before `ir2ast.py`. There is no NumPy object, runtime handle, persistent native +descriptor, data copy, ownership transfer, or automatic release. + +This lane follows Phase 6 because its semantic object kind is +`ObjectKind.NUMPY_ARRAY` and its pointee layout reuses the array shape record. +It remains a distinct transport from an ordinary array buffer. The fixed +dispatch algorithm is: + +1. match `ObjectKind.NUMPY_ARRAY`; +2. match the completed Python barrier action; +3. lower `ARRAY_STORAGE` through the Phase 6A-F buffer path or `RAW_ADDRESS` + through Phase 6G; +4. require the matching native action, handoff mode, bridge data action, and + array-shape facts; and +5. fail validation rather than substituting the other transport. + +The same algorithm already separates scalar and string value, storage, and +raw-address forms. Phase 6G must extend that system; it must not add a parallel +raw-pointer planner, a datatype-based backend branch, or a special function or +module plan. + +#### Public Contract And Explicit Non-Scope + +The maintained public contract is already documented in +`docs/user/reference/semantic-pyi-format.md` and +`docs/user/guide/data-types.md`. Preserve it exactly: + +- `Addr(T[d1, ..., dr])` is depth one and has positive rank; +- the pointee dtype is primitive; +- every extent expression is resolved from literals and visible scalar + arguments or visible rank-zero scalar storage; +- the integer carries no dtype, rank, shape, order, alignment, bounds, + ownership, or lifetime metadata; +- x2py cannot prove that the supplied address actually points to compatible, + sufficiently large, live storage; and +- edited semantic `.pyi` raw-address storage is mutable caller storage unless + a completed policy explicitly says otherwise. + +The initial compiled oracle is `Addr(Float64[n])`. Before declaring the lane +complete, audit every public primitive family already accepted by semantic +policy, including bool, integer, real, complex, and fixed-width character +array pointees. Add compiled coverage for a family only when an existing native +routine can prove it without broadening the public contract. A fixed scalar +`Addr(String[n])` remains the completed Phase 5D string path; a rank-positive +`Addr(String[k][n, ...])` is an array path and must carry both the fixed element +length and the resolved array shape. + +Explicitly excluded from this lane are: + +- scalar `Addr(T)`, already completed in Phase 2E; +- fixed scalar `Addr(String[n])`, already completed in Phase 5D; +- NumPy `T[...]` storage, already completed in Phases 6A-F; +- unresolved or assumed shapes such as `Addr(Float64[:])`, assumed rank, + assumed size, and stride-marker shapes; +- optional, nullable, projected, direct-result, and hidden-output raw addresses + unless a separate public-contract audit first proves their intended Python + ownership and absence/result behavior; +- wrapped/derived pointees, pointer graphs deeper than one, and callbacks; +- `Allocatable[T[...]]`, `Pointer[T[...]]`, runtime native handles, and C + descriptors, which belong to Phase 7; and +- any implicit conversion from an ndarray or runtime handle to its address. + +#### One Action Vocabulary, Three Array Transports + +| Contract | Object kind | Python action | Native action | Handoff mode | Bridge data action | +| --- | --- | --- | --- | --- | --- | +| NumPy `T[...]` | `NUMPY_ARRAY` | `ARRAY_STORAGE` | `PASS_ARRAY_BUFFER` | `ARRAY_BUFFER` | `ASSOCIATE_VIEW` | +| Raw `Addr(T[...])` | `NUMPY_ARRAY` | `RAW_ADDRESS` | `PASS_RAW_ADDRESS` | `OPAQUE_ADDRESS` | `ASSOCIATE_VIEW` | +| Native descriptor contract | completed handle kind | completed handle action | `PASS_NATIVE_DESCRIPTOR` | Phase 7 descriptor mode | completed Phase 7 action | + +`ASSOCIATE_VIEW` means the bridge creates a typed, non-owning view; it does not +mean that the Python binding extracted a NumPy buffer. The Python and native +barrier actions remain the authoritative distinction. Do not introduce names +such as `PASS_RAW_ARRAY`, `COPY_RAW_ARRAY`, or datatype-specific address +actions. + +The completed ownership/action tuple for the required mutable public form is: + +- `OwnershipOwner.CALLER`; +- `TransferMode.IN_PLACE`; +- `DestructionPolicy.CALLER`; +- `StorageMode.STACK` for the call-local pointer carrier, not for the pointee; +- `CodegenAction.IN_PLACE_ARGUMENT`; +- `PythonBarrierAction.RAW_ADDRESS`; +- `NativeBarrierAction.PASS_RAW_ADDRESS`; +- `ArgumentHandoffMode.OPAQUE_ADDRESS`; and +- `BridgeDataAction.ASSOCIATE_VIEW` with no copy reason. + +If a retained source-derived contract can be read-only, policy may instead +complete `CALL_LOCAL` / `CALL_LOCAL_INPUT` / `NONE` destruction. Both +backends must consume that completed tuple; neither may infer mutability from +the pointee type or raw-address spelling. A raw array never has copy-in, +copy-out, projected-identity, allocation, destruction, release, or lifecycle +actions in this lane. + +#### Required Policy And Plan Shape + +Keep the feature under the existing `ArgumentTransferPlan`: + +```text +ArgumentTransferPlan + object_kind = NUMPY_ARRAY + binding.python_action = RAW_ADDRESS + bridge.native_action = PASS_RAW_ADDRESS + bridge.handoff_mode = OPAQUE_ADDRESS + bridge.data_action = ASSOCIATE_VIEW + array = ArrayHandoffPlan + native_call_slot = the same referenced NativeCallSlotPlan +``` + +Do not add `RawArrayPlan`, a second native slot, or a raw-address lifecycle +owner. Generalize the existing completed `ArrayHandoffPolicy` and +`ArrayHandoffPlan` only enough to carry raw pointee layout: + +- concrete rank and one shape expression per axis; +- one `data_role` equal to the binding/bridge/native-slot address role; +- `extent_reference_roles` naming the existing visible scalar handoff roles + used by each shape expression; +- the completed orientation used for native pointer association; +- fixed character element length/itemsize when the pointee family is string; + and +- no binding-extracted runtime rank, extent, upper-bound, stride, or itemsize + ABI roles. + +For a raw address, the shape record describes the pointee view; it does not +describe fields packed by the binding. A visible `n` used by +`Addr(Float64[n])` already has its own `ArgumentTransferPlan` and native-call +slot. Reference that role rather than passing a duplicate array extent. A +literal extent requires no extra ABI field. The bridge resolves the shape +expression from those planned native role names. + +Post-IR policy completion must explicitly select multidimensional orientation +before planning. Preserve the current legacy interpretation, including its +default orientation, only after capturing a rank-two artifact/runtime oracle. +Do not leave `ir2ast.py`, a codegen-model `order` default, or the bridge's local +shape reversal to make that decision. + +#### Completed Direct-Plan Seams + +The implementation split completed array policy by Python barrier action, +selected `OPAQUE_ADDRESS` and `ASSOCIATE_VIEW` before lowering, projected raw +pointee layout into the shared array record, omitted packed NumPy-buffer roles, +and added named raw-address checks and association methods to both backends. +Ordinary-array buffer checks remain unchanged and fail closed; neither backend +substitutes one transport for another. + +#### Dependency-Ordered Implementation Slices + +##### Phase 6G1 — Complete Raw Array Policy + +- [x] Make the `NUMPY_ARRAY` boundary validator dispatch on + `PythonBarrierAction` and add a named raw-address branch with the exact + ownership/action tuple above. +- [x] Complete raw pointee rank, shape expressions and their visible-scalar + dependencies, primitive family, fixed character element length, and + orientation before `ir2ast.py`. +- [x] Complete `OPAQUE_ADDRESS` and `ASSOCIATE_VIEW` from the action pair; do + not infer either in a backend. +- [x] Keep unresolved dimensions, unsupported pointee families, optionality, + projection, nullability, and deeper pointer graphs blocked with owner-path + diagnostics. +- [x] Freeze current behavior for zero/negative extent expressions, zero or + negative integer addresses, and integer overflow against the public docs and + legacy conversion before changing any rule. If a rule changes, change it in + policy and public docs, not in one backend. + +Audit result: resolved zero and negative extent expressions remain accepted +without a positivity check; integer zero becomes a null pointer without a +conversion error; negative integers follow `PyLong_AsVoidPtr`; and pointer-size +overflow raises `OverflowError`. Public documentation now states that these are +unsafe caller responsibilities, and tests prove the conversion guard and +generated shape without dereferencing an invalid address. + +##### Phase 6G2 — Project And Validate The Shared Plan + +- [x] Populate the existing `ArgumentTransferPlan.array` and its shared + `NativeCallSlotPlan.array` with one identical raw pointee layout record. +- [x] Reuse the scalar/string address handoff role and + `ArgumentHandoffMode.OPAQUE_ADDRESS`; add no raw-array ABI action. +- [x] Resolve every shape symbol to an existing visible scalar role and reject + unavailable, cyclic, hidden, non-scalar, or result-only dependencies before + lowering. +- [x] Split central array diagnostics by the completed Python action so buffer + validation still requires packed extent/layout roles while raw validation + forbids them. +- [x] Add editable-plan tests that independently corrupt object kind, Python + action, native action, handoff mode, bridge data action, rank, shape, + reference roles, element family, character length, orientation, and native + slot identity. + +##### Phase 6G3 — Reuse Binding Raw-Address Extraction + +- [x] Reuse `_lower_argument_required_raw_address()` for the Python integer + check and `PyLong_AsVoidPtr` conversion. Scalar, string, and array raw + addresses should share this extraction code. +- [x] Emit one `void *` handoff value and no `PyArray_*`, dtype, rank, shape, + layout, writeability, or itemsize checks. +- [x] Keep object-kind-specific logic out of the conversion method; array + shape affects only validation, the bridge view, and native call. +- [x] Preserve the existing conversion rule under which integer zero produces + a null pointer without itself raising a Python conversion error. Prove that + rule without dereferencing the null pointer; runtime tests must never call + native code with an invalid test address. + +##### Phase 6G4 — Add Named Raw Array Bridge Association + +- [x] Add directly named raw-array declaration and association methods in the + array method group. Dispatch to them only for + `NUMPY_ARRAY` / `RAW_ADDRESS` / `PASS_RAW_ADDRESS` / + `OPAQUE_ADDRESS` / `ASSOCIATE_VIEW`. +- [x] Declare one `type(c_ptr), value` bridge parameter and one backend-local + typed pointer view. The local view is an emitted-code helper, not a new plan + owner. +- [x] Associate the view with `c_f_pointer` using only the planned shape and + orientation, then pass that view in the existing native-call slot position. +- [x] Preserve fixed character element length when the pointee is a character + array. Do not pass a runtime itemsize unless a future public contract + explicitly requires one. +- [x] Emit no copy, writeback, allocation, release, descriptor, or NumPy + mechanics. + +##### Phase 6G5 — Prove The Route Before Widening It + +- [x] Retain semantic conversion coverage in + `tests/semantics/conversion/pyi/test_calls_and_projections.py` for round-trip, + visible extent sources, primitive pointees, and rejection of unresolved or + wrapped forms. +- [x] Add focused completed-policy tests for every authoritative action and + blocker, plus `array-raw-address-inputs` support classification. +- [x] Add `tests/wrapper_codegen/test_phase6g_raw_array_addresses.py` for plan + shape, edits, validation, C nodes, Fortran nodes, native order, and the + absence of buffer/descriptor/lifecycle nodes. +- [x] Extract `fill_vector_raw` from + `test_editable_contract_can_use_native_order_arguments_without_native_call` + into a reduced legacy/direct-plan parity test. Cover mutation through a valid + `raw_vector.ctypes.data`, ndarray rejection, wrong Python types, a visible + rank-zero scalar extent, and the established native argument order. +- [x] Prove raw-array native argument reordering in the direct-plan generated + call test. The legacy AST route retains only a projection marker and is not + an oracle for reordered projection-slot lowering. +- [x] Add literal and arithmetic extent-role cases. Add a rank-two runtime + parity case before freezing default/explicit orientation. Add a fixed-width + character-array case if the public family audit retains that contract. +- [x] Keep the broad native-order test `legacy` until its derived-type owner is + migrated; only the reduced raw-array row may move to `wrapper-plan` here. +- [x] Run the focused policy/plan/backend tests, the relevant wrapper test, + documentation checks, wrapper-codegen complexity checker, and required + static-analysis suite before changing route support. + +#### Phase 6G Exit Gate + +- [x] Expand raw array addresses as the explicit next lane using the public + contract, completed semantic policy, legacy binding/bridge primitives, and + the existing compiled `Addr(Float64[n])` oracle. +- [x] Complete Phases 6G1 through 6G5 without changing the public raw-address + contract or introducing a parallel action vocabulary. +- [x] Prove that one maintainer algorithm—object kind, Python action, native + action, handoff mode, data action, then typed shape facts—covers ordinary and + raw arrays without backend inference. +- [x] Move only dependency-closed raw-array test rows after generated-artifact + comparison and compiled legacy/direct parity pass. +- [x] Begin Phase 7 only after this exit gate is complete. Phase 7 must consume + the established distinction among array buffers, raw addresses, and native + descriptors rather than revisiting it. ## Phase 7 — Native Array Handles And Descriptors -Scope: `Allocatable[T[...]]`, `Pointer[T[...]]`, scalar descriptor-backed -values including allocatable or pointer `String`, descriptor-backed handoffs, -and runtime native array handle objects. +Implementation status: complete, pending the final verification gate recorded +at the end of this phase. Phase 6G completed first. The direct route now owns +the dependency-closed Phase 7A-H cases while every field, pointer-result, +callback, and deferred-real-library exclusion remains on its later blocker. + +Scope: migrate the existing native descriptor and runtime-handle contract into +the wrapper-plan path without redefining that public contract. The maintained +`native-array-handle-checklist.md` remains the feature-level behavioral oracle; +this section owns only its migration into completed wrapper policy, +`ArgumentTransferPlan`, `ResultPlan`, `ModuleVariablePlan`, subordinate native +slots and lifecycle actions, direct C/Fortran lowering, and production route +selection. + +The shared descriptor family includes: + +- rank-positive `Allocatable[T[...]]` and `Pointer[T[...]]` handle arguments; +- optional-absent array handles, where omission or `None` means the native + optional dummy is absent; +- projected writable descriptors whose mutation must remain attached to the + same caller handle; +- wrapper-owned allocatable array results and hidden outputs; +- borrowed module allocatable and pointer handles plus their generated operation + tables; +- native handles passed as actual values to ordinary `T[...]` dummies without + an implicit `.to_numpy()` call; +- build requirements for standard C descriptors; and +- the remaining rank-zero allocatable/pointer result cases, including nullable + deferred-length scalar character values, which return copied Python values + rather than native-array handle objects. + +Allocatable and Pointer remain separate public contract types but share one +plan and lowering structure. Descriptor kind selects only the operations that +genuinely differ: allocation state versus association state, allowed +shape-changing operations, target lifetime, extraction policy, and release. +Do not create independent allocatable and pointer planner hierarchies. + +### Phase 7 Boundary And Explicit Non-Scope + +The following four boundaries must remain distinct: + +| Python contract | Planned Python input | Native transport | Owner phase | +| --- | --- | --- | --- | +| `T[...]` with a NumPy array | validated NumPy storage | `PASS_ARRAY_BUFFER` | Phase 6 | +| `T[...]` with an allocated/associated native handle actual | validated handle array-data facet | `PASS_ARRAY_BUFFER` | Phase 7A | +| `Allocatable[T[...]]` / `Pointer[T[...]]` | matching runtime handle object | `PASS_NATIVE_DESCRIPTOR` | Phase 7B onward | +| `Addr(T[n, ...])` | caller-supplied integer address | `PASS_RAW_ADDRESS` | Phase 6G prerequisite, not Phase 7 | + +`Addr(Float64[n])` is a supported public semantic `.pyi` contract when +every extent is a literal or an expression over visible scalar arguments or +rank-zero scalar storage. It accepts an integer such as `array.ctypes.data` and +forwards that address without ownership, dtype, alignment, lifetime, or bounds +validation. Parsing, policy completion, printing, and both compiled wrapper +routes support it through the completed +`RAW_ADDRESS` / `PASS_RAW_ADDRESS` selector pair. Do not misclassify this raw +pointer as a NumPy buffer, native handle, or C descriptor while maintaining +Phase 7. + +Other exclusions and dependencies are: + +- ordinary NumPy-only buffer extraction, shape, stride, output identity, and + copy-result behavior already completed in Phase 6; +- caller-supplied raw array addresses, completed separately by the Phase 6G + entry dependency; +- derived-type field attachment, class construction, parent-wrapper creation, + and property orchestration, which require Phases 8 and 9 even though the + shared native-handle plan must already be reusable by those later owners; +- pointer results without completed stable owner storage and target lifetime; +- callback descriptor arguments or results, which remain in Phase 10; +- compiler-private descriptor layout inspection or copying; +- any implicit `.to_numpy()` conversion when a native handle is passed to an + ordinary array dummy; and +- the deferred BLAS/LAPACK generation unit until final cutover. + +### Existing Semantic Authority And Legacy Oracle + +Do not redesign the public feature while migrating it. Reuse these completed +sources of truth: + +- `x2py/semantics/native_array_handles.py` defines + `NativeArrayHandlePolicy`, `ArrayInteropPolicy`, handle facts, descriptor + kinds, and completed build requirements. +- `x2py/semantics/policy_completion.py` completes handle kind, origin, owner, + owner retention, descriptor ownership, getter/setter behavior, output + projection, release, target lifetime, destruction, extraction, interop, + nullability, storage mode, operations, and blockers before `ir2ast.py`. +- `x2py/runtime/handles.py` owns the reusable runtime protocol, including + `_native_array_actual_argument_for_binding_positional`, + `_native_array_descriptor_argument_for_binding_positional`, and + `_native_array_descriptor_handoff_for_binding_positional`. Direct lowering + must call these helpers rather than duplicate their Python validation. +- `x2py/codegen/bindings/c_to_python.py` is the legacy binding oracle. Its + `_ARRAY_INTEROP_POLICY_DISPATCHER`, `_NATIVE_ARRAY_HANDLE_DISPATCHER`, + descriptor-argument handlers, owned-result handlers, operation wrappers, and + descriptor reader define the currently passing C behavior. +- `x2py/codegen/bridges/fortran_to_c.py` is the legacy bridge oracle. Its + corresponding dispatchers, descriptor-argument handlers, module/field + operation generators, and owned-allocatable result helpers define the + currently passing Fortran behavior. +- `x2py/pipeline/build.py` already derives native-array build requirements from + completed semantic policy and records them in manifests. The wrapper plan + must carry and emit the matching artifact requirements without rediscovering + them from generated source text. + +The legacy generators are behavioral oracles, not dependencies of +`x2py/wrapper_codegen`. Reuse the runtime helpers and completed semantic +records directly. Rewrite the smallest equivalent node/lowering methods in the +direct generators; do not import legacy binding/bridge generator methods or +legacy codegen-model nodes into the wrapper-plan package. + +### Completed Direct-Plan Shape + +Wrapper policy now carries the completed native-handle and array-actual facts. +`ArgumentTransferPlan`, `ResultPlan`, and `ModuleVariablePlan` distinguish a +NumPy data-buffer transfer, a normal array dummy receiving a handle actual, and +a descriptor-handle transfer. Central validation fails closed when any typed +handoff, operation, role, ownership fact, or required header is inconsistent; +neither backend infers policy from datatype or `descriptor_boundary`. + +### Required Plan Shape + +Keep all descriptor-specific state subordinate to the existing datatype- +varying owners: -- [ ] Before implementation, expand this phase under the mandatory expansion - gate and reconcile it with the maintained native-array-handle checklist. - Split allocatable and pointer behavior only after extracting their shared - descriptor, presence, runtime element-length, ownership, release, - module/field, argument, and result sub-lanes. Include nullable deferred- - length scalar character results in that shared audit rather than creating a - string-only descriptor path. Keep rank-zero scalar descriptor results as - copied Python scalar values; reserve native-array handles for rank-positive - array storage. - -- [ ] Define descriptor handoff specs for CFI descriptors, descriptor ownership, - optional-absent handles, owner retention, extraction policy, and required - headers. -- [ ] Move bridge-created `ArrayInteropPolicy` decisions into plan generation or - completed policy for source/contract values. -- [ ] Replace bridge-created semantic `OwnershipDecision` values for generated - helper temporaries with explicit bridge-local helper specs. -- [ ] Validate descriptor/data-buffer mismatches before emission. -- [ ] Keep generated helper storage local to bridge/binding implementation - methods; do not represent helper temporaries as semantic ownership policy. +```text +ArgumentTransferPlan + array: ArrayHandoffPlan | None + native_array_actual: NativeArrayActualPlan | None + native_array_handle: NativeArrayHandlePlan | None + handoff: NativeDescriptorHandoffPlan + native_call_slot: NativeCallSlotPlan + +ResultPlan + native_array_handle: NativeArrayHandlePlan | None + native_call_slot: NativeCallSlotPlan | None + +ModuleVariablePlan + native_array_handle: NativeArrayHandlePlan | None + +FunctionPlan + native_call_slots: shared ordered references + lifecycle actions: ordered handle materialization/release references +``` + +`NativeCallSlotPlan` and `LifecycleActionPlan` are not competing top-level +semantic owners. A native slot is the argument/result ABI facet shared by its +owning transfer plan, while lifecycle records are function-wide ordering +indexes back to argument/result roles. Descriptor ownership, release, and +operation policy stay under `ArgumentTransferPlan`, `ResultPlan`, or +`ModuleVariablePlan`. Backend-local CFI storage, copy buffers, and failure +cleanup remain inside the named lowerer selected by those plans. + +`NativeArrayActualPlan` is used only when an ordinary `T[...]` argument permits +a runtime native handle as another source for the existing array-buffer ABI. It +records the explicitly accepted Python source kinds and the shared dtype, rank, +shape, layout, writeability, native-byte-order, alignment, and ABI-role checks. +It never carries descriptor ownership or extraction policy. + +`NativeArrayHandlePlan` is one editable projection of the completed handle +policy. It must contain, using typed values rather than free-form backend +method names: + +- descriptor kind and handle kind; +- origin, owner, owner-retention mode, descriptor ownership, and borrowed state; +- element datatype family, dtype, rank, declared shape, order, and character + element length when applicable; +- getter behavior, Python setter exposure, and native setter assignment; +- output projection and same-handle identity requirements; +- release responsibility, target lifetime, destroy behavior, and storage mode; +- `.to_numpy()` extraction action and allowed generated operations; +- descriptor-interop requirement and required headers; +- nullability and optional-absent-handle behavior; and +- one `NativeDescriptorHandoffPlan` with its ABI form and symbolic roles. + +`NativeDescriptorHandoffPlan` must distinguish these typed ABI forms: + +- `FACT_PACKED_CALL_LOCAL`: a non-projected descriptor argument supplies + validated standard descriptor facts; the binding passes those fields and the + bridge establishes call-local standard C descriptor storage. +- `DIRECT_STANDARD_DESCRIPTOR`: a projected writable handle passes its + persistent standard-descriptor pointer so allocation, deallocation, + reassociation, and shape changes remain attached to that handle. +- `OWNED_RESULT_STORAGE`: an allocatable result is materialized into persistent + wrapper-owned CFI storage and later destroyed by the runtime handle. + +The handoff records the descriptor-pointer role when present, `base_addr`, +`elem_len`, runtime rank, per-axis lower-bound/extent/stride-multiplier roles, +an optional presence role, owner-storage role, and generated-operation roles. +The `NativeCallSlotPlan` and its owning argument or hidden result must reference +the same mutable handoff record; do not duplicate descriptor facts that a +maintainer would need to edit twice. + +Convert the current string-valued completed policy selectors into typed plan +enums or validate and translate them exactly once while building wrapper +policy. Backends must not match raw strings such as `argument_descriptor`, +`projected_handle`, or `pointer_c_descriptor` to choose behavior. + +### Consistent Action Vocabulary + +Reuse the existing orthogonal actions: + +| Case | `ObjectKind` | Python action | Native action | `CodegenAction` | Bridge data action | +| --- | --- | --- | --- | --- | --- | +| Ordinary array with ndarray or handle actual | `NUMPY_ARRAY` | `ARRAY_STORAGE`, with explicitly planned accepted sources | `PASS_ARRAY_BUFFER` | existing Phase 6 input/in-place action | `ASSOCIATE_VIEW` | +| Read-only descriptor handle argument | `NUMPY_ARRAY` | `WRAPPER_INSTANCE` | `PASS_NATIVE_DESCRIPTOR` | `CALL_LOCAL_INPUT` | `ASSOCIATE_VIEW` | +| Writable projected descriptor handle | `NUMPY_ARRAY` | `WRAPPER_INSTANCE` | `PASS_NATIVE_DESCRIPTOR` | `IN_PLACE_ARGUMENT` | `DIRECT_TRANSFER` | +| Owned allocatable handle result | `NUMPY_ARRAY` | `NONE` | `NONE` or hidden `PASS_NATIVE_DESCRIPTOR` | `WRAPPER_INSTANCE` | `COPY_REPRESENTATION` with an ownership-transfer reason | +| Borrowed module handle getter | `NUMPY_ARRAY` | module getter action `NATIVE_ARRAY_HANDLE` | operation-specific | `BORROWED_VIEW` | completed per operation | + +Using `WRAPPER_INSTANCE` for the Python handle is consistent with the existing +action axis: the binding validates and consumes a generated runtime wrapper +object, while `ObjectKind.NUMPY_ARRAY` still identifies its array semantic +family. Add a new Python action only if a proven backend operation cannot be +expressed by this existing pair. Add `ArgumentHandoffMode.NATIVE_DESCRIPTOR` +because descriptor tuples are a genuinely different binding-to-bridge ABI from +`ARRAY_BUFFER`; do not overload the Phase 6 mode. + +Keep rank-zero descriptor values on the scalar or string object-kind route. +Their result action creates a Python scalar/string or `None`, not +`WRAPPER_INSTANCE`, and they must not carry `NativeArrayHandlePlan`. + +### Cross-Backend Validation Invariants + +Before either backend emits source, `_validate_plan()` must reject every one of +these inconsistencies: + +- a descriptor plan whose completed `ObjectKind` is not `NUMPY_ARRAY`; +- `PASS_ARRAY_BUFFER` carrying descriptor ownership or CFI roles; +- `PASS_NATIVE_DESCRIPTOR` carrying ordinary data-buffer handoff roles without + a descriptor handoff; +- a disagreement among handle policy, interop ABI, descriptor kind, handle + kind, argument/result plan, and native-call slot; +- a required handle accepting `None`; +- an optional absent handle without a presence role, or a required handle with + one; +- collapsing optional absence into present-unallocated/present-unassociated + state: an absent handle has null fields and a null presence token, whereas a + present handle may have null `base_addr` but must have a non-null presence + token; +- fact-packed handoff for a projected writable descriptor, or direct persistent + descriptor handoff for a policy that does not permit descriptor mutation; +- direct descriptor handoff without a typed + `_NativeArrayDescriptorHandoff`-compatible runtime operation; +- descriptor dtype, rank, shape, element length, or per-axis field counts that + disagree with the declared handle data facet; +- pointer reassociation, allocation, deallocation, or resize without completed + `PointerPolicy` permission; +- a pointer result without stable owner storage and target lifetime; +- an owned result without wrapper ownership, heap/alias boundary storage, + destroy behavior, owner retention, or a failure-path release action; +- a borrowed module/field handle that claims to destroy native owner storage; +- descriptor-view extraction without its completed C-descriptor build + requirement; +- a C-descriptor header requirement on a generation unit whose completed plans + do not need that interop; and +- any semantic helper temporary represented by a fabricated + `OwnershipDecision`. Call-local CFI variables, decoded-dimension locals, + pointer views, status locals, and operation tables are backend-local emitted + storage inside the already selected method. + +### Phase 7A — Ordinary Array Dummies Accepting Native Handle Actuals + +Included: concrete-rank numeric `T[...]` arguments already supported by Phase 6 +when the runtime value is either a valid ndarray, an allocated allocatable +handle, or an associated pointer handle. The handle path validates the same +dtype, rank, shape, layout, writeability, byte-order, and alignment contract, +then calls the handle's internal `array_actual` operation and packs the existing +Phase 6 pointer/extent/stride ABI. It never calls `.to_numpy()` and never passes +the allocatable/pointer descriptor to the ordinary native dummy. + +Initially excluded: optional, assumed-rank, character, and unsupported +noncontiguous handle actuals. Audit each against live runtime-helper behavior +before widening this sub-lane; a rejected form must remain an explicit blocker, +not silently fall back to `.to_numpy()` or a raw address. + +Those exclusions remain visible as the uncompleted +`array-handle-actuals-excluded` rollout lane. Their direct Phase 6 ndarray +lowerers remain testable with a forced wrapper-plan route, but automatic +production selection stays on the legacy route until each corresponding handle +source has parity evidence. + +Legacy oracle: `CPythonBindingGenerator._native_array_actual_argument_body`, +the normal-array runtime helpers in `x2py/runtime/handles.py`, and the existing +Phase 6 bridge array-buffer lowering. Reuse the runtime helpers and bridge ABI; +rewrite only the minimal direct binding call and source-kind branch. + +Plan and lowering requirements: + +- [x] Add `NativeArrayActualPlan` or equivalent accepted-source facts beneath + the existing ordinary `ArgumentTransferPlan`; keep + `PASS_ARRAY_BUFFER`, `ArgumentHandoffMode.ARRAY_BUFFER`, and + `ArrayHandoffPlan` unchanged. +- [x] Make the C binding's named ordinary-array input method call + `_native_array_actual_argument_for_binding_positional` with only planned + validation flags and ABI-field selections. +- [x] Keep the Fortran bridge on the exact Phase 6 array-buffer method; it must + not know whether Python supplied an ndarray or a handle. +- [x] Validate that handle actuals are allocated/associated, have a non-null + data address, and satisfy the same declared contract as ndarray inputs; + preserve allocated/associated zero-length arrays. +- [x] Add the `array-native-handle-actuals` support lane and remove the current + production gate on ordinary array actuals only after reduced compiled parity + proves both runtime source kinds and all rejection paths. +- [x] Reuse the normal-array calls in + `test_module_and_derived_pointer_handles_track_native_association` and + allocatable handle fixtures as the legacy baseline, but extract a class-free, + dependency-closed parity contract so Phase 8 does not determine this lane's + route. + +### Phase 7B — Required Read-Only Descriptor Handle Arguments + +Included: required, non-projected `Allocatable[T[...]]` and +`Pointer[T[...]]` arguments. The Python binding accepts only the matching +runtime handle class. A present unallocated allocatable or unassociated pointer +is still a present descriptor argument and may carry a null `base_addr`. + +The binding uses the existing descriptor runtime helper to obtain validated +standard descriptor facts. The bridge establishes rank-specific call-local CFI +storage from `base_addr`, `elem_len`, rank, and dimension records, then passes +the native allocatable or pointer dummy. This association is an emitted-code +view, not a semantic data copy. + +Legacy oracle: + +- binding `_bind_allocatable_descriptor_argument`, + `_bind_pointer_descriptor_argument`, and + `_bind_fact_packed_native_array_descriptor_argument`; +- bridge `_bridge_allocatable_descriptor_argument`, + `_bridge_pointer_descriptor_argument`, and + `_bridge_native_array_descriptor_argument`; and +- runtime `_native_array_descriptor_argument_for_binding_positional`. + +- [x] Carry the completed `NativeArrayHandlePolicy` and descriptor + `ArrayInteropPolicy` into `ArgumentPolicy`, `ArgumentTransferPlan`, and its + shared native slot. +- [x] Add `ArgumentHandoffMode.NATIVE_DESCRIPTOR` and a + `FACT_PACKED_CALL_LOCAL` descriptor handoff with exact symbolic roles. +- [x] Add directly named C and Fortran descriptor-input methods grouped under + the native-array-handle family; backend-local tuple items and CFI locals may + be created only inside those selected methods. +- [x] Validate matching handle class, descriptor kind, dtype, rank, declared + shape, and element length before the call. Reject ndarray inputs. +- [x] Add separate `allocatable-descriptor-inputs` and + `pointer-descriptor-inputs` support lanes after reduced descriptor-argument + parity passes. +- [x] Replay the descriptor calls in + `test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` + and the allocatable descriptor fixtures through minimal class-free contracts; + retain the mixed original nodes as legacy until all their later owners migrate. + +### Phase 7C — Optional Absent Descriptor Handles + +Included: `Allocatable[T[...]] | None = ...` and +`Pointer[T[...]] | None = ...` callable arguments. Omission and explicit +`None` both mean native `present(...)` is false. A present handle remains +present even when its descriptor has absent allocation/association state. + +This is a two-level handle-presence contract, not the Phase 3 scalar descriptor +three-state value contract. Do not reuse the value pointer as the presence +token. The runtime helper already produces null fact fields plus null presence +for absence, and a distinct non-null token for every present handle. + +- [x] Project `optional_absent`, `nullable`, presence mode, and the dedicated + presence role from completed handle policy without inspecting the Python + object in planning or bridge code. +- [x] Generate both required and optional fact-packed descriptor calls through + the Phase 7B methods, adding only the planned presence ABI field and native + branch. +- [x] Validate required-versus-optional annotation, field count, presence role, + and the distinction between absent handle and present null `base_addr`. +- [x] Add `optional-native-array-handles` route coverage only after compiled + tests exercise omission, explicit `None`, present allocated/associated, + present unallocated/unassociated, wrong handle kind, and wrong dtype/rank. +- [x] Treat the lack of one isolated compiled optional array-handle fixture as + a coverage gap: create a reduced semantic `.pyi` entry over an existing + native optional descriptor routine instead of inventing behavior from the + runtime-only tests. + +### Phase 7D — Writable And Projected Descriptor Handles + +Included: descriptor arguments whose allocation, deallocation, resize, +reassociation, or nullification must remain visible through the same Python +handle, plus a matching projected result that returns that identical handle. +Allocatable mutation follows completed ownership. Writable pointer descriptor +mutation requires explicit `PointerPolicy` permissions and target-lifetime +facts. + +Fact-packed call-local descriptors are forbidden here because native mutation +would be discarded at return. The binding must request the handle's typed +persistent standard-descriptor pointer and the bridge must pass it directly. +Returning the projection increments/transfers the existing Python reference; it +does not construct a replacement handle or call `.to_numpy()`. + +The direct handoff requires generated persistent standard-descriptor storage. +Wrapper-owned result handles provide it. Borrowed module handles expose current +descriptor facts for read-only calls, but they are not accepted for projected +writable mutation because a reconstructed call-local descriptor would lose the +native descriptor update. + +Legacy oracle: + +- binding `_bind_direct_native_array_descriptor_argument` and + `_bind_projected_native_array_handle_result`; +- bridge descriptor argument dispatch with completed output projection; and +- runtime `_native_array_descriptor_handoff_for_binding_positional`. + +- [x] Add `DIRECT_STANDARD_DESCRIPTOR` handoff and same-handle result identity + to the owning `ArgumentTransferPlan`, shared native slot, `ResultPlan` or + lifecycle consumer, and function-wide result order. +- [x] Reuse `CodegenAction.IN_PLACE_ARGUMENT` and `DIRECT_TRANSFER`; do not add + a descriptor-copy action for same-handle mutation. +- [x] Plan success and failure reference handling so a projected handle is + returned exactly once and borrowed caller storage is never destroyed. +- [x] Validate operation permissions, descriptor ownership, target lifetime, + direct handoff type, result identity, and optional presence before emission. +- [x] Add `projected-native-array-handles` support only after + `test_allocatable_inout_arrays_mutate_and_return_the_same_handle` has a + reduced legacy/direct-plan parity replay covering allocation, reallocation, + deallocation, identity, wrong input types, and native-memory checks. +- [x] Keep writable pointer reassociation blocked unless the completed policy + proves every required permission and lifetime fact; never downgrade it to a + read-only fact-packed call. + +### Phase 7E — Owned Allocatable Results And Hidden Outputs + +Included: rank-positive allocatable direct function results and hidden output +descriptors whose completed policy selects `owned_result_descriptor`. A valid +allocatable function result is allocated when returned; an unallocated +nonpointer function result is a nonconforming native procedure and the wrapper +does not compensate for it. An allocatable output dummy may validly remain +unallocated and still returns a present `AllocatableArray` handle whose state +lives inside that handle. Pointer handle results remain blocked until stable +owner storage and target lifetime are explicit. + +For a numeric direct allocatable function result, the bridge assigns the native +function expression once into a procedure-local allocatable and then uses +`move_alloc` to transfer that allocation into the allocatable `intent(out)` +dummy backed by persistent wrapper-owned `CFI_CDESC_T(rank)` storage. The move +does not copy the array payload. Do not insert a collector helper, an +`allocated(...)` guard, or a second intrinsic assignment. The native function +must return an allocated, defined result; an unallocated result is a +nonconforming native procedure and remains the user's responsibility rather +than a wrapper fallback. Other procedure-local storage remains permitted only +when representation conversion genuinely requires it, such as +deferred-character byte materialization. The binding constructs the complete +generated operation table and Python handle only after owner storage is valid. +Ownership transfers to the handle exactly once; every earlier failure path +releases the persistent allocation and any genuinely required bridge-local +allocation. + +Character-element handles carry runtime `elem_len` and declared element-length +policy in the same descriptor record. Because a deferred character width is +unknown until the native result exists, the bridge first copies the bytes and +the binding then establishes and allocates persistent CFI storage with that +runtime width. This is a named lowering method under the same result handle +plan, not a separate string-result ownership hierarchy. + +Legacy oracle: + +- binding `_bind_owned_allocatable_result_handle`, owned-result operation + builders, `_bind_materialized_native_array_handle_result`, and destroy body; +- bridge `_bridge_owned_allocatable_result_handle` plus allocatable result + helper/copy logic; and +- the runtime handle factory and exactly-once `close()`/finalizer protocol. + +- [x] Attach one `NativeArrayHandlePlan` with `OWNED_RESULT_STORAGE` to direct + and hidden `ResultPlan` owners; hidden outputs share their exact descriptor + native slot. +- [x] Use `CodegenAction.WRAPPER_INSTANCE` and an explained + `COPY_REPRESENTATION` only for materialization into persistent owner storage; + source hiddenness remains `source_kind`, not a codegen action. +- [x] Record owner storage, materialization, handle construction, ownership + transfer, destroy behavior, and release responsibility under the result's + typed handle plan. Keep backend-local CFI allocation/copy/free nodes inside + the selected result lowerer rather than fabricating lifecycle policy records. +- [x] Require generated `shape`, `array_actual`, `descriptor`, extraction/state, + allowed mutation, and `destroy` operations before publishing the handle. +- [x] Validate CFI rank, dtype, element length, allocated state, owner + retention, release responsibility, destroy behavior, and all success/failure + paths before emission. +- [x] Add `owned-allocatable-results` and + `owned-allocatable-hidden-outputs` support lanes after reduced parity from + `test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, + `test_output_arguments_and_multiple_results_follow_python_projection_rules`, + and `test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`. +- [x] Keep pointer result tests on their explicit policy blocker; do not make + their matrix rows `wrapper-plan` merely because allocatable results pass. + +### Phase 7F — Borrowed Module Handles And Generated Operations + +Included: rank-positive allocatable and pointer module variables exposed as one +stable borrowed handle object at module initialization. Repeated attribute reads +return the same handle. Replacement assignment is rejected. The generated +operation table accesses current native state and includes only operations +allowed by completed policy. + +Allocatable operations include allocation state, shape, array actual, +descriptor handoff, extraction, deallocation, and resize where allowed. Pointer +operations include association state, shape, array actual, descriptor handoff, +nullification, extraction, and policy-gated allocation/deallocation/resize. +Borrowed module handles retain the Python module and never destroy native-owned +descriptor storage. + +Deferred-character handles also expose runtime `element_length`. Shape-only +`allocate` and `resize` operations are omitted because they cannot state the +new character width; native procedures that declare the width remain the +authoritative mutation path. + +Legacy oracle: the bridge's `_native_array_module_handle`, +`_native_array_module_handle_operations`, and operation-specific module methods; +the binding's `_bind_borrowed_native_array_module_handle`, operation wrappers, +and handle creation; and the current runtime handle factory. + +- [x] Add a native-handle getter action and one `NativeArrayHandlePlan` beneath + `ModuleVariablePlan`; keep Python attribute exposure and native operation + generation in its binding and bridge child views. +- [x] Plan operation roles and export names explicitly while leaving operation + call locals backend-local. Do not store generated method names in the plan. +- [x] Validate stable handle identity, module owner retention, rejected + replacement, descriptor kind, operation completeness, and borrowed/no-destroy + lifecycle. +- [x] Add `allocatable-module-handles` and `pointer-module-handles` support + lanes only after module-only reduced parity covers state changes, zero-length + state, extraction policy, operation permissions, stale-view behavior, and + module lifetime. +- [x] Use `test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, + `test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction`, + and the module portion of + `test_module_and_derived_pointer_handles_track_native_association` as legacy + oracles. Split out field/class assertions, which remain Phase 8/9 work. + +### Phase 7G — Pointer Descriptor Extraction And Build Requirements + +Included: pointer `descriptor_view`, `contiguous_view`, `copy_only`, and +explicitly unsupported extraction actions already selected by completed +policy; standard descriptor decoding; positive and negative strides; and local +build/header requirements. + +Only `descriptor_view` and persistent allocatable owner storage require standard +C descriptor support. Generated code may read `CFI_cdesc_t` through +`ISO_Fortran_binding.h` when the completed plan requests it. It must never guess +or expose a compiler-private descriptor layout. Unsupported toolchains fail +readiness/build with the completed owner path and requirement. + +- [x] Carry typed extraction and descriptor-interop actions plus required + headers into handle/module/result plans and rendered artifact metadata. +- [x] Reuse the runtime descriptor-view helper for shape, stride, buffer-window, + dtype, rank, and null-address validation; direct C lowering only decodes the + standard descriptor fields into its expected mapping. +- [x] Add directly named C descriptor-reader and operation-wrapper methods; + decoded dimension objects and mapping temporaries remain binding-local. +- [x] Validate that build requirements equal the union of completed plans, + appear in replayable manifests, and do not leak into wrappers that need only + ordinary buffers or non-CFI borrowed allocatable handles. +- [x] Replay + `test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` + and `test_pyi_manifest_records_pointer_descriptor_interop_requirements`, plus + focused `tests/runtime/handles`, before enabling + `pointer-descriptor-extraction`. +- [x] Preserve the explicit readiness failure when required C descriptor + support is unavailable; no contiguous-copy fallback may be inferred in the + backend. + +### Phase 7H — Remaining Rank-Zero Descriptor Results And Strings + +Phase 3 already owns ordinary and optional scalar descriptor inputs, including +omitted/present-null/present-value state. Phase 4 already owns nullable scalar +descriptor module reads as copied Python snapshots. Do not rebuild those paths +or turn rank-zero descriptors into runtime handle objects. + +Included here: direct scalar descriptor function results, hidden scalar +descriptor outputs, projected scalar descriptor readback, and allocatable or +pointer scalar character results with runtime/deferred length. The Python result +is `T | None` or `String | None`; absent allocation/association returns `None`. +An allocated/associated value is copied exactly once before the call-local or +native descriptor is released. Deferred-length strings use runtime element +length and preserve the existing encoding/byte contract. + +- [x] Add a subordinate scalar-descriptor handoff/result record to the existing + scalar or string `ArgumentTransferPlan`/`ResultPlan`; do not attach + `NativeArrayHandlePlan` or use `ObjectKind.NUMPY_ARRAY` for rank zero. +- [x] Complete result source, descriptor kind, presence, runtime element length, + copy action/reason, release owner, and failure cleanup in wrapper policy before + planning. +- [x] Reuse existing Phase 3 presence records and typed lifecycle ordering; + extend named scalar/string result lowering only for the descriptor producer + and copy/release steps. +- [x] Validate direct versus hidden descriptor source, nullable result spelling, + result ordering, runtime string length, null state, copy count, and cleanup on + conversion/status failure. +- [x] Add isolated legacy/direct parity for numeric allocatable and pointer + results and for `string_result_deferred` from + `test_modern_fortran_character_arguments_and_results`, including an absent + result and non-ASCII encoded data. +- [x] Keep pointer array results blocked even after pointer scalar values pass; + copied scalar readback does not prove array target lifetime. + +### Derived Fields Remain A Recorded Later Dependency + +The shared handle plan must be capable of recording +`borrowed_field_descriptor`, `owner_retention=parent_wrapper`, field operation +roles, and parent-owned destruction behavior. Do not add field/class traversal +or route eligibility in Phase 7. `BindCNativeArrayHandleProperty`, field +operation generation, and the field portions of allocatable/pointer tests remain +legacy oracles for Phases 8 and 9, where the owning wrapper instance and property +lifecycle exist in the plan. + +This boundary prevents Phase 7 from either duplicating future `FieldPlan` +ownership or falsely marking mixed module-and-field generation units supported. + +### Legacy Primitive Inventory And Rewrite Rule + +| Primitive | Legacy source | Direct-plan treatment | +| --- | --- | --- | +| Normal array handle actual | binding `_native_array_actual_argument_body`; runtime normal-array helpers | reuse runtime helper and Phase 6 bridge ABI; rewrite minimal binding nodes | +| Required/optional descriptor argument | binding descriptor argument helpers; bridge descriptor handlers | rewrite named direct methods around shared runtime packer and planned CFI roles | +| Projected writable descriptor | binding direct descriptor handler; bridge descriptor projection | rewrite direct pointer handoff and identity lifecycle; no fact-packed fallback | +| Owned allocatable result | binding owned-result/operation helpers; bridge allocatable result helper | rewrite minimal CFI owner-storage and result lifecycle nodes; assign once locally and transfer the allocation into the CFI-backed output dummy with `move_alloc` | +| Borrowed module handle | binding handle creation/operation wrappers; bridge module operations | rewrite under `ModuleVariablePlan`; reuse runtime factory | +| Pointer descriptor view | binding descriptor reader; runtime view helper | reuse runtime view helper; rewrite only standard-descriptor decoding nodes | +| Scalar descriptor result | legacy scalar descriptor/result conversion | extend existing scalar/string plan route; do not create an array handle | +| Build requirement | semantic `native_array_handle_build_requirements`; build manifest | reuse completed requirements and carry them through rendered artifacts | + +For every primitive, first retain generated legacy C/Fortran/header artifacts +from the cited passing wrapper test. Explain each material direct-plan artifact +difference before compilation. Copy a small legacy method only when it already +matches the direct node API and complexity limit; otherwise rewrite the minimal +equivalent. Do not copy legacy dispatcher classes, scope mutation machinery, or +datatype/policy inference. + +### Route And Test Migration Matrix For Phase 7 + +Mixed rows retain their later derived/field owners. The dependency-closed +Phase 7 rows were split, proved through both routes, and then recorded as +`wrapper-plan` in the complete ledger. + +| Existing node or group | Current role/status | Phase 7 owner and target | +| --- | --- | --- | +| `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | ordinary/allocatable result generation unit; `wrapper-plan` | Phase 6 ordinary and Phase 7E allocatable results now share the production plan route | +| `build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | non-generating manifest policy; `not-applicable` | Phase 7G plan/header union is covered by direct generated-artifact tests | +| `derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | module, normal-array actual, and field mix; `legacy` | split Phase 7A/7F module subsets; field subset remains Phase 8/9 | +| `derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | module/field descriptor views; `legacy` | Phase 7B/7G module subset; field owner remains Phase 8/9 | +| `derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | explicit supported blocker; `legacy` | remain blocker until owner/lifetime policy changes; never auto-promote | +| `edit_pyi_contracts/test_ownership_contracts.py::*` | module, field, result lifetime mix; `legacy` | Phase 7E/7F subsets; field/finalizer owners remain Phase 8/9 | +| `function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | scalar baseline; `wrapper-plan` | reuse Phase 3 behavior; no status change | +| `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | mixed scalar/array/string/derived/allocatable outputs; `legacy` | Phase 7E reduced allocatable result; retain mixed row | +| `module_state/test_allocatable_replacement.py::*` | projected same-handle descriptor mutation plus a derived factory generation unit; `legacy` | Phase 7D reduced parity is `wrapper-plan`; the broad factory/class unit remains Phase 8/9 | +| `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | module, result, and derived-field mix; `legacy` | field/class owner retention remains Phase 8/9 | +| `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | scalar descriptor arguments/results/module state; `wrapper-plan` | source conversion records descriptor kind and argument/return reference before completed Phase 7H policy | +| `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction[*]` | plain allocatable module snapshot; `wrapper-plan` | Phase 7F binding-owned detached read-only snapshot and capsule lifetime | +| `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | fixed strings plus deferred allocatable result; `legacy` | Phase 7H reduced deferred-result parity; retain mixed row as needed | +| `edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | includes raw `Addr(Float64[n])` plus a derived result; `legacy` | raw-array subset is the completed Phase 6G prerequisite; derived subset remains Phase 8 | + +| Completed sub-lane | Dependency-closed compiled evidence | +| --- | --- | +| Phase 7A, 7B, 7F, and 7G | `derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes` | +| Phase 7C | `function_calls/test_optional_arguments.py::test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes` | +| Phase 7D | `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes` | +| Phase 7E numeric | `arrays/test_array_results.py::test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes` | +| Phase 7E deferred character | `strings/test_character_arguments.py::test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes` | +| Phase 7H numeric | `scalars/test_scalar_boundary_plan.py::test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_route` | +| Phase 7H deferred scalar character | `strings/test_character_arguments.py::test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes` and the nullable case in the deferred-character handle test | +| Phase 7H source/default projection | `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | + +Required focused intermediate coverage includes: + +- completed semantic handle/interop policy projection tests; +- editable plan tests for every descriptor kind, handoff form, operation set, + ownership, optional presence state, and lifecycle edit; +- C and Fortran preflight rejection of mismatched or incomplete descriptor + plans; +- generated artifact assertions for standard descriptor fields, optional + presence, operation functions, owner storage, destroy paths, and local header + requirements; +- runtime helper tests under `tests/runtime/handles` without duplicating their + validation in wrapper-codegen tests; and +- compiled legacy/direct parity for each reduced sub-lane before any migration + ledger or production-route change. + +### Phase 7 Completion + +- [x] Expand Phase 7 under the mandatory gate using the live completed policy, + runtime handle implementation, legacy backends, build integration, public + contract, and focused wrapper tests. +- [x] Complete the shared typed handle, array-actual, and descriptor-handoff + plan records without adding a parallel top-level plan hierarchy. +- [x] Complete every missing semantic selector before `ir2ast.py`; remove + bridge-created `ArrayInteropPolicy` and fabricated semantic ownership choices. +- [x] Finish Phases 7A through 7H individually with legacy artifact capture, + direct lowering, validation, compiled parity, route evidence, and matrix + updates. +- [x] Preserve the completed Phase 6G raw-address boundary while keeping every + derived-field, pointer-result, callback, and deferred-real-library exclusion + on its explicit later blocker. +- [x] Run focused policy/plan/backend tests, relevant runtime-handle tests, + documentation checks, the wrapper suite excluding LAPACK, the wrapper-codegen + complexity checker, and the required static-analysis suite before declaring + implementation complete. +- [x] Close Phase 7 only when every live non-field native-handle/descriptor case + is migrated or explicitly removed from the product contract, and no backend + infers descriptor policy or silently substitutes a data buffer, raw pointer, + `.to_numpy()` extraction, or copy fallback. + +Closure evidence: 538 focused semantic/policy/plan/backend/runtime tests, 1,133 +documentation and layout tests, and all 318 wrapper tests outside the deferred +BLAS/LAPACK file pass. The wrapper-codegen complexity check, Ruff, formatting, +Bandit, Vulture, whitespace, and explicit-base Radon policy pass; full Radon +complexity and maintainability reports were also recorded. The source/default +scalar-descriptor projection now records its descriptor kind and argument or +return reference before policy completion. Binding and bridge lowering dispatch +only from the resulting typed plans; the remaining mixed derived-field unit is +an explicit Phase 8/9 blocker rather than a descriptor fallback. ## Phase 8 — Derived Types And Snapshots diff --git a/docs/user/guide/allocatables.md b/docs/user/guide/allocatables.md index fca9a0a27..9aaa8321a 100644 --- a/docs/user/guide/allocatables.md +++ b/docs/user/guide/allocatables.md @@ -56,8 +56,11 @@ def maybe_resize(values: Allocatable[Float64[:]] | None = ...) -> None: ... That spelling is valid only for optional callable arguments. Do not use `Allocatable[T[...]] | None` for module variables, derived-type fields, or -function results; those surfaces return a present handle, and unallocated state -is represented inside that handle. +function results; those surfaces return a present handle. Module variables, +fields, allocatable output dummies, and handles changed by later operations may +be unallocated. A native allocatable function result must be allocated when the +function returns; returning it unallocated is nonconforming Fortran and x2py +does not define a fallback for it. Passing a handle to `Allocatable[T[...]]` passes the native descriptor. Passing the same allocated handle to a normal `T[...]` argument uses ordinary Fortran @@ -140,9 +143,11 @@ def update_scale( Passing `None` creates a present but unallocated call-local descriptor. Omitting a defaulted scalar descriptor argument creates native optional absence, so `present(scale)` is false. Passing a value creates a present allocated -call-local descriptor. An unallocated function result or projected output -becomes `None`. Ordinary scalar projection rules still apply: `intent(out)` uses -`Allocatable(Return("name", j))`, while `intent(inout)` uses +call-local descriptor. A projected output becomes `None` when its descriptor is +unallocated. An allocatable function result must instead be allocated and +defined when returned; an unallocated result is nonconforming native Fortran, +not a nullable x2py value. Ordinary scalar projection rules still apply: +`intent(out)` uses `Allocatable(Return("name", j))`, while `intent(inout)` uses `Allocatable(Arg(i))` plus a matching `Returns["name", T] | None` readback. The singular `result=Allocatable(Return(j))` mapping describes the native function result and places it among any other Python results. @@ -350,8 +355,14 @@ Allocatable character arrays use fixed-width NumPy bytes storage. Create ```fortran module character_names implicit none - character(len=:), allocatable :: stored_names(:) contains + function make_names() result(names) + character(len=:), allocatable :: names(:) + + allocate(character(len=3) :: names(2)) + names = [character(len=3) :: "red", "sky"] + end function make_names + subroutine replace_names(names) character(len=:), allocatable, intent(inout) :: names(:) integer :: count @@ -379,7 +390,7 @@ the native allocation at runtime: ```python from x2py.contracts import Allocatable, Returns, String -stored_names: Allocatable[String[:][:]] +def make_names() -> Allocatable[String[:][:]]: ... def replace_names( names: Allocatable[String[:][:]] @@ -403,7 +414,7 @@ sys.path.insert(0, "build/character_allocatables") import character_allocatables api = character_allocatables.character_names -names = api.stored_names +names = api.make_names() assert api.replace_names(names) is names assert names.to_numpy().dtype.itemsize == 5 assert names.to_numpy().tolist() == [b"red ", b"blue "] @@ -415,6 +426,13 @@ handle-typed parameter. When extracting character storage, x2py uses NumPy bytes dtype `S`; Unicode (`U`) and object (`O`) arrays are not descriptor-handle substitutes. +Projected writable descriptor mutation requires a handle with persistent +wrapper-owned standard-descriptor storage, such as the owned result returned by +`make_names()`. A borrowed module handle can be passed to a read-only descriptor +argument through descriptor facts, but it cannot be passed to a projected +writable descriptor argument: native mutation of a call-local reconstructed +descriptor would not update the module handle reliably. + ## Module Handles And Views An allocatable module array is native-owned. Reading the Python attribute @@ -451,6 +469,9 @@ independent = view.copy() - Mutable scalar deferred-length character storage is blocked. - Borrowed views require a proved native or wrapper owner and `Aliased` storage when the owner is a module variable. +- Borrowed module handles do not provide the persistent direct descriptor + handoff required by projected writable descriptor arguments; use an owned + result handle for that operation. - An edited `.pyi` cannot relabel a native-owned descriptor as Python-owned. Use an implemented owned-result handle or copy an extracted NumPy value when Python needs independent storage. diff --git a/docs/user/guide/arrays.md b/docs/user/guide/arrays.md index 38de3d558..75763e1bb 100644 --- a/docs/user/guide/arrays.md +++ b/docs/user/guide/arrays.md @@ -132,6 +132,31 @@ runs. Use `numpy.asfortranarray` or `order="F"` for a multidimensional contract that requires Fortran orientation, as shown by `matrix` in the complete example. +Layout annotations describe the exact storage accepted by the native contract; +they do not request an automatic conversion. `ORDER_F` passes a +Fortran-contiguous array with its logical axes unchanged. `ORDER_C` passes the +same C-contiguous data address without copying and constructs the Fortran +bridge view with reversed axes. For example, a C-order Python shape `(2, 3)` +is a Fortran bridge shape `(3, 2)` over the same six elements. Use `ORDER_C` +only when the native operation intentionally accepts that transposed storage +view. + +Add `COPY_F` when Python should accept C-contiguous storage but native Fortran +must observe the same logical axes in Fortran order: + +```python +values: Annotated[Float64[:, :], ORDER_C, COPY_F] +``` + +The binding owns this complete representation lifecycle. It creates an +F-contiguous NumPy temporary before the call, passes that ordinary F-order +buffer through the unchanged bridge path, copies values back into the original +C-order array after the call, and releases the temporary. Projected results +return the original C-order object. Native `intent(in)` remains a property of +the native procedure call; neither the semantic `.pyi` nor the bridge temporary +needs a separate direction annotation for `COPY_F`. The bridge performs neither +half of this argument conversion. + Rank-one contiguous arrays can satisfy their documented contiguous contract without a meaningful row/column distinction. Legacy fixed-form array contracts are contiguous-only. A modern Fortran dummy is stride-aware only when its diff --git a/docs/user/guide/data-types.md b/docs/user/guide/data-types.md index cf673ad79..8f2a1aabe 100644 --- a/docs/user/guide/data-types.md +++ b/docs/user/guide/data-types.md @@ -117,7 +117,11 @@ Use `T[()]` when the Python API itself exposes safe caller-provided scalar storage as a rank-0 NumPy array. Use `Addr(T)` only when the caller passes a raw address such as `array.ctypes.data`. For raw array addresses such as `Addr(Float64[n])`, every extent must be a fixed literal or a visible argument; -the address value itself does not carry shape. +the address value itself does not carry shape. x2py does not reject zero or +negative integer addresses or validate that raw-address extents are positive. +It reports integer-to-pointer overflow, but otherwise the caller is responsible +for supplying a live address and a valid pointee shape before native code uses +either value. Arrays and strings are storage-like at the native boundary. `Float64[n]` already passes the NumPy data address, and `String[8]` already passes the address of diff --git a/docs/user/guide/editing-semantic-pyi-contracts.md b/docs/user/guide/editing-semantic-pyi-contracts.md index b2d841307..72c04ec91 100644 --- a/docs/user/guide/editing-semantic-pyi-contracts.md +++ b/docs/user/guide/editing-semantic-pyi-contracts.md @@ -360,6 +360,10 @@ def scalar_status( Removing the explicit `status` parameter and adding the result projection are one edit. A projection must map every required native argument exactly once; incomplete, duplicate, or out-of-range mappings are contract errors. +Native `intent` does not override this edited signature: an output dummy may +remain caller-supplied storage, while a projected output exists only when the +contract explicitly requests that projection. The bridge may use permissive +writable local storage; the called native procedure enforces its own direction. ### Make mutation replacement-only diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 99f8dcd57..763d1a6f7 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -1165,6 +1165,13 @@ caller must provide enough storage for the native routine. Generated semantic `.pyi` contracts spell this final assumed-size dimension as `Flat`, for example `Float64[Flat]` for `real(8) :: values(*)`. +`Flat` marks one storage axis rather than forcing rank one. For example, +`Float64[:, Flat]` remains a rank-two Fortran-contiguous Python and bridge +contract. An external interface uses `values(*)` when the preceding extent is +available only from the Python array, because `values(:, *)` is not a legal +Fortran assumed-size declaration; the bridge nevertheless associates the +address with both runtime extents. + Non-default lower bounds are preserved when computing shape constraints; they diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index bacef39d8..c52fd997a 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -480,9 +480,13 @@ def DAXPY( ``` `Float64[3, Flat]` maps to `real :: a(3, *)`, and -`Float64[3, 4, Flat]` maps to `real :: a(3, 4, *)`. The Python-visible flat -dimension remains unconstrained, but the explicit Fortran interface generated -from the `.pyi` uses `DX(*)`/`DY(*)` instead of assumed-shape descriptors. +`Float64[3, 4, Flat]` maps to `real :: a(3, 4, *)`. `Flat` is an axis marker, +not a request to collapse the whole array to rank one: `Float64[:, Flat]` +remains a rank-two Python and bridge contract. Because `real :: a(:, *)` is not +a legal Fortran assumed-size declaration, an external interface whose prefix +extent is known only at runtime uses the sequence-associated `a(*)` spelling; +the bridge view still has rank two and receives both runtime extents. The +Python-visible flat dimension remains unconstrained. The Python argument may provide more storage than the declared explicit @@ -1045,10 +1061,11 @@ Use local constants or generated `Final[...]` names for shape symbols. source-language argument direction: ```python -from x2py.contracts import Annotated, Float64, ORDER_F +from x2py.contracts import Annotated, COPY_F, Float64, ORDER_C, ORDER_F def fill( a: Annotated[Float64[:, :], ORDER_F], + c_input: Annotated[Float64[:, :], ORDER_C, COPY_F], out: Float64[()], ) -> None: ... ``` @@ -1058,6 +1075,7 @@ Generated canonical metadata: | Metadata | Meaning | | --- | --- | | `ORDER_F` | multidimensional Fortran-oriented storage | +| `COPY_F` | accept the declared C-contiguous Python layout, create an F-contiguous temporary with the same logical axes, and copy back after visible native mutation | | `PointerAssociation("runtime")` | pointer association is a runtime state rather than a declaration-time constant | | `Name("native-name")` | source name cannot be represented directly as the Python target name | | `Aliased` | native storage may be exposed across the Python boundary as an alias | @@ -1085,6 +1103,21 @@ Loaded compatibility metadata: | `ORDER_C` | explicit C-oriented storage; this is also the default for plain multidimensional arrays | X2PY_C_DOCS_END --> +Without `COPY_F`, `ORDER_C` is zero-copy and native Fortran observes the +reversed-axis storage view. With `COPY_F`, the binding performs both copy-in and +any required copy-out; the bridge receives an ordinary F-order buffer and does +not know that a representation conversion occurred. `COPY_F` is initially +limited to required, concrete-rank, dense numeric ndarray arguments. It does +not apply to `Flat`, assumed-rank or strided arrays, optional arrays, character +arrays, native descriptor arguments, or handle actuals. + +Semantic `.pyi` types do not need to repeat the native procedure's `intent` +for this transformation. The binding may use a mutable temporary and copy it +back after the call even when the native dummy is `intent(in)`; the native +procedure interface still enforces its own direction. When an argument is +projected with `Returns`, Python receives the original C-order object after +copy-back. + Persistent native descriptors use wrapper type syntax instead of descriptor metadata inside `Annotated[...]`: diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index 292c35e18..4e15b95c9 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -36,6 +36,14 @@ C_DOCS_DISABLED = " diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index fbacc8e06..5dd2ba405 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -44,6 +44,7 @@ change crosses ownership boundaries. | Source-driven Fortran wrapper orchestration | `x2py/pipeline/build.py` | `docs/user/guide/fortran-wrapper.md`, `docs/user/examples/recipes/build-and-import-cli.md` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py` | | Semantic `.pyi` wrapper orchestration from native artifacts | `x2py/pipeline/build.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py` | `docs/user/guide/fortran-wrapper.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py`, `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py` | | Ownership, lifetime, output projection, and unsupported wrapper policy | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py`, `x2py/semantics/ir2ast.py` | `docs/user/guide/fortran-wrapper.md`, `docs/user/guide/editing-semantic-pyi-contracts.md` | `tests/lowering/test_semantic_ir.py`, `tests/wrapper/fortran/` | +| Immediate callback policy, typed adapters, and trampolines | `x2py/semantics/wrapper_policy.py`, `x2py/semantics/policy_completion.py`, `x2py/wrapper_codegen/plan.py`, `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/c/binding.py`, `x2py/wrapper_codegen/fortran/bridge.py` | `docs/user/guide/callbacks.md`, `docs/user/reference/callbacks.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/wrapper_codegen/test_phase10_callbacks.py`, `tests/wrapper/fortran/callbacks/` | | Native compilation, runtime support, and shared-library linking | `x2py/compiling/python_wrapper.py`, `x2py/compiling/runtime_support.py` | `docs/user/guide/fortran-wrapper.md`, `docs/developer/build-system.md` | `tests/wrapper/fortran/build_from_source/test_runtime_abi.py`, `tests/wrapper/fortran/build_from_source/test_build_modes.py` | | Public Python exports | `x2py/__init__.py` | `README.md`, `docs/user/reference/python-api.md` | `tests/parsing/fortran/test_public_entrypoints.py` | | Source navigation documentation | `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md`, package README files | `docs/developer/source-map.md` | `tests/docs/test_structure.py` | diff --git a/docs/maintainer/design/wrapper-design-notes.md b/docs/maintainer/design/wrapper-design-notes.md index 43464679c..9f1685243 100644 --- a/docs/maintainer/design/wrapper-design-notes.md +++ b/docs/maintainer/design/wrapper-design-notes.md @@ -53,7 +53,7 @@ X2PY_C_DOCS_END --> | Gap | Current risk | Proposed direction | | --- | --- | --- | | Procedure pointers and dummy procedures | A broad `Procedure` type loses enough signature and lifetime information that wrappers cannot safely call or receive callbacks. | Resolve abstract interface signatures into a first-class semantic callable form. Preserve procedure pointer, optional, pass-through, and callback lifetime facts; block wrapper generation until call direction and ownership policy are explicit. | -| Pointer and allocatable ownership | Allocatable and pointer arrays use explicit descriptor handles. Module and derived-field handles borrow their native owner; owned allocatable results retain persistent wrapper-owned descriptor storage. An unallocated or unassociated descriptor remains a present handle whose `to_numpy()` result is `None`. Allocatable `intent(inout)` descriptor arguments accept handles and project the same caller handle, while ordinary arrays and ordinary array results keep NumPy data-buffer semantics. Pointer targets remain non-owning, and unsupported result or reassociation shapes fail readiness. | Complete descriptor kind, handle kind, owner retention, extraction, mutation, release, and operation permissions in post-IR policy before lowering. Route module, field, argument, and result generation through named handle-policy dispatch. Keep borrowed views, detached read-only copies, and descriptor handles distinct; never fall back from an incomplete handle policy to an ndarray copy contract. Block allocatable scalar derived-type replacement until ownership and destruction policy is defined. | +| Pointer and allocatable ownership | Allocatable and pointer arrays use explicit descriptor handles. Module and derived-field handles borrow their native owner; owned allocatable results retain persistent wrapper-owned descriptor storage. An unallocated or unassociated descriptor remains a present handle whose `to_numpy()` result is `None`; otherwise `to_numpy()` returns a current live view and callers use `.copy()` explicitly for independent storage. Allocatable `intent(inout)` descriptor arguments accept handles and project the same caller handle, while ordinary arrays and ordinary array results keep NumPy data-buffer semantics. Rank-zero derived module allocatables/pointers use nullable live member proxies. Wrapper-owned allocatable and pointer derived results use persistent typed holders. Module allocatable dummies use reversible `move_alloc` holder transactions; module pointer dummies use typed association transactions and exact restoration. C transports opaque holder addresses and typed operation pointers, never descriptors. A pointer holder owns its association container, not an unknown native target. | Complete descriptor kind, handle/storage kind, actual declaration, dummy form, owner retention, live extraction or member mechanism, mutation/writeback, release, transaction cleanup, and operation permissions in post-IR policy before lowering. Route module, field, argument, and result generation through named policy dispatch. Keep contiguous-view, descriptor-view, scoped-reference, module-transaction, and typed-holder mechanisms distinct; never fall back from incomplete policy to a copy, fabricated address, or compiler-private descriptor. | | Assumed-rank, assumed-type, and optional descriptor-heavy arguments | Descriptors such as `dimension(..)` and `type(*)` can accept many native shapes that Python cannot infer safely. | Represent descriptor category, rank constraints, element type availability, optional presence, and contiguity. Generate wrappers only for explicit accepted rank/dtype policies or through backend shims that validate descriptors. | | Generic interfaces and operators | Named generics, defined operators, named operators, and defined assignment now preserve explicit concrete-target links. Python cannot intercept `=`, arbitrary named operators, or infer safe in-place mutation. Static extension-type inheritance is represented in Python, and scalar polymorphic input dispatch reuses the same generated overload selection path. | Use Python data-model slots for intrinsic operators, `operator_name`/`r_operator_name` methods for named operators, and mutating `assign` methods for defined assignment. Keep exact dtype/rank/extension-class dispatch and reject indistinguishable signatures during generation. | | Coarrays, teams, events, and directive-driven device/offload behavior | These introduce parallel runtime or device-memory semantics outside normal host wrappers. | Treat as out of the initial wrapper scope. Preserve diagnostics where detected and require a separate runtime design before claiming support. | @@ -62,7 +62,7 @@ X2PY_C_DOCS_END --> | `character(len=...)` and character ABI | Mapping all character forms to `String` loses length, kind, hidden length arguments, fixed buffers, and `bind(c)` byte-string behavior. | Represent character storage with length expression, kind, assumed-length status, array shape, and C-interoperability metadata. Require explicit encoding, termination, copy, and hidden-length ABI handling in wrapper policy. | | Polymorphic `class(...)` and unlimited polymorphism | Static extension-type inheritance is represented by Python C-type inheritance. Scalar `class(base), intent(in)` dummies are safe when the accepted dynamic types are the closed set of known wrapped base/descendant classes, but replacement, allocation, pointer association, results, and unlimited polymorphism still need stronger contracts. | Preserve the `class(...)` source fact. Allow concrete type-bound passed-object arguments. For scalar `class(base), intent(in)` arguments, generate concrete dispatch candidates through the normal overload dispatcher, ordered from descendants to base. Block polymorphic results, arrays, `intent(out)`/`intent(inout)`, allocatable scalars, pointer scalars, and `class(*)` until wrapper policy defines accepted dynamic types, allocation behavior, and ownership. Keep `class(*)` under the assumed-type descriptor blocker. | | Advanced type-bound procedure details | Default `pass`, explicit `pass(name)`, `nopass`, concrete type-bound generics, concrete type-bound operators, and concrete overrides are preserved and wrapped. Finalizers and deferred bindings still need stronger contracts. | Preserve complete binding metadata on semantic classes. Type-bound generics and operators use explicit `.pyi` `@overload("specific")` links and generated C-extension dispatch; unresolved or deferred targets are readiness blockers. | -| Derived-type layout and interoperability | `sequence`, `bind(c)`, common ABI expectations, and component layout are wrapper-critical but not yet a complete runtime contract. | Add explicit Fortran derived-type markers and metadata for `bind(c)`, `sequence`, component order, and interoperable layout. Use compiler layout probes or generated Fortran/C shims before passing derived types by value or exposing memory views. | +| Derived-type layout and interoperability | `sequence`, `bind(c)`, component order, and layout remain semantic facts, but all Python wrappers are opaque. Exact monomorphic by-value dummies are called by typed Fortran bridge code; the C boundary never mirrors or byte-copies the aggregate. | Preserve the exact qualified native type and by-value fact before lowering. Keep component access in typed Fortran operations. Require compiler-proved layout only for a future direct C view, never for the opaque typed-value path. | X2PY_C_DOCS_END --> ## Settled Scope diff --git a/docs/maintainer/roadmap/native-array-handle-checklist.md b/docs/maintainer/roadmap/native-array-handle-checklist.md index d51bedbc1..fdf3ec1bd 100644 --- a/docs/maintainer/roadmap/native-array-handle-checklist.md +++ b/docs/maintainer/roadmap/native-array-handle-checklist.md @@ -71,12 +71,11 @@ storage. Array handles keep a different rule: `Allocatable[T[...]] | None` and spelling after migration. - [x] `Annotated[T[...], Pointer]` is not an active public pointer-array spelling after migration. -- [x] `Snapshot[T]` is removed from generated and accepted active semantic - `.pyi` contracts for this feature. -- [x] Whole-object snapshots are treated as a future feature, not part of this - contract. -- [x] Borrowed views, detached snapshots, and descriptor handles remain distinct - concepts in docs, diagnostics, runtime names, and tests. +- [x] `Snapshot[T]` is not an allocatable- or pointer-array extraction mode and + is no longer an active public contract. +- [x] Live native-array views, explicit user-requested NumPy copies, live + derived objects, and descriptor handles remain distinct concepts in docs, + diagnostics, runtime names, and tests. ## Public `.pyi` Examples @@ -229,8 +228,8 @@ changes. - [x] Document that `Allocatable[T[...]]` is a handle, not an ndarray. - [x] Document that `Pointer[T[...]]` is a handle to pointer association state, not an ndarray. -- [x] Document that `h.to_numpy()` returns a borrowed view, read-only detached - copy, or `None` depending on completed policy and current allocation state. +- [x] Document that `h.to_numpy()` returns a live view of the current allocation + or `None`, never an automatic detached copy. - [x] Document that `p.to_numpy()` returns the current target view or `None`, and can expose strided pointer targets when descriptor support is available. - [x] Document that passing a handle to a handle parameter is descriptor @@ -260,8 +259,8 @@ changes. `allocate()`, `deallocate()`, and `resize()` require explicit policy. - [x] Document stale-view hazards after descriptor-changing operations, reassociation, nullification, deallocation, or reallocation. -- [x] Remove active public examples of `Annotated[T[...], Allocatable]`, - `Annotated[T[...], Pointer]`, and `Snapshot[T]` for this feature. +- [x] Remove active public examples of `Annotated[T[...], Allocatable]` and + `Annotated[T[...], Pointer]` for this feature. ### 2. Public Contract Symbols, Parser, And Printer @@ -281,9 +280,9 @@ Implement the public contract wrappers once and parameterize by descriptor kind. explicit `| None` spelling. - [x] Preserve normal `T[...]` type identity as array data semantics, not descriptor semantics. -- [x] Reject `Snapshot[T]` in active `.pyi` contracts with a clear diagnostic. -- [x] Remove `Snapshot` from `x2py.contracts` and `CONTRACT_SYMBOLS` once - active tests and docs are migrated. +- [x] Do not interpret `Snapshot[T]` as a native-array descriptor wrapper. +- [x] Remove the obsolete public `Snapshot` contract, its semantic `.pyi` + parsing/printing, generated contracts, and recursive derived-object lowering. - [x] Reject or fully migrate `Annotated[T[...], Allocatable]` from active public contracts. - [x] Reject or fully migrate `Annotated[T[...], Pointer]` from active public @@ -327,8 +326,8 @@ descriptor-kind field rather than separate unrelated models. nullable-value path, not on array handle types. - [x] Preserve `T[...]` arguments/results as normal array data in semantic IR even when runtime may later accept a handle through data coercion. -- [x] Model `Snapshot[T]` as unsupported or future-only rather than active - semantic IR. +- [x] Verify `Snapshot[T]` is absent from active semantic IR and remains + unrelated to native-array-handle extraction. ### 4. Post-IR Policy Completion @@ -381,11 +380,9 @@ blocker still prevents wrapper lowering. - `borrowed_view`; - `descriptor_view`; - `contiguous_view`; - - `copy_only`; - - `read_only_detached_copy`; - `unsupported`. -- [x] Complete pointer C-descriptor interop requirement before lowering: - `none` or `pointer_c_descriptor`. +- [x] Complete standard C-descriptor interop requirement before lowering: + `none`, `module_allocatable_c_descriptor`, or `pointer_c_descriptor`. - [x] Complete nullability and optional-absent-handle behavior before lowering. - [x] Complete contract-value storage mode before lowering: `stack`, `heap`, or `alias`. @@ -399,12 +396,11 @@ blocker still prevents wrapper lowering. #### Allocatable Policy Items - [x] Complete allocated-state support. -- [x] Complete addressability/aliasability policy for `h.to_numpy()`. -- [x] Use `borrowed_view` only when safe and legal addressability is proven. -- [x] Use read-only detached copy when live aliasing is not safe. -- [x] Do not expose live NumPy views for non-addressable Fortran allocatables. -- [x] Do not call a read-only borrowed view a snapshot; snapshots are detached - copies. +- [x] Complete live-view mechanism independently from `Aliased`: direct + borrowed access where legal, otherwise standard descriptor access. +- [x] Give plain and `Aliased` module allocatable handles the same native-owned + borrowed lifetime, mutability, and live-view-or-`None` public behavior. +- [x] Block unsupported descriptor extraction explicitly instead of copying. - [x] Complete `deallocate()` permission. - [x] Complete `resize(shape)` permission. - [x] Complete function-result ownership as wrapper-owned stable descriptor @@ -419,12 +415,11 @@ blocker still prevents wrapper lowering. - [x] Complete `to_numpy()` extraction policy: - descriptor view; - contiguous view; - - copy-only fallback if explicitly implemented; - unsupported. - [x] Select pointer `to_numpy()` policy from completed `PointerPolicy(...)` - facts before lowering: contiguous copy requests use `copy_only`, other - contiguous policies use `contiguous_view`, and strided/general policies use - `descriptor_view`. + facts before lowering: contiguous targets use `contiguous_view`, and + strided/general targets use `descriptor_view`. A copy-oriented pointer policy + may retain unrelated meaning, but must not make extraction copy. - [x] Complete `nullify()` permission as the default pointer descriptor operation. - [x] Complete a default conservative handle profile for plain @@ -529,10 +524,10 @@ Add or reuse one internal runtime base for both public handle classes. handles produce present descriptor facts whose `base_addr` may be null. - [x] Carry the completed `.to_numpy()` extraction policy on the runtime handle. -- [x] Apply read-only detached-copy policy in the runtime handle, after the - generated operation supplies current storage. +- [x] Remove detached-copy dispatch from the runtime handle; extraction-enabled + operations must supply live storage or standard descriptor facts. - [x] Validate generated `.to_numpy()` operations return either a NumPy array or - `None` before applying borrowed-view, descriptor-view, or detached-copy + `None` before applying borrowed-view, descriptor-view, or contiguous-view policy. - [x] Validate non-`None` `.to_numpy()` results against the handle's declared dtype and rank before returning them to Python. @@ -546,8 +541,8 @@ Add or reuse one internal runtime base for both public handle classes. `to_numpy_policy` to provide the generated `to_numpy` operation at construction time; handles without extraction support must use `to_numpy_policy="unsupported"`. -- [x] Enforce contiguous-view and copy-only `.to_numpy()` policies in the - shared runtime handle. +- [x] Enforce live contiguous-view and descriptor-view `.to_numpy()` policies + in the shared runtime handle without a copy fallback. - [x] Implement `AllocatableArray` as a descriptor-specific subclass. - [x] Require allocatable runtime handles to provide the generated `allocated` operation at construction time. @@ -569,12 +564,12 @@ below. - [x] `h.deallocate()` - [x] `h.resize(shape)` - [x] `h.to_numpy()` returns `None` when unallocated. -- [x] `h.to_numpy()` returns a live mutable borrowed view when - addressability/aliasability policy proves it safe. -- [x] `h.to_numpy()` returns a read-only detached NumPy snapshot when live - aliasing is not safe. +- [x] `h.to_numpy()` returns a live mutable view for every supported allocated + handle, using direct or descriptor access as selected by completed policy. - [x] Users can call `.copy()` on the returned NumPy array when they need independent lifetime. +- [x] Existing views may become stale after descriptor-changing operations; + accessing stale views is unsupported and may crash. #### Pointer Runtime API @@ -870,9 +865,9 @@ handoff when the wrapped call is native. blocked unless explicit pointer policy allows them. - [x] Verify unsafe/user-responsibility deallocation is available only through the explicit policy value and never by default. -- [x] Verify completed `PointerPolicy(...)` facts select `copy_only`, - `contiguous_view`, or `descriptor_view` before lowering, and only - descriptor-view paths request pointer C-descriptor interop. +- [x] Verify completed `PointerPolicy(...)` facts select `contiguous_view` or + `descriptor_view` before lowering, never an extraction-only copy action, and + only descriptor-view paths request pointer C-descriptor interop. - [x] Verify pointer C-descriptor interop requirements produce an explicit readiness blocker while that interop path is unavailable. @@ -908,7 +903,7 @@ handoff when the wrapped call is native. - [x] Verify owned handles are marked closed after a failing destroy attempt, preventing finalizer retries of the same generated release operation. - [x] Verify generated `.to_numpy()` operations cannot return non-NumPy objects - from borrowed-view or detached-copy policies, and cannot return non-NumPy + from borrowed-view or contiguous-view policies, and cannot return non-NumPy objects from descriptor-view policy unless the value is a decoded pointer descriptor field mapping or field-record object. - [x] Verify generated `.to_numpy()` arrays and decoded pointer descriptor @@ -921,8 +916,8 @@ handoff when the wrapped call is native. descriptor state. - [x] Verify extraction-enabled handles without generated `to_numpy` fail at construction, while unsupported extraction raises the completed-policy error. -- [x] Verify contiguous-view policy rejects non-contiguous arrays and copy-only - policy returns detached NumPy storage. +- [x] Verify contiguous-view policy rejects non-contiguous arrays and never + copies; descriptor-view policy preserves validated shape and strides. - [x] Verify the internal runtime array-actual hook rejects absent descriptor state and uses generated handoff ops instead of `to_numpy()`. - [x] Verify the internal runtime array-actual hook validates expected dtype, @@ -970,12 +965,12 @@ handoff when the wrapped call is native. - [x] `h.allocated` updates after allocate, deallocate, and resize. - [x] `h.shape` updates after allocate, deallocate, and resize. - [x] `h.to_numpy()` returns `None` when unallocated. -- [x] `h.to_numpy()` returns a mutable borrowed view when - aliasable/addressable. -- [x] Mutating a borrowed view mutates native storage when policy allows a live - view. -- [x] `h.to_numpy()` returns a read-only detached snapshot when not - aliasable/addressable. +- [x] Plain and `Aliased` allocated module handles both return mutable live + views, and mutating either view updates native module storage. +- [x] A fresh extraction follows allocation, deallocation, resize, and + reallocation state; an explicit `.copy()` remains independent. +- [x] Tests state the stale-view contract without dereferencing deliberately + stale storage. - [x] Derived allocatable field is a handle object. - [x] Derived-field handle keeps the parent wrapper alive. - [x] Derived-field `deallocate()` operates on `parent%field`. @@ -1013,8 +1008,8 @@ handoff when the wrapped call is native. no descriptor-extraction `to_numpy()` operation is generated. - [x] Runtime pointer descriptor-view extraction validates required decoded TS29113 fields before constructing a NumPy view. -- [x] If C descriptors are unavailable, test the selected explicit fallback: - contiguous-only view, copy fallback, or clear readiness diagnostic. +- [x] If C descriptors are unavailable, test the selected explicit behavior: + contiguous live view or clear readiness diagnostic, never a copy fallback. - [x] Pointer `deallocate()` and `resize()` are absent or raise when policy disallows them. - [x] Pointer `allocate()`, `deallocate()`, and `resize()` work only when diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index d86fecbbf..7d71f3b2a 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -423,6 +423,13 @@ parity. Route selection is atomic per merged extension: a generation unit uses either the direct wrapper-plan route or the legacy route. It never combines one backend from one route with the other backend from the other route. +The documented public contract is authoritative when it intentionally corrects +legacy behavior. In that case, use the legacy implementation to simplify the +mechanical ABI, conversion, ownership, and cleanup audit, improve the design +where the legacy path is unsafe or unnecessarily complex, and record every +intentional behavioral difference in focused tests. Do not preserve a known +legacy defect merely to obtain byte-for-byte or semantic parity. + An unsupported owner may select the legacy route before planning. Once the plan route is selected, planning, validation, lowering, printing, or compilation failure fails the build; it must not fall back to legacy generation. @@ -586,10 +593,10 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 94 | +| `wrapper-plan` | 236 | | `dual-route` | 5 | -| `legacy` | 123 | -| `not-applicable` | 96 | +| `legacy` | 113 | +| `not-applicable` | 95 | | `deferred-real-library` | 2 | #### Recorded Route Progression @@ -618,7 +625,10 @@ blockers. | Phase 6G raw array addresses | 80 | 5 | 130 | 95 | 2 | 312 | | Phase 6 `COPY_F` representation copy | 81 | 5 | 130 | 95 | 2 | 313 | | Phase 7 native handles/descriptors | 88 | 5 | 129 | 96 | 2 | 320 | -| Phase 7 production route reconciliation | 94 | 5 | 123 | 96 | 2 | 320 | +| Phase 7 production route reconciliation | 94 | 5 | 123 | 95 | 2 | 319 | +| Phase 8 scalar-derived object lifetimes | 106 | 5 | 123 | 95 | 2 | 331 | +| Phase 8 complete scalar-derived actual/dummy matrix | 213 | 5 | 123 | 95 | 2 | 438 | +| Phase 8H failure, qualified-type, and typed-value closure | 222 | 5 | 123 | 95 | 2 | 447 | Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating @@ -696,30 +706,33 @@ already covered by the new generator. | `tests/wrapper/fortran/build_from_source/test_compiler_verbose.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | | `tests/wrapper/fortran/build_from_source/test_runtime_abi.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | | `tests/wrapper/fortran/build_from_source/test_source_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines | `legacy` | -| `tests/wrapper/fortran/callbacks/test_array_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines | `wrapper-plan` | +| `tests/wrapper/fortran/callbacks/test_array_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/callbacks/test_callback_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/callbacks/test_derived_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; derived types/snapshots | `legacy` | -| `tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots | `legacy` | -| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/callbacks/test_derived_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; derived types/object lifetimes | `wrapper-plan` | +| `tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `legacy` | | `tests/wrapper/fortran/derived_types/test_derived_type_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/snapshots; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::*` | reduced passing legacy/source artifacts compared with direct typed-plan generation; plain non-target module objects intentionally use the safer member-proxy correction described in Phase 8 | scalar derived arguments/results; optional and by-value inputs; projected identity; owned/borrowed lifecycle; plain/`Aliased` module objects; scalar/string/array/nested/native-handle fields; production routing | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::*` | reduced direct-plan bound-constructor runtime and artifact proof | explicit bound construction; shared method plan; allocation and owner commit | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py::*` | complete source/generated-contract and direct-plan proof over the canonical scalar-derived matrix fixture; replaces the former isolated descriptor rejection unit; final Phase 8H cross-suite verification remains a separate closure gate | all five actual declarations from module and wrapper origins; all six dummy forms; exact action/error selection; holder, scoped-address, allocation and pointer transactions; mixed multi-argument acquisition and reverse cleanup | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes` | reduced module-only contract with deliberate legacy rollback comparison | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/snapshots; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/object lifetimes; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; raw array addresses completed by Phase 6G; derived result remains Phase 8 | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entries over existing vector/matrix native routines | raw numeric addresses; visible scalar-storage extents; rank one/two; default C and explicit Fortran orientation; mutation; integer-only conversion | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_copy_f_preserves_logical_axes_through_binding_owned_temporary` | reduced edited semantic `.pyi` entries over the existing matrix native routine | explicit C-to-Fortran representation copy; native-input and inout calls; projected original identity; binding-owned copyback and cleanup | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fnative_call_examples_f90` native unit | fixed mutable rank-zero NumPy bytes storage; raw fixed-string addresses; in-place mutation; rank/dtype/itemsize/writability/type validation | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `not-applicable` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/snapshots; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `not-applicable` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | | `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; scalar module visibility and namespace projection | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | @@ -734,8 +747,8 @@ already covered by the new generator. | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | scalar external symbols; explicit bridge interfaces | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_package_entry_rejects_non_external_root_declaration_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | | `tests/wrapper/fortran/function_calls/test_function_call_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `legacy` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `legacy` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | production plan route with deliberate legacy rollback comparison | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | @@ -743,7 +756,7 @@ already covered by the new generator. | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `dual-route` | | `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/snapshots; native handles/descriptors | `legacy` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/object lifetimes; native handles/descriptors | `legacy` | | `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `legacy` | @@ -751,9 +764,9 @@ already covered by the new generator. | `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes` | reduced owned-result plus projected-descriptor contract with deliberate legacy rollback comparison | direct persistent descriptor mutation; allocation/reallocation/deallocation; same-handle result identity | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | source/generated-.pyi parity with one mixed generation unit | derived class/field handles and parent retention remain Phase 8/9 blockers | `legacy` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | production plan route in source/generated-.pyi parity modes | rank-zero allocatable/pointer arguments, writeback, results, and copied nullable module values | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction[*]` | production plan route in source/generated-.pyi parity modes | detached read-only snapshots of plain allocatable module arrays; capsule-owned lifetime | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view[*]` | production plan route after the Phase 7 contract correction | plain and `Aliased` module handles return a current live view or `None`; explicit `.copy()` is independent and a fresh extraction follows current native state | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | scalar calls with internal common-block storage | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/snapshots | `legacy` | +| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/object lifetimes | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | | `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route[*]` | production plan route with deliberate legacy rollback comparison | scalar inputs/results; scalar module variables/state; build/artifact integration | `wrapper-plan` | @@ -766,6 +779,7 @@ already covered by the new generator. | `tests/wrapper/fortran/naming/test_defined_operators.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; operators; generic dispatch | `legacy` | | `tests/wrapper/fortran/naming/test_generic_interfaces.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; generic dispatch | `legacy` | | `tests/wrapper/fortran/naming/test_naming_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | +| `tests/wrapper/fortran/naming/test_phase9_class_overloads.py::*` | reduced direct-plan constructor and method overload runtime proof | class-owned exact predicates; constructor ownership; no speculative calls | `wrapper-plan` | | `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | | `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::*` | full BLAS/LAPACK wrapper generation unit | external symbols/native linkage; build/compile/link orchestration; broad wrapper corpus | `deferred-real-library` | @@ -2142,10 +2156,13 @@ generated shape without dereferencing an invalid address. ## Phase 7 — Native Array Handles And Descriptors -Implementation status: complete, pending the final verification gate recorded -at the end of this phase. Phase 6G completed first. The direct route now owns -the dependency-closed Phase 7A-H cases while every field, pointer-result, -callback, and deferred-real-library exclusion remains on its later blocker. +Implementation status: reopened for the view-only `to_numpy()` contract +correction. The previously completed direct Phase 7A-H slices remain evidence +for unaffected descriptor handoffs, but Phase 7 is not closed again until +plain and `Aliased` module-array handles both return a current live view or +`None` without an implicit copy and the final verification gate is rerun. +Every field, pointer-result, callback, and deferred-real-library exclusion +remains on its later blocker. Scope: migrate the existing native descriptor and runtime-handle contract into the wrapper-plan path without redefining that public contract. The maintained @@ -2178,6 +2195,15 @@ genuinely differ: allocation state versus association state, allowed shape-changing operations, target lifetime, extraction policy, and release. Do not create independent allocatable and pointer planner hierarchies. +For every rank-positive module handle, `to_numpy()` has one public result: +`None` for an unallocated/unassociated native object and a live NumPy view of +the current allocation/target otherwise. Plain and `Aliased` allocatable +module variables use the same behavior. `Aliased` remains semantic metadata +but never selects a detached copy. Users call `.copy()` explicitly for +independent storage; an old live view may become stale after native +deallocation, reallocation, nullification, or reassociation, and a fresh +`to_numpy()` call must inspect current native state. + ### Phase 7 Boundary And Explicit Non-Scope The following four boundaries must remain distinct: @@ -2208,6 +2234,12 @@ Other exclusions and dependencies are: - derived-type field attachment, class construction, parent-wrapper creation, and property orchestration, which require Phases 8 and 9 even though the shared native-handle plan must already be reusable by those later owners; +- scalar derived module-variable member access and argument compatibility, + which belong to Phase 8. Phase 7 descriptor machinery remains limited to + array handles; scalar derived module allocatables use the exact local + move-out/move-back route specified in Phase 8H and do not consume Phase 7 CFI + descriptor machinery. A failed scalar-object call handoff must not become a + module-access blocker; - pointer results without completed stable owner storage and target lifetime; - callback descriptor arguments or results, which remain in Phase 10; - compiler-private descriptor layout inspection or copying; @@ -2667,23 +2699,60 @@ and handle creation; and the current runtime handle factory. state, extraction policy, operation permissions, stale-view behavior, and module lifetime. - [x] Use `test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, - `test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction`, + `test_plain_allocatable_module_array_exposes_current_live_view`, and the module portion of `test_module_and_derived_pointer_handles_track_native_association` as legacy oracles. Split out field/class assertions, which remain Phase 8/9 work. +#### Phase 7F Contract Correction — View-Only Module Extraction + +The checked Phase 7F items above record the original migration slice; they do +not close this changed public contract. Complete this correction before Phase +8 implementation. + +- [x] Update public docs, maintainer docs, generated/checked semantic `.pyi` + evidence, and wrapper coverage rows to specify current live view or `None`, + explicit `.copy()`, and the unsupported stale-view window. +- [x] Complete plain and `Aliased` allocatable module arrays as native-owned + borrowed handles with the same extraction result. Keep addressability, + descriptor mechanism, owner retention, mutability, nullability, storage, + operation permissions, and release responsibility as separate completed + facts. +- [x] Remove `read_only_detached_copy` and extraction-only `copy_only` policy, + plan, runtime, binding, and bridge dispatch. Preserve only typed live-view + mechanisms such as contiguous or standard-descriptor views; unsupported + extraction fails instead of copying. +- [x] For a plain allocatable module array, add the completed standard- + descriptor module-state mechanism needed to inspect the current allocation + on each extraction. Keep it beneath `ModuleVariablePlan`; do not retain a + descriptor or data address as if it were permanently current. +- [x] Keep binding/bridge ownership explicit: the bridge exposes current native + descriptor facts without NumPy knowledge, and the binding validates + dtype/rank/shape/strides and creates the NumPy view with its handle owner as + the base. Native-handle argument handoff must not call `to_numpy()`. +- [x] Replace the obsolete read-only-copy test with source/generated-`.pyi` + parity covering plain and `Aliased` live mutation, allocated/unallocated and + associated/unassociated state, fresh extraction after state changes, + explicit-copy independence, stale-view documentation, parent/owned-result + retention, and contiguous/strided pointer views. +- [x] Rerun focused policy/plan/backend/runtime tests, documentation checks, + the wrapper suite excluding LAPACK, the wrapper-codegen complexity checker, + and the required static-analysis suite before closing Phase 7 again. + ### Phase 7G — Pointer Descriptor Extraction And Build Requirements -Included: pointer `descriptor_view`, `contiguous_view`, `copy_only`, and -explicitly unsupported extraction actions already selected by completed -policy; standard descriptor decoding; positive and negative strides; and local -build/header requirements. +Included: pointer `descriptor_view`, `contiguous_view`, and explicitly +unsupported extraction actions already selected by completed policy; standard +descriptor decoding; positive and negative strides; and local build/header +requirements. A `copy_only` `to_numpy()` action is obsolete and must not reach +the corrected plan. -Only `descriptor_view` and persistent allocatable owner storage require standard -C descriptor support. Generated code may read `CFI_cdesc_t` through -`ISO_Fortran_binding.h` when the completed plan requests it. It must never guess -or expose a compiler-private descriptor layout. Unsupported toolchains fail -readiness/build with the completed owner path and requirement. +Descriptor views, the corrected plain allocatable module-state path, and +persistent allocatable owner storage require standard C descriptor support. +Generated code may read `CFI_cdesc_t` through `ISO_Fortran_binding.h` when the +completed plan requests it. It must never guess or expose a compiler-private +descriptor layout. Unsupported toolchains fail readiness/build with the +completed owner path and requirement. - [x] Carry typed extraction and descriptor-interop actions plus required headers into handle/module/result plans and rendered artifact metadata. @@ -2790,7 +2859,7 @@ Phase 7 rows were split, proved through both routes, and then recorded as | `module_state/test_allocatable_replacement.py::*` | projected same-handle descriptor mutation plus a derived factory generation unit; `legacy` | Phase 7D reduced parity is `wrapper-plan`; the broad factory/class unit remains Phase 8/9 | | `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | module, result, and derived-field mix; `legacy` | field/class owner retention remains Phase 8/9 | | `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | scalar descriptor arguments/results/module state; `wrapper-plan` | source conversion records descriptor kind and argument/return reference before completed Phase 7H policy | -| `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_handle_with_read_only_extraction[*]` | plain allocatable module snapshot; `wrapper-plan` | Phase 7F binding-owned detached read-only snapshot and capsule lifetime | +| `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view[*]` | corrected source/generated-`.pyi` production-plan evidence | proves Phase 7F plain/`Aliased` current live-view or `None` parity, native mutation, explicit-copy independence, and fresh extraction after state changes | | `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | fixed strings plus deferred allocatable result; `legacy` | Phase 7H reduced deferred-result parity; retain mixed row as needed | | `edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | includes raw `Addr(Float64[n])` plus a derived result; `legacy` | raw-array subset is the completed Phase 6G prerequisite; derived subset remains Phase 8 | @@ -2835,83 +2904,1728 @@ Required focused intermediate coverage includes: - [x] Preserve the completed Phase 6G raw-address boundary while keeping every derived-field, pointer-result, callback, and deferred-real-library exclusion on its explicit later blocker. -- [x] Run focused policy/plan/backend tests, relevant runtime-handle tests, +- [x] Complete the Phase 7F view-only correction for plain and `Aliased` + allocatable module arrays and remove every implicit-copy extraction path. +- [x] Rerun focused policy/plan/backend tests, relevant runtime-handle tests, documentation checks, the wrapper suite excluding LAPACK, the wrapper-codegen - complexity checker, and the required static-analysis suite before declaring - implementation complete. + complexity checker, and the required static-analysis suite after the + correction. - [x] Close Phase 7 only when every live non-field native-handle/descriptor case is migrated or explicitly removed from the product contract, and no backend infers descriptor policy or silently substitutes a data buffer, raw pointer, `.to_numpy()` extraction, or copy fallback. -Closure evidence: 538 focused semantic/policy/plan/backend/runtime tests, 1,133 -documentation and layout tests, and all 318 wrapper tests outside the deferred -BLAS/LAPACK file pass. The wrapper-codegen complexity check, Ruff, formatting, -Bandit, Vulture, whitespace, and explicit-base Radon policy pass; full Radon -complexity and maintainability reports were also recorded. The source/default -scalar-descriptor projection now records its descriptor kind and argument or -return reference before policy completion. Binding and bridge lowering dispatch -only from the resulting typed plans; the remaining mixed derived-field unit is -an explicit Phase 8/9 blocker rather than a descriptor fallback. +Historical pre-correction evidence: 538 focused semantic/policy/plan/backend/ +runtime tests, 1,133 documentation and layout tests, and all 318 wrapper tests +outside the deferred BLAS/LAPACK file passed. The wrapper-codegen complexity +check, Ruff, formatting, Bandit, Vulture, whitespace, and explicit-base Radon +policy also passed. This evidence remains valid for unaffected sub-lanes but is +not closure evidence for the changed view-only extraction contract. Record a +new success signal after the Phase 7F correction. + +Post-correction closure evidence (2026-07-14): 214 focused runtime-handle, +policy, readiness, lowering, legacy-dispatch, and Phase 7 direct-plan tests; +199 complete `tests/wrapper_codegen` tests; 1,123 documentation tests; 317 +wrapper tests outside the shared real-library parameter plus the BLAS-only +parameter; and zero locally executed LAPACK tests all passed. The wrapper +complexity checker, Ruff lint/format, Bandit, Vulture, whitespace, and the +explicit-`origin/main` Radon policy passed. The required `--base-ref auto` +Radon invocation could not resolve a CI base SHA locally; the explicit base +rerun passed. Advisory full Radon complexity and maintainability reports were +also produced. + +Phase 7 was re-verified again with the final Phase 8 closure run on +2026-07-15: the 704-test semantic/policy/plan/backend regression batch, all 79 +runtime-handle tests, 1,123 documentation tests, and all 326 wrapper tests +outside LAPACK passed. No LAPACK test was run locally. + +## Phase 8 — Derived Types And Object Lifetimes + +Expansion status: complete. Implementation proceeds only after the Phase 7 +view-only correction is re-verified. + +Implementation status: reopened for the complete rank-zero scalar-derived +actual/dummy compatibility matrix in Phase 8H. The previous Phase 8A-I +evidence remains authoritative for unaffected fields and lifecycle paths, but +the old module-allocatable rejection, nonreassociating pointer-only path, +interoperable-only value restriction, and incomplete module-object call routes +are superseded. Phase 8 must not close again until direct, scoped-address, +wrapper-holder, module-transaction, pointer-input, and typed-value actions are +implemented without a fallback and re-verified with multi-argument calls. + +Scope: migrate scalar derived-type storage, arguments, results, borrowed +objects, and field handoffs into the wrapper-plan route. Phase 8 owns the +opaque native-instance substrate and the typed transfers that use it. Phase 9 +owns public constructors, methods, overloads, inheritance, and general +class-surface orchestration built on that substrate. Phase 8 owns public field +descriptors and their typed getters/setters because every live object origin, +including plain module proxies, needs the same readable and writable member +surface. + +Do not begin implementation while a Phase 7 native-array-handle correction is +open. In particular, Phase 8 must consume the final view-only `to_numpy()` +contract: an array-handle extraction is a live view or `None`, and an +independent array is obtained with an explicit `.copy()`. + +`Snapshot[T]` is no longer an active public contract. Plain and `Aliased` +rank-zero derived module variables both expose the normal live generated object +surface. Their lowering mechanisms remain distinct: `Aliased` proves a direct +address-backed borrow, while a plain declaration requires typed module-specific +bridge access and must not fabricate a native address. `Aliased` remains an +addressability and aliasing fact for raw-address legality, pointer association, +C-pointer policy, and direct derived-object handoff; it does not select +array-handle `to_numpy()` behavior. + +### Phase 8 Boundary And Explicit Non-Scope + +The first implementation slice is rank-zero, non-polymorphic derived values. +The runtime wrapper is opaque: the binding carries a native address, ownership +state, and an optional retained Python owner, while the bridge performs typed +native association, assignment, allocation, and destruction. The binding must +not depend on component offsets or reproduce native aggregate layout. + +The following surfaces are in Phase 8: + +- required and optional scalar derived arguments; +- visible `out` and `inout` wrappers whose identity remains caller-visible; +- hidden output and direct-function-result values materialized as + wrapper-owned instances; +- native `value` arguments for an exact rank-zero monomorphic derived type, + using a Fortran bridge-owned typed value copy rather than C-side layout + inference; the native type need not be `bind(C)` when the bridge imports its + exact definition; +- derived `parameter` and other explicit constant-value origins materialized + through the existing wrapper-owned immutable-value path, never as a fallback + for an ordinary mutable module object; +- plain rank-zero native module objects exposed as live module-backed proxies + through typed bridge operations; +- `Aliased` rank-zero native module objects exposed as live borrowed wrappers; +- borrowed nested component wrappers, their public field descriptors, and the + owner-retention facts required by those descriptors; +- Phase 7 allocatable/pointer field-handle plans attached to a derived owner; +- exact destruction, finalization, cleanup, and failure ownership for each of + those origins. + +The following remain outside Phase 8: + +- public default/keyword constructors, explicit `@bind(...)` constructors, + `tp_init`, methods, static methods, overload dispatch, Python inheritance, + and ordinary type-bound surface assembly; these remain Phase 9. Public field + descriptors, getters, and setters are Phase 8 and are not a Phase 9 blocker. + A generated semantic `.pyi` field constructor is therefore a whole-unit + Phase 9 blocker; only an opaque contract that suppresses default construction + may use the direct Phase 8 object route; +- scalar polymorphic dispatch and inheritance even where the legacy route + supports them; Phase 9 owns the class relationship needed to validate the + accepted runtime type set; +- callback-derived arguments and results, adapter procedures, and trampoline + ownership; these remain Phase 10; +- arrays of derived values, whose element layout, construction, destruction, + copy, and partial-failure behavior remain explicit readiness blockers; +- polymorphic results, mutable polymorphic arguments, `class(*)`, abstract + instantiation, deferred bindings, and allocatable/pointer polymorphic + scalars; +- polymorphic descriptor-backed scalars. Wrapper-owned allocatable and pointer + holders plus scalar derived module ordinary/`TARGET`/allocatable/pointer + variables are supported only by their explicit Phase 8H matrix rows. A + pointer holder owns its association container, never its target by default; + target retention and native release responsibility are completed separately + before lowering. Module allocation and pointer transactions use shared typed + holder addresses in interoperable callbacks, never CFI or a compiler-private + descriptor; +- any other derived origin that cannot use one of the explicit matrix rows. It + remains blocked rather than being silently turned into an address-backed + borrow or detached object; +- C-side aggregate casts, `ctypes` layout promises, compiler-private descriptor + inspection, or direct component offsets; +- ownership of targets reachable through pointer components. A containing + derived wrapper does not own such a target without completed pointer policy; + and +- detached whole-object snapshot classes or recursive member-copy graphs. They + are removed rather than retained as a compatibility path. + +### Public Representation And Lifetime Matrix + +Complete this matrix in post-IR policy before adding planner or backend code. +The rows are distinct origins, not datatype guesses made during lowering. + +| Surface | Python representation | Native handoff/storage | Owner and release | +| --- | --- | --- | --- | +| required `in` argument | existing wrapper instance | pass its opaque wrapper address and associate a typed native view for the call | wrapper remains owned by its existing Python object; call destroys nothing | +| required visible `inout` or caller-supplied `out` | same wrapper instance | pass the same address for native mutation | caller-visible wrapper retains identity; its normal wrapper finalizer remains the sole destroyer | +| optional argument, omitted or `None` | no wrapper instance | explicit absence token/branch; no fabricated native object | no allocation or cleanup | +| optional argument, present | validated wrapper instance | same typed address handoff as the required case | existing wrapper owner remains responsible | +| hidden output | new opaque wrapper object | allocate persistent wrapper-owned native storage before the call and pass its address | wrapper deallocator invokes native-aware destruction exactly once | +| direct function result | new opaque wrapper object | move or copy the native result before its temporary expires into persistent wrapper-owned storage | wrapper deallocator invokes native-aware destruction exactly once | +| constructor-created instance | Phase 9 only | Phase 9 must allocate through the same persistent wrapper-owned storage and native-aware destructor established here | explicitly blocked until Phase 9 class construction orchestration; no Phase 8 fallback constructor | +| native `value` input | existing wrapper instance | exact Fortran bridge passes the typed pointee to the native by-value slot; C never lays out or copies the aggregate | call-local native copy only; wrapper ownership is unchanged | +| plain rank-zero module variable | normal live generated object | module-specific typed getter/setter operations plus a synchronous scoped-address consumer when the object is passed to another procedure | native module owns storage; wrapper retains the module and never destroys storage; a temporary target/address cannot escape its consumer scope | +| `Aliased` or explicit `TARGET` rank-zero module variable | live borrowed wrapper | use `C_LOC` as the sole whole-object handoff; reconstruct the exact typed bridge view without copying | native module owns storage; wrapper never destroys it and rejects replacement | +| derived `parameter` or other explicit constant-value origin | wrapper-owned value copy with an immutable module binding | materialize the native value into persistent wrapper-owned storage | wrapper destroys only its materialized copy; no native module setter; normal writable fields modify only that independent copy | +| nested derived field | live borrowed child wrapper | address/alias of the component through the parent wrapper | child retains parent; child never destroys component storage | +| allocatable scalar derived module origin | nullable live module-backed proxy carrying its runtime origin | scoped-address consumer for payload-only calls; for an allocatable dummy, module-specific interoperable operations move between the module variable and a bridge-local shared typed holder addressed by `C_PTR` | native module owns storage before and after a transaction; successful move-out has exactly one reverse-order move-back; no descriptor crosses C | +| pointer scalar derived module origin | nullable live module-backed proxy carrying its runtime origin | current-target address for payload-only calls; for a pointer dummy, a bridge-local shared typed pointer holder receives the initial association and its address is passed to the module-specific restore operation | native module owns the pointer variable and, by default, its target; final association is restored exactly once after a normally returning native call | +| wrapper-owned allocatable scalar derived result | nullable live generated wrapper backed by one persistent typed allocatable holder per native type | result is moved into `holder%value`; ordinary, target, allocatable, allocatable-target, pointer-input, and value dummies use the explicit compatible matrix actions | each Python wrapper owns one target-capable holder and destroys it exactly once; allocation-state writeback preserves wrapper identity | +| wrapper-owned pointer scalar derived result | nullable live generated wrapper backed by one persistent typed pointer holder per native type | holder component stores current association and is passed directly to a compatible pointer dummy; payload-only calls use its associated target | wrapper owns and destroys only the holder; target ownership stays native unless completed policy retains a known wrapper/module target; destruction nullifies the component and never deallocates an unowned target | +| detached whole-object snapshot | removed | no recursive copy graph or snapshot helper is generated | no compatibility parser, lowering, or fallback; read the live object through normal fields instead | + +`Aliased` remains a public, language-neutral addressability/aliasing fact and +must survive parsing, semantic IR, and printing. For derived module objects it +distinguishes direct-address lowering from module-proxy lowering, not live +versus copied public behavior. It must never be reused to select live versus +copied native-array-handle extraction. + +### Existing Semantic Authority And Legacy Oracle + +Use the current implementation as an oracle, not as permission to preserve its +architecture: + +- `x2py/semantics/ownership.py` already names `DERIVED_TYPE`, + `PASS_WRAPPER_ADDRESS`, `WRAPPER_INSTANCE`, and `BORROWED_VIEW`, and contains + the current argument/result/module/field owner defaults. Remove the obsolete + derived whole-object snapshot action without disturbing ordinary result + copies, scalar descriptor value copies, or explicit non-object uses of + `snapshot_copy` transfer policy. +- `x2py/semantics/policy_completion.py` is the only allowed owner of origin, + ownership, transfer, destruction, mutability, nullability, projection, + release, storage, getter/setter, owner-retention, module-object handoff, and + field decisions. It must complete module-proxy policy for plain module + objects and direct-address borrowed policy for `Aliased` module objects. +- `x2py/semantics/wrapper_policy.py` must gain a derived-specific policy branch. + Derived values must not continue through primitive-scalar blockers, + primitive result checks, or primitive bridge data-action selection. +- `x2py/semantics/ir2ast.py` and the legacy generators remain the generated + artifact oracle. Direct lowering must not call `semantic_ir_to_codegen_ast()` + or reconstruct legacy codegen variables. +- `x2py/codegen/bindings/c_to_python.py` contains the existing wrapper-instance + conversion, checked casts, owned/borrowed result construction, owner + retention, and allocator/destructor helpers. Remove recursive snapshot + construction rather than migrating it into the direct route. +- `x2py/codegen/bridges/fortran_to_c.py` contains the existing typed wrapper + address conversion, native result materialization, borrowed field/module + access, native-aware destruction, and typed component getters/setters. Reuse + those live member-access mechanics as the artifact oracle while moving every + decision into typed plans. + +Capture complete legacy artifacts before each direct slice. Preserve observable +runtime behavior while replacing backend inference with completed typed plans. +Do not copy the broad legacy generator control flow into `wrapper_codegen`. + +The existing wrapper tests decompose as follows: + +| Existing test or generation unit | Phase 8 oracle | Required split or later owner | +| --- | --- | --- | +| `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | hidden derived result selected by `Return(...)` | add a reduced object-result entry; retain the mixed unit until every included lane is direct | +| `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | hidden derived output and mixed result aggregation | add a reduced derived-output entry; retain the broad unit until its complete tuple is direct | +| `edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | edited projected derived replacement | isolate `make_point` as Phase 8 evidence; retain the mixed policy unit until whole-unit eligibility follows | +| `derived_types/test_derived_type_boundaries.py::test_scalar_derived_types_cross_procedure_boundaries[*]` | required input, in-place mutation, hidden/direct result, nested borrowed component | reduce first to result-created opaque objects passed back to `point_sum`/`move_point`; field descriptors and nested borrowing are Phase 8, while construction remains Phase 9 | +| `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | optional derived input and exact type/absence behavior | complete the optional transfer in Phase 8; the existing constructor-dependent broad runtime unit remains Phase 9 until it can route whole | +| `module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | native-owned borrowed module object and replacement rejection | use as the direct-address oracle; add a reduced plain-module proxy case with the same live field behavior; methods remain Phase 9 | +| `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component[*]` | parent retention and exactly-once owner finalization | Phase 8 owns storage/lifetime plans and public field descriptors; constructor/method orchestration remains Phase 9 | +| `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | Phase 7 field handle attached to a derived owner | reuse the existing `NativeArrayHandlePlan` and expose its public property in Phase 8 | +| `derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | pointer field handle and parent lifetime | reuse Phase 7 descriptor extraction; do not move pointer target ownership into Phase 8 | +| `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy[*]` | opaque `bind(C)` wrapper, field accessors, and typed native `value` copy | Phase 8 owns the handoff and field properties; constructor orchestration remains Phase 9 | +| former `module_state/contracts/fmodule_derived_snapshot_f90/` snapshot fixture | obsolete detached-object behavior | remove the `Snapshot[box]` fixture and snapshot-only runtime assertions; reuse the native unit only for reduced live module-proxy evidence where applicable | +| `derived_types/test_constructors_and_finalizers.py::*`, `derived_types/test_derived_type_methods.py::*`, and `derived_types/test_inheritance.py::*` | owned-instance finalizer and type facts may inform Phase 8 | production migration remains Phase 9 because the observable unit is constructor/method/property/inheritance owned | +| `callbacks/test_derived_callbacks.py::*` | none | remain Phase 10 even after ordinary derived transfers are complete | + +The plain non-target module-object row has one recorded intentional correction: +the legacy whole-object getter attempts `c_loc` on storage without the required +addressability property and therefore has no passing whole-object artifact. +Phase 8 uses the real source declaration, the passing legacy typed component +getters/setters, and the passing `Aliased` direct-address behavior as its +mechanical oracles, then improves the plain path to a typed member proxy. The +compiled Phase 8 evidence asserts that this proxy never emits a fabricated +whole-object `c_loc` while its reads and writes remain live. + +### Mandatory Phase 8 Migration Algorithm + +Apply this same sequence to every dependency-closed sub-lane. A checked item in +a later step cannot compensate for an incomplete earlier step. + +1. Capture the complete generated artifacts and runtime assertions from one + real passing legacy/source case. Record which constructor/method or + callback assertions remain outside the reduced unit. +2. Complete object kind, origin, ownership, transfer, destruction, mutability, + nullability, projection, storage, release, owner retention, getter/setter, + native assignment, and any module-object mechanism before `ir2ast.py`. +3. Project those facts mechanically into `ArgumentTransferPlan`, `ResultPlan`, + or `ModuleVariablePlan`, with native slots and lifecycle actions remaining + subordinate references. +4. Validate type identity, roles, actions, owners, storage, result positions, + releases, and cross-backend handoffs before either backend emits source. +5. Lower through small named binding and bridge methods selected by typed + object kind and action. Backend-local temporaries remain implementation + details inside the already selected method. +6. Compare binding, bridge, header, and build artifacts with the captured + oracle and explain every intentional difference before compiling. +7. Add focused policy, plan-edit, validation, printer, backend, runtime, + documentation, and source/generated-`.pyi` parity tests. +8. Promote the reduced generation unit only after compiled legacy/direct parity + passes; otherwise retain one exact blocker without a fallback route. + +The maintainer trace is therefore always: + +```text +completed semantic facts + -> typed argument/result/module-variable plan + -> subordinate native slots and lifecycle actions + -> validation + -> binding and bridge lowering + -> generated artifacts + -> compiled runtime evidence +``` -## Phase 8 — Derived Types And Snapshots +### Plan Shape And Stable Action Vocabulary + +Do not create a second function plan, a second result hierarchy, or a rendered +derived-plan layer. Extend the existing plan tree as follows: + +- add one explicit derived datatype family or equivalent non-primitive marker + so a derived semantic type never indexes the primitive scalar dtype maps; +- add a concise namespace-owned derived-type definition record containing + canonical semantic/native identity, native scope, Python exports, opaque + runtime type symbol, allocation role, destruction/finalization role, and the + minimal field identities needed by later field plans; +- add one `DerivedHandoffPlan`-style facet, analogous to `ArrayHandoffPlan`, to + `ArgumentTransferPlan`, `ResultPlan`, `ModuleVariablePlan`, and the owning + `NativeCallSlotPlan` only where that transfer needs it; +- give a derived `ModuleVariablePlan` one typed module-object access facet that + records the completed direct-address or opaque-callback mechanism, its + context/address roles, compiler capability, and module-lifetime owner. This + mechanism is subordinate to the module-variable policy and must not change + its public borrowed-wrapper facts; +- keep native slot order, symbolic roles, and ABI positions subordinate to the + owning argument or result transfer; +- represent result destruction, failed-construction cleanup, parent retention, + through transfer-owned `LifecycleActionPlan` records in function-wide + execution order; +- add field-handoff records beneath the owning derived-type definition. Do not + put their ownership decisions into a backend registry; and +- keep `FunctionPlan`, `ModulePlan`, namespace assembly, result ordering, GIL + envelope, and status-error behavior stable. + +Reuse the existing action vocabulary: + +- Python boundary: `WRAPPER_INSTANCE` for accepted live wrapper objects and + `NONE` for native-produced results; +- native boundary: `PASS_WRAPPER_ADDRESS` for opaque live objects and `NONE` + when the bridge itself owns result production; module-backed proxies use a + distinct typed module-origin handoff rather than a fabricated address; +- transfer/codegen: `CALL_LOCAL_INPUT`, `IN_PLACE_ARGUMENT`, + `IDENTITY_OUTPUT`, `WRAPPER_INSTANCE`, and `BORROWED_VIEW` according to the + completed matrix row; +- bridge data: `DIRECT_TRANSFER`, `ASSOCIATE_VIEW`, or + `COPY_REPRESENTATION`, with a completed copy reason only when a real native + representation copy occurs; and +- lifecycle: existing ordered copy-in/native-mutation/copy-out/cleanup phases, + extended only with a genuinely missing release phase/action rather than a + derived-only parallel lifecycle system. + +If one of these actions cannot express a required operation, document the +missing semantic distinction before adding exactly one typed action. Do not use +method-name strings, datatype conditionals, `intent`, `is_alias`, dotted-name +shape, or local temporary existence as hidden dispatch. + +### Binding, Bridge, And Validation Ownership + +| Layer | Owns | Must not own | +| --- | --- | --- | +| post-IR policy | origin, dynamic/static type allowance, owner, transfer, destruction, mutability, projection, nullability, storage, owner retention, getter/setter behavior, and blockers | emitted local names or source syntax | +| wrapper planner | mechanical projection into derived type/handoff facets, native roles, ordered results, and lifecycle indexes | new ownership or lifetime decisions | +| binding lowering | exact Python type checks, opaque wrapper address extraction, Python wrapper allocation, retained-owner references, result aggregation, and Python reference cleanup | native component layout, native assignment, or native finalization semantics | +| bridge lowering | typed association from opaque addresses, exact Fortran-owned `value` calls, native instance allocation/assignment, module/component access, and native-aware destruction/finalization | Python classes, C aggregate layout, reference counting, detached-copy fallback, or ownership inference | +| plan validation | matching type identity, roles, actions, owners, releases, result positions, and cross-backend handoffs before emission | fallback selection | + +Validation must reject at least: + +- a derived transfer without canonical type identity or an exported runtime + wrapper type; +- a wrapper-address slot whose binding and bridge roles or ABI positions differ; +- a primitive scalar action or datatype family applied to `DERIVED_TYPE`; +- a wrapper-owned result without persistent storage, allocator, destroy action, + or failure cleanup; +- a borrowed wrapper with a destroy action, or without its required native + module/parent owner retention; +- a call-local argument that schedules destruction of the caller's wrapper; +- a visible in-place argument projected as a replacement without completed + policy; +- a hidden output or direct result whose native temporary can escape by + address; +- a plain module proxy without complete typed member-path operations, or an + `Aliased` live module borrow without a completed direct-address handoff; +- binding and bridge module-object access roles that disagree; +- an obsolete `Snapshot` contract, recursive detached-copy action, or backend + fallback that manufactures a detached object; +- a derived array or unsupported polymorphic form entering scalar-derived + lowering; and +- any backend request to infer a class, owner, addressability, or release from + semantic datatype or `intent`. + +### Phase 8A — Contract, Origin, And Post-IR Policy Completion + +Complete the semantic contract before defining direct plan records. + +- [x] Inventory every live scalar derived origin from source and semantic + `.pyi`: constructor-created storage, wrapper-owned result, caller-supplied + argument, native module object, and nested component. +- [x] Introduce one typed completed origin/retention representation shared by + class-instance, argument, result, module-variable, and field policy. + Do not encode origins as ad hoc reason strings. +- [x] Keep generated and edited `.pyi` type identity stable across module + namespaces, imported derived types, renamed Python exports, and same-name + types from different native scopes. +- [x] Complete required, optional, visible `out`, visible `inout`, hidden + output, and direct-result ownership without treating `intent` as the final + Python signature. The editable signature and `@native_call(...)` projection + decide visibility and order; policy only ensures the native call is valid. +- [x] Complete wrapper-owned result storage and destruction, borrowed + module/field owner retention, native setter rejection, result projection, + and failure cleanup before `ir2ast.py`. +- [x] Preserve `Aliased` parsing, printing, and source-derived metadata. Use it + for a live derived-module borrow and direct-address legality, but never as a + native-array extraction mode. +- [x] Complete a plain ordinary module object as `owner=NATIVE`, + `transfer=BORROWED_VIEW`, native-owner destruction, module lifetime, module + owner retention, typed member-path access, and replacement rejection. Do not + claim or require a whole-object native address. +- [x] Complete an `Aliased` module object as `owner=NATIVE`, + `transfer=BORROWED_VIEW`, native-owner destruction, alias storage, module + owner retention, direct address acquisition, and replacement rejection. +- [x] Remove the obsolete public `Snapshot` keyword from `x2py.contracts`, + parser, printer, generated `.pyi`, semantic IR, policy actions, legacy + generators, documentation, and fixtures. Do not remove unrelated explicit + copy-result or scalar descriptor value-copy policy. +- [x] Complete finite typed member-path traversal for plain module proxies. + Memoize derived type identities so recursive graphs do not expand forever; + require explicit pointer/allocatable association, ownership, and stale-child + policy at recursive descriptor-backed edges. +- [x] Remove the obsolete wrapper-owned pointer-result blocker. Complete a + persistent typed pointer-holder origin whose wrapper owns the holder but not + its target, then keep only arrays of derived values and unsupported + polymorphic forms on exact readiness blockers. Supported scalar module + allocatable/`TARGET`/pointer origins use only their explicit Phase 8H + actions. +- [x] Add focused parser, printer, source-conversion, ownership, accessor, + policy-completion, and readiness tests for every active matrix row and + blocker. Assert the deliberate module-proxy versus direct-address mechanism + distinction, their shared live public behavior, and that neither changes a + contained native handle's view-only extraction. + +### Phase 8B — Derived Plan Records And Preflight Validation + +- [x] Add the minimal namespace-owned opaque derived-type definition record and + derived handoff facets described above. Keep all per-call decisions in + `ArgumentTransferPlan`, `ResultPlan`, or `ModuleVariablePlan`. +- [x] Add an explicit derived datatype-family/type-reference representation so + documentation, roles, native slots, lifecycle records, and printers never + fall through primitive scalar maps. +- [x] Project class instance/self policies, native type identity, wrapper type + symbol, native scope, allocator/destroy roles, and finalizer requirements + mechanically from completed semantic policy. +- [x] Project optional presence, input/in-place/output action, native call + position, ownership, storage, owner retention, and result position into the + existing transfer records. +- [x] Share the exact `DerivedHandoffPlan` object with its owning + `NativeCallSlotPlan` where the array/handle lanes already share subordinate + facets; do not duplicate editable state. +- [x] Add recursive validation for the namespace type definitions, arguments, + results, module variables, module-object access facets, field facets, and + lifecycle indexes. +- [x] Make plan edits observable: changing a derived owner, action, type + identity, retained owner, or release must either change both backend + artifacts consistently or fail `_validate_plan()` before source emission. +- [x] Extend support analysis with precise derived lanes and blockers. Do not + remove the blanket class-owner blocker until the minimal opaque type surface + is direct and every remaining Phase 9 dependency is reported separately. +- [x] Add normal-print plan tests and direct generator preflight tests under + `tests/wrapper_codegen/test_phase8_derived_types.py`. + +### Phase 8C — Minimal Opaque Wrapper Storage And Lifecycle + +This sub-lane creates the runtime substrate needed to return and pass opaque +objects. It does not implement public construction, fields, or methods by +itself; Phase 8F/H add the public field surface on this substrate. + +- [x] Emit one minimal runtime wrapper type per exported semantic derived type, + with an opaque native address, an owned/borrowed state, and an optional + retained Python owner. Keep the public constructor unavailable until Phase 9. +- [x] Generate bridge allocation and destruction helpers from completed type + policy. Native-aware destruction owns allocatable components and supported + finalization; the binding must not free native storage directly. +- [x] Ensure owned allocation, initialization, and result conversion failures + run native destruction and Python cleanup exactly once. +- [x] Ensure borrowed wrappers never run native destruction, including when + their retained owner is released through cyclic or delayed garbage + collection. +- [x] Register the minimal type in the correct exported namespace so result and + module-variable materialization use the same class identity in source and + generated-`.pyi` builds. +- [x] Keep wrapper struct/type declaration, allocation, owner retention, and + destruction methods grouped under a derived-type comment in the binding; + keep native allocate/associate/destroy helpers grouped likewise in the + bridge. +- [x] Add source-printer and artifact tests for owned, borrowed, failed + allocation, failed conversion, and exactly-once native destruction paths. + +### Phase 8D — Wrapper-Owned Hidden Outputs And Function Results + +- [x] Plan hidden `Return(...)` outputs and direct derived function results as + `WRAPPER_INSTANCE` results with persistent wrapper-owned native storage. +- [x] For hidden output, allocate the result wrapper before the native call and + pass its native address at the declared native slot. On failure, destroy it + before returning the Python error. +- [x] For a function result, move or copy the returned native value into + persistent wrapper-owned storage before the native temporary expires. Never + retain an address into a bridge local. +- [x] Preserve result order and mixed-result aggregation through the existing + `ResultPlan` and lifecycle sequence; do not special-case a derived result in + function/module orchestration. +- [x] Reuse the same result type object and destructor for direct results, + hidden outputs, and edited `Returns[...]` projections. +- [x] Add reduced legacy/direct compiled parity over the existing + `make_point` cases in `test_native_call_examples.py`, + `test_output_arguments.py`, and `test_derived_type_boundaries.py`, inspecting + result storage, slot order, allocation failure, and cleanup artifacts. +- [x] Promote only those reduced generation units after both source and + generated-`.pyi` routes return the correct opaque wrapper and finalization is + proved. Field-based assertions remain on Phase 8F/H until their typed member + operations are complete. + +### Phase 8E — Required, Optional, In-Place, And Caller-Supplied Outputs + +- [x] Accept only the exact completed wrapper type for a concrete derived + argument. Subclass acceptance belongs to completed Phase 9 polymorphic + policy, not normal Python `isinstance` convenience. +- [x] Extract the opaque native address in the binding and pass it through the + single planned role. The bridge associates the matching typed native pointer + and calls the native procedure without copying for ordinary reference + arguments. +- [x] Preserve the same Python wrapper identity for visible `inout` and + caller-supplied `out` arguments. Return it only when the edited projection + requests that sole result; otherwise return `None`. Keep a mixed direct or + hidden result plus visible derived writeback on an exact policy blocker until + general mixed result/writeback aggregation is completed; do not let the + direct route select it and then drop the wrapper identity. +- [x] Represent optional omission and explicit `None` as native absence. A + present wrapper follows the same typed handoff as a required input; no empty + wrapper or call-local default object may be fabricated. +- [x] Keep native slot order independent of normalized Python argument order + and preserve user edits to argument visibility and projection. +- [x] Keep an immutable visible derived replacement on its existing exact + blocker because no passing legacy contract defines its native copy and + finalization semantics. Existing hidden/direct derived outputs use the owned + result path completed in Phase 8D; do not mutate an immutable input or invent + a generic object copy merely to remove the blocker. +- [x] Add focused type-error, optional-presence, wrong-wrapper-class, + in-place-identity, caller-supplied-output, projection, and cleanup tests. +- [x] Add reduced compiled parity that creates a `point` through the Phase 8D + result path, passes it to `point_sum`, mutates it through `move_point`, and + observes the new value through another native call without requiring a + constructor; the follow-on Phase 8F evidence also observes public fields. + +### Phase 8F — Module Objects, Components, And Field Owners + +- [x] Plan every eligible plain rank-zero derived module variable as a + native-owned live module proxy with rejected replacement; plan every + supported `Aliased` equivalent as a native-owned direct-address borrowed + wrapper. Both retain the module and have no destroy action. +- [x] Preserve `Aliased` in generated semantic `.pyi` only when supplied by the + native/source contract. Prove its module-proxy-versus-direct-address lowering + meaning while separately proving that both are live and that it does not + affect any contained native handle's view-only extraction. +- [x] Repeated `Aliased` module reads may create separate Python wrappers, but + every wrapper must refer to the same native object and never claim ownership; + repeated plain reads may create separate proxies, but every proxy must + delegate to the same current native module object. +- [x] Plan a nested derived component as a borrowed wrapper whose retained + owner is the containing wrapper. Releasing the parent name must not destroy + the parent while a child wrapper remains live. +- [x] Ensure a borrowed child never invokes its own native finalizer; releasing + the final child/owner reference triggers the containing owned instance's + destruction exactly once. +- [x] Reuse the Phase 7 `NativeArrayHandlePlan` for allocatable/pointer fields, + changing only origin=`derived_field`, owner retention=`parent_wrapper`, and + the completed field operation roles. Do not create a derived-only handle. +- [x] Plan scalar, string, ordinary-array, nested-derived, and native-handle + field getter/setter handoffs beneath the owning type for both address-backed + and module-backed objects. Use typed bridge procedures rather than C layout + offsets. Phase 8 emits both the typed low-level operations and public property + descriptors, including setter exposure completed by semantic policy. +- [x] Traverse nested value components by finite member paths and type identity. + Memoize recursive type definitions; recursive pointer/allocatable edges use + their completed association and owner policy instead of unbounded flattening. +- [x] Preserve pointer-field target ownership and stale-view rules from + completed pointer policy; parent retention does not make the parent own an + external pointer target. +- [x] Add direct plan/backend lifetime tests, then reduced compiled evidence + for the distinct plain proxy and `Aliased` direct-address origins, plus the + borrowed-finalizer, allocatable-field, and pointer-field fixtures, without + promoting constructor or method surfaces that remain Phase 9. + +### Phase 8G — Exact Native `value` Copies And Opaque Layout + +- [x] Preserve `bind(C)`/`sequence`/ordinary derived-type facts and native + `value` metadata through generated `Annotated[T, ByValue]`, post-IR policy, + and the derived handoff plan. +- [x] For every supported exact rank-zero monomorphic native `value` argument, + keep Python on the opaque wrapper contract. The Fortran bridge imports the + exact native type, reads the typed pointee, and performs the typed call. The + binding and C boundary never cast, lay out, or byte-copy the aggregate. +- [x] Remove the obsolete requirement that the native type itself be + interoperable. Ordinary, `sequence`, and `bind(C)` exact derived types use + the same Fortran-owned typed-value action; polymorphic or unresolved native + types remain exact blockers for type-identity reasons, not layout guesses. +- [x] Keep ordinary reference arguments and all component access on generated + bridge helpers even when a type is `bind(C)`; interoperability does not turn + fields into a public binary-layout promise. +- [x] Replace the obsolete unsupported-aggregate-layout assertions with + policy, plan, artifact, and compiled tests for ordinary, `sequence`, and + `bind(C)` exact typed value calls. Field-property assertions are Phase 8 + evidence; retain only constructor-dependent assertions in + `test_derived_layout.py` for Phase 9 production promotion. + +### Phase 8H — Direct-Address And Module-Proxy Object Access + +This sub-lane supplies the distinct lowering mechanisms for the two completed +module-object origins in Phase 8A/8F: direct address acquisition for an +`Aliased` live borrow, and typed live member access for a plain module proxy. + +- [x] Add one typed module-object access facet beneath `ModuleVariablePlan`. + Record `DIRECT_ADDRESS` or `MODULE_PROXY`, the native object type, member-path + operations, owner/release behavior, and failure behavior. Do not encode a + backend method name. +- [x] Use the direct path only when completed source/semantic facts make the + native address legal. The bridge exposes the opaque address mechanically; + the binding constructs the borrowed wrapper and retains its module owner. +- [x] For a plain module object, use typed per-field bridge getters/setters and + operations selected by the completed member graph. The binding constructs a + module-retaining proxy with no native destroy action; every read observes + current module state and every permitted write updates it. +- [x] Keep the initial direct-address and module-proxy paths rank-zero, + nonallocatable, nonpointer, noncoindexed, and nonpolymorphic; the explicit + descriptor-backed correction below adds only its named storage origins and + call actions. Record exact blockers for + unsupported type parameters, dynamic types, unresolved recursive pointer + ownership, or any member without a complete live operation. Do not switch + mechanisms as a fallback. +- [x] Validate direct-address roles or module-proxy member-operation coverage, + exported wrapper type identity, owner/release behavior, and + replacement rejection before either backend emits source. +- [x] Prove the `Aliased` address/lifetime premise and plain proxy live + read/write behavior in focused compiled source/generated-`.pyi` tests. +- [x] Remove the `Snapshot` contract name, metadata, recursive copy policy, + generated helper classes, documentation, and snapshot-only fixtures. Do not + retain a compatibility parser, printer, alias, or backend fallback. +- [x] Preserve ordinary result materialization, explicit constant-value + materialization, scalar descriptor value copies, ordinary array copy + results, and any unrelated active transfer action. Their copy semantics are + separate from removed whole-object snapshot behavior. +- [x] Replace the former plain-module snapshot fixture with source/generated- + `.pyi` parity and runtime evidence for live scalar, string, ordinary-array, + allocatable/pointer-handle, and nested-derived member paths, including + recursive-edge blockers and parent/module retention. + +#### Phase 8H Contract Correction — Complete Scalar-Derived Call Matrix + +This correction replaces every earlier isolated module-allocatable, stable +pointer-target, direct-address-only, and interoperable-value proposal with one +complete compatibility matrix. It covers exact rank-zero, monomorphic +`type(item)` objects. Phase 9 still owns `class(item)`, inheritance, and dynamic +dispatch; arrays of derived values remain outside this matrix. + +The actual declaration and its runtime origin are independent axes. The five +actual declaration forms are ordinary, `TARGET`, `ALLOCATABLE`, +`ALLOCATABLE,TARGET`, and `POINTER`; each can be module-owned or represented by +wrapper-owned storage where such storage is meaningful. The six native dummy +forms are: + +| Key | Exact native dummy | +| --- | --- | +| `O` | `type(item) :: arg` | +| `T` | `type(item), target :: arg` | +| `A` | `type(item), allocatable :: arg` | +| `AT` | `type(item), allocatable, target :: arg` | +| `P` | `type(item), pointer :: arg` | +| `V` | `type(item), value :: arg` | + +`OPTIONAL`, rank, qualified type identity, and `INTENT` remain separate facts. +For the `P` column, a nonpointer actual is legal only for an explicitly +`INTENT(IN)` pointer dummy. An absent `INTENT` is reassociable, not read-only. +If x2py lacks the intent but the compiler has the authoritative imported module +interface, select a compiler-validated target adapter; if neither has an +authoritative interface, report an interface error rather than fabricating a +pointer actual. + +Use these completed action names. Parenthesized state requirements are runtime +preconditions, not alternative fallback actions: + +| Action | Meaning | +| --- | --- | +| `DIRECT_REFERENCE` | wrapper-owned or direct module address; reconstruct the exact typed object and pass it by reference | +| `SCOPED_REFERENCE` | originating module synchronously invokes a generic address consumer; the native call completes before the temporary target scope returns | +| `HOLDER_REFERENCE` | reconstruct a persistent typed holder and pass its component directly | +| `MODULE_ADDRESS` | originating module returns `C_LOC` for an explicit durable target | +| `ALLOCATABLE_HOLDER` | pass a persistent wrapper-owned allocatable holder component directly, including unallocated state | +| `MODULE_ALLOCATABLE_TRANSACTION` | move between the module variable and a bridge-local shared typed transaction holder through interoperable holder-address operations | +| `POINTEE_REFERENCE` | pass the current target of a pointer holder or module pointer to a nonpointer dummy | +| `POINTER_HOLDER` | pass a persistent wrapper-owned pointer holder component directly so association writeback updates the same holder | +| `MODULE_POINTER_TRANSACTION` | initialize one bridge-local typed pointer holder from the current target and restore its final association through an interoperable holder-address operation | +| `POINTER_INPUT_ADAPTER` | expose a nonpointer actual as a target only for a pointer dummy proved or compiler-validated as `INTENT(IN)` | +| `TYPED_VALUE_COPY` | the exact Fortran bridge passes the typed object into the native `VALUE` slot; C never copies aggregate bytes | +| `INCOMPATIBLE` | language-level storage mismatch; raise the specified `TypeError` and never enter native code | + +`[allocated]` means an allocated value is required. `[associated]` means an +associated pointer target is required. `A`, `AT`, and `P` descriptor calls +accept unallocated or disassociated state where the table does not carry one of +those preconditions. + +| Actual declaration | Origin | `O` | `T` | `A` | `AT` | `P` | `V` | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `type(item) :: var` | non-module | `DIRECT_REFERENCE` | `DIRECT_REFERENCE` with call-scoped target | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | `TYPED_VALUE_COPY` from direct reference | +| `type(item) :: var` | module proxy | `SCOPED_REFERENCE` | `SCOPED_REFERENCE` with call-scoped target | `INCOMPATIBLE` | `INCOMPATIBLE` | scoped `POINTER_INPUT_ADAPTER` | scoped `TYPED_VALUE_COPY` | +| `type(item), target :: var` | non-module | `DIRECT_REFERENCE` | `DIRECT_REFERENCE` with owner target lifetime | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | direct `TYPED_VALUE_COPY` | +| `type(item), target :: var` | module | `MODULE_ADDRESS` | `MODULE_ADDRESS` with module target lifetime | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_INPUT_ADAPTER` | module-address `TYPED_VALUE_COPY` | +| `type(item), allocatable :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | +| `type(item), allocatable :: var` | module | `SCOPED_REFERENCE [allocated]` | `SCOPED_REFERENCE [allocated]` with call-scoped target | `MODULE_ALLOCATABLE_TRANSACTION` | `MODULE_ALLOCATABLE_TRANSACTION` with call target | scoped `POINTER_INPUT_ADAPTER [allocated]` | scoped `TYPED_VALUE_COPY [allocated]` | +| `type(item), allocatable, target :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | +| `type(item), allocatable, target :: var` | module | `MODULE_ADDRESS [allocated]` | `MODULE_ADDRESS [allocated]` with module target lifetime | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `POINTER_INPUT_ADAPTER [allocated]` | module-address `TYPED_VALUE_COPY [allocated]` | +| `type(item), pointer :: var` | non-module holder | `POINTEE_REFERENCE [associated]` | `POINTEE_REFERENCE [associated]` with retained target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_HOLDER` | pointee `TYPED_VALUE_COPY [associated]` | +| `type(item), pointer :: var` | module | module `POINTEE_REFERENCE [associated]` | module `POINTEE_REFERENCE [associated]` with native target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `MODULE_POINTER_TRANSACTION` | module-pointee `TYPED_VALUE_COPY [associated]` | + +An `Aliased` ordinary module object follows `MODULE_ADDRESS` instead of +`SCOPED_REFERENCE`, but its original target-lifetime fact still controls whether +a native pointer may outlive the call. This does not change `Aliased` array-view +semantics. + +The matrix is exhaustive for this Phase 8 scope. Every cell becomes either one +completed action or one deliberate language-level error before lowering. No +backend may infer a different action from datatype, `intent`, module shape, +address presence, or local memory checks. + +##### Shared Holder And Callback ABI + +Define these support types once per qualified native derived type and import +the same definitions in every producer, origin operation, and consumer: + +```fortran +type :: item_allocatable_holder + type(item), allocatable :: value +end type + +type :: item_pointer_holder + type(item), pointer :: value => null() +end type +``` -Scope: opaque derived-type wrappers, borrowed views, wrapper-owned instances, -and snapshot copies. +A persistent wrapper-owned holder is allocated through a Fortran pointer and +its opaque holder address is stored by the Python wrapper. Its nonpointer +allocatable component is a targetable subobject of the persistent holder +target, so the same carrier supports both `A` and `AT`; do not invent a second +allocatable-target holder. + +Module allocation and pointer transactions use bridge-local holder objects +declared `TARGET`. The module-specific operations are interoperable +`BIND(C)` procedures taking only `type(C_PTR), value :: holder_address` plus +interoperable status/context values. Each operation reconstructs the exact +shared holder with `C_F_POINTER` and performs `MOVE_ALLOC` or pointer assignment +entirely in Fortran. The binding transports a typed function pointer and an +opaque holder address; no allocatable or pointer descriptor crosses C. + +The old proposal to pass `type(item), allocatable` or `type(item), pointer` +directly through a runtime C callback is removed as noninteroperable. The old +proposal to avoid a transaction holder for module allocation/pointer restore is +also removed. A bridge-local transaction holder is the portable carrier; it is +not a persistent replacement for the originating module variable. + +For a module allocatable transaction, the bridge performs the equivalent of: + +```fortran +type(item_allocatable_holder), target :: transaction + +status = move_out(c_loc(transaction)) +if (status == X2PY_STATUS_OK) then + call native_procedure(transaction%value) + restore_status = move_back(c_loc(transaction)) +end if +``` -- [ ] Before implementation, expand this phase under the mandatory expansion - gate, separating origin, owned/borrowed/aliased/snapshot lifetime, +`move_out` executes `move_alloc(module_value, transaction%value)` and +`move_back` executes `move_alloc(transaction%value, module_value)`. A successful +move-out makes the module variable unavailable until restoration. When the +module actual has `TARGET`, both destinations preserve pointer association; +when it lacks `TARGET`, aliases created through a temporary target have only +call lifetime. + +For a module pointer transaction, the bridge initializes +`transaction%value` from the current `C_LOC`/`C_NULL_PTR`, passes that component +to the native pointer dummy, and invokes `restore_pointer(c_loc(transaction))`. +The origin reconstructs the pointer holder and executes +`module_pointer => transaction%value`. The final nullification, +reassociation, allocation, or deallocation is therefore visible in the module +pointer. + +Operation tables use typed C function-pointer fields; do not round-trip a +function pointer through `void *`. The proxy retains its originating extension +until every active scoped call or transaction has unwound. + +##### Pointer Target Ownership + +A pointer holder owns the holder and association variable, not its target. +Default scalar-derived pointer target ownership is native: holder destruction +nullifies the component and deallocates only the holder. It must never +deallocate an unowned target. When final association matches a known module, +parent, or wrapper-owned target, retain that owner in completed policy; an +otherwise durable native target retains the originating extension and remains +the native program's release responsibility. Native code that returns a pointer +to an expired local target violates the contract rather than creating an x2py +fallback. + +This completed owner/release rule removes the old wrapper-owned pointer-result +blocker. Reassociation is supported, but it never silently transfers target +ownership to Python. Public documentation must warn that a native pointer saved +through a wrapper-owned target remains valid only while the wrapper and target +allocation remain alive. + +##### Multiple Scalar-Derived Arguments + +Do not generate `2**N` native call branches. Build one call context with one +slot per native argument and an ordered acquisition program: + +1. validate every Python wrapper, exact qualified type, storage capability, + allocation/association precondition, optional presence, and pointer-target + owner before entering any native origin operation; +2. retain all Python/module owners and acquire module transaction guards in a + deterministic total order; +3. deduplicate repeated actual identities so one module allocation or pointer + is checked out once and its holder/address can feed multiple native slots; +4. move out module allocatables in deterministic order, rolling back already + moved values in reverse order if a later acquisition fails; +5. initialize module pointer transaction holders; +6. enter all `SCOPED_REFERENCE` producers as a nested continuation chain, + storing each address in the context; and +7. invoke the native procedure exactly once after every slot is ready, then + unwind scoped producers, pointer restorations, allocation restorations, + guards, and retained owners in reverse order. + +If the same actual appears in multiple slots and any corresponding dummy may +define it while another slot references or defines it, reject the call before +checkout unless completed `INTENT` facts prove the aliasing legal. Read-only +duplicates share one acquisition. Never move the same module allocatable twice +or restore the same module pointer through independent locals. + +The generic scoped-address consumer ABI remains +`consumer(object_address, context) -> status`. The context carries all earlier +addresses, holders, ordinary arguments, result slots, and the first error. A +consumer never retains `object_address`; multiple module variables are handled +by nesting producers, not by generating one origin-module cross product per +native procedure. + +##### Error And Cleanup Contract + +Use one status protocol across scoped consumers and module transaction +operations. Do not raise a Python exception, `longjmp`, or unwind C++ through a +Fortran frame. Record status and any Python exception data in the call context, +return normally through every producer, complete cleanup, and only then raise +in the binding. + +- wrong qualified wrapper type, an incompatible matrix cell, or a known + reassociable pointer dummy receiving nonpointer storage raises `TypeError` + before native entry; +- a required ordinary/target/value/pointer-input actual whose allocatable is + unallocated or pointer is disassociated raises `ValueError` before native + entry; +- `A`, `AT`, and `P` descriptor calls preserve valid unallocated or + disassociated state and do not reinterpret it as optional omission; +- only an omitted Python argument or explicit `None` for an optional contract + selects native absence; a present empty handle never becomes omitted by + accident; +- an active recursive/concurrent transaction raises `RuntimeError` before the + affected origin changes state; +- every successful move-out has exactly one attempted move-back on every + normally returning path, and every native module-pointer call has exactly one + attempted association restore; +- cleanup continues in reverse order after the first restoration failure so + independent origins are not stranded; the first failure is reported with + later cleanup failures attached as context; +- a failed restoration leaves its origin guard poisoned instead of advertising + a usable proxy, and raises `RuntimeError` after all other cleanup attempts; +- conversion, result allocation, and Python-object creation that can fail are + completed before checkout where possible; failures after native return still + restore every transaction before propagating; and +- process termination, `ERROR STOP`, signals, or invalid native pointers are + not recoverable wrapper exceptions. The documentation must state that this + cleanup guarantee covers paths that return through the generated ABI. + +The per-origin guard must be thread-safe, or the binding must prove that the +GIL remains held for the complete transaction and that no callback re-entry is +possible. An unsynchronized Fortran `logical` is not a sufficient concurrency +guard. Internal synchronous address consumers are Phase 8 bridge machinery; +they do not expose the public callback semantics deferred to Phase 10. + +##### Implementation And Proof Checklist + +- [x] Preserve actual declaration attributes, module/non-module origin, + `TARGET` lifetime, allocatable/pointer state, exact type identity, and + pointer-dummy `INTENT` authority through parsing, semantic IR, and edited or + generated `.pyi` round trips. +- [x] Replace the former category/action-only contract with completed facets + capable of representing all six dummy forms and every action in the matrix. + `DerivedDummyCategory` remains the completed declared-form label and + `DerivedCallAction` remains the completed selected-action label; neither is + allowed to stand in for the lifetime, access, failure, cleanup, target-owner, + or release facets. The complete record includes `ALLOCATABLE,TARGET`, typed + value, target lifetime, pointer-input + validation, transaction cleanup, and target owner/release. Remove + `RUNTIME_POINTER_TARGET`, the module-allocatable incompatibility, and all old + fallback/rejection actions they made obsolete. +- [x] Complete every matrix decision in post-IR policy before `ir2ast.py`. + Binding and bridge generation only dispatch named actions; neither backend + inspects datatype, `intent`, module shape, address presence, or allocation + state to select a different mechanism. +- [x] Generate one shared allocatable holder and pointer holder per qualified + native type, with persistent create/destroy helpers and bridge-local + transaction use. Prove source/generated-`.pyi` bundles import the identical + holder definition and reject ABI/type mismatch before reconstruction. +- [x] Generate scoped-address producer operations for plain module objects and + non-`TARGET` allocated module allocatables, direct address operations for + durable module targets, move-out/move-back holder-address operations for + module allocatables, and current-target/restore holder-address operations for + module pointers. +- [x] Implement the ordered multi-argument acquisition/unwind program, + deduplicated origin identity, legal read-only aliasing, reverse rollback, + poisoned restoration failures, and a single final native invocation. +- [x] Implement the exact Python error mapping and optional/empty-state rules + above. Add injected failures before first acquisition, after one of several + acquisitions, during scoped nesting, after native return, and during each + cleanup category. +- [x] Remove the interoperable-`bind(C)` restriction from typed derived + `VALUE` calls. The Fortran bridge must perform the exact typed call without a + C aggregate cast, byte copy, layout promise, or detached-object fallback. +- [x] Support wrapper-owned pointer results with a persistent pointer holder, + native target ownership by default, explicit known-owner retention, direct + association writeback, and holder-only destruction. Remove the old blanket + target-ownership blocker rather than retaining it as a compatibility path. +- [x] Update public and maintainer documentation to teach the five actual + declarations, six dummy forms, complete matrix, direct versus scoped + address acquisition, holder and module transactions, `INTENT(IN)` pointer + exception, target lifetime, native pointer-target ownership, multi-argument + nesting, errors, and cleanup. Examples must show more than one scalar-derived + argument and link back to one canonical explanation instead of repeating + incomplete fragments. +- [x] Add one comprehensive native fixture at + `tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90`, its + reduced source/generated contract under + `tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/`, + focused policy/plan/artifact tests in + `tests/wrapper_codegen/test_phase8_scalar_derived_actual_dummy_matrix.py`, and + compiled tests in + `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py`. + Replace the earlier proposed separate module-allocatable and + module-target/pointer fixtures; do not retain tests that assert their old + rejection paths. +- [x] Make that native fixture a complete Fortran module containing all five + module actual declarations, wrapper-owned ordinary/allocatable/pointer + producers, all six dummy forms, both pointer `INTENT(IN)` and reassociable + pointer procedures, two qualified native types with the same short name, + optional arguments, injected operation failures, and state-reset helpers. +- [x] Parameterize policy/plan tests over every matrix cell. Every legal cell + must select its one completed action; every incompatible cell must assert its + exact pre-native `TypeError`; allocated/unallocated and + associated/disassociated states must assert their exact `ValueError`, valid + descriptor call, or optional-absence behavior. +- [x] Compiled tests must exercise mixed calls containing several + scalar-derived arguments. Include at least: multiple nested scoped module + objects; two module allocatable transactions plus a module pointer + transaction; mixed direct, holder, scoped, allocatable, pointer, target, and + value slots in one native call; repeated read-only actual identity; rejected + writable duplicate identity; failure after the first of several checkouts; + reverse restoration; native deallocation/reallocation; pointer + nullification/reassociation/allocation/deallocation; and owner retention. + Phase 8 cannot close if the new compiled procedures test only one derived + argument at a time. +- [x] Run the portable ABI fixture with the supported GNU toolchain and every + available secondary compiler in the development environment. The proof must + cover scoped `C_FUNPTR` consumers, `C_PTR` transaction holders, + `C_F_PROCPOINTER`, holder targetability, target-preserving `MOVE_ALLOC`, and + the accepted-`INTENT(IN)`/rejected-reassociable pointer distinction. + +### Phase 8I — Production Routing, Regression, And Completion + +- [x] Add separate support-report lanes for derived inputs, optional derived + inputs, in-place derived arguments, wrapper-owned derived results, plain + module proxies, `Aliased` borrowed module objects, borrowed field owners, + and the exact typed-value slice. +- [x] Replace the isolated scalar-derived descriptor routes with one + dependency-closed actual/dummy-matrix lane only after every unchecked Phase + 8H row passes. It must cover direct and scoped references, target adapters, + allocatable and pointer holders, module allocation and association + transactions, exact typed values, and multi-argument acquisition/unwind. + No old call-incompatible, nonreassociating-only, or interoperability-only + compatibility route may remain selectable. +- [x] Add one deliberate legacy/direct parity node for every dependency-closed + Phase 8 lane and append it to the production rollout evidence only after its + generated artifacts and runtime behavior match. +- [x] Update the migration matrix row for each reduced unit. Keep broad units + containing constructors, methods, inheritance, or callbacks on + their explicit Phase 9/10 blockers until whole-generation-unit support is + complete. +- [x] Treat source and generated-`.pyi` default field constructors as Phase 9 + class-surface blockers. Do not select the Phase 8 route merely because the + generated constructor was consumed into origin metadata rather than retained + as a semantic method; reduced opaque Phase 8 contracts must explicitly + suppress construction. +- [x] Prove an eligible opaque-derived generation unit selects the production + wrapper-plan route and no longer invokes `semantic_ir_to_codegen_ast()`. +- [x] Keep direct plan edits meaningful across both backends and preserve the + global no-fallback rule when a derived type, owner, release, or field action + is incomplete. +- [x] In every relevant planner, validator, binding generator, and bridge + generator, keep scalar, string, ordinary-array/native-handle, and + derived-type lowering methods in consistent groups with one short comment + above each group. Preserve typed object-kind/action matching; grouping must + not introduce datatype inference or a second dispatcher. +- [x] Run focused parser/printer, ownership/policy/readiness, plan/validation, + binding/bridge/printer, and runtime tests; relevant source/generated-`.pyi` + wrapper parity; and regressions for scalar, string, array, and Phase 7 handle + lanes. +- [x] Run the wrapper suite excluding the deferred LAPACK coverage, the wrapper + codegen complexity checker, documentation checks, whitespace check, and the + required static-analysis suite before closing implementation. +- [x] Run the comprehensive Phase 8 scalar-derived actual/dummy matrix policy, + artifact, and multi-argument compiled tests; all retained holder and Phase + 7/8 regressions; the wrapper suite excluding LAPACK; documentation and + whitespace checks; the wrapper complexity checker; and the required static + suite after the replacement route is implemented. +- [x] Close Phase 8 only when every supported rank-zero non-polymorphic derived + input/result/module transfer is direct, both plain module-proxy and `Aliased` + address-backed module-object paths are direct, live member operations and + recursive-edge policy are validated before emission, and + every remaining class-surface/callback/derived-array case has an exact Phase + 9/10 or unsupported-policy blocker. + +### Phase 8 Implementation Evidence + +- Post-IR origin, identity, handoff, ownership, field, lifecycle, and exact + blocker evidence lives in + `tests/wrapper_codegen/test_phase8_derived_types.py`, with supporting parser, + printer, source-conversion, ownership, and readiness suites named in + `tests/wrapper/CHECKLIST_COVERAGE.md`. +- Public-field validation is split into named completed-policy, descriptor, + typed object-kind, and setter checks so no single semantic-policy routine + becomes a second backend-style dispatcher. +- Compiled legacy/source and direct-plan evidence lives in + `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py`. It covers + required, optional, in-place, caller-supplied output, ordinary and `bind(C)` + typed native `value`, direct/hidden owned result, module-proxy, + direct-address module object, constant value, field, owner-retention, + allocation/cleanup artifact, and exactly-once finalization behavior. +- The former isolated scalar-derived descriptor evidence in + `tests/wrapper_codegen/test_phase8_scalar_derived_descriptors.py`, + `tests/wrapper/fortran/derived_types/test_scalar_derived_descriptor_plan.py`, + and `tests/data/fortran/wrapper/fscalar_derived_descriptors_f90.f90` is + superseded by the comprehensive policy/artifact and compiled matrix files + named in Phase 8H. They cover all 60 declaration/dummy cells, empty states, + qualified same-short-name identities, `sequence` typed values, holder and + module transactions, multi-origin unwind, pointer target ownership, injected + cleanup failures, and the exact retained incompatibilities. No obsolete + module-allocatable rejection, stable-pointer-only, or wrapper-pointer-result + blocker remains as negative compatibility coverage. +- `x2py/pipeline/build.py` registers the dependency-closed Phase 8 support + lanes and their passing production evidence. The automatic-route test + replaces `semantic_ir_to_codegen_ast()` with a failure sentinel and proves an + eligible opaque-derived unit never invokes it. +- Constructors, methods, properties beyond the completed field descriptors, + inheritance, and polymorphic class orchestration remain Phase 9. Public + callbacks remain Phase 10. Arrays of derived values, non-scalar holder member + operations, recursive value edges without completed descriptor policy, + unresolved imported types without an exact runtime definition, immutable visible + derived replacement, and mixed native result plus visible-writeback envelopes + carry exact unsupported-policy blockers instead of selecting a fallback. + Internal synchronous scoped-address consumers and module transactions are + Phase 8 implementation machinery, not deferred public callbacks. + +Historical Phase 8 closure evidence before the module-allocatable and +module-pointer restore redesigns (2026-07-15): all 39 focused Phase 8 +plan/compiled tests, the 711-test +cross-stage regression batch, all 79 runtime-handle tests, 1,133 documentation +and layout tests, and all 329 wrapper tests outside the deferred full +BLAS/LAPACK file passed. The wrapper complexity checker, Ruff lint/format, +Bandit, Vulture, whitespace, and explicit-`origin/main` Radon policy passed; +the advisory full Radon complexity and maintainability reports were also +produced. The required `--base-ref auto` Radon invocation could not resolve +CI-only base-SHA variables locally, and the explicit-base rerun passed. No +LAPACK test was run locally. This evidence does not close the reopened Phase 8H +rows. + +Final Phase 8H/I closure evidence (2026-07-15): the focused Phase 8 plus route- +ledger batch passed 180 tests; the affected cross-stage semantic, lowering, +runtime-handle, planner, and backend batch passed 611 tests; and the complete +wrapper suite outside the deferred combined BLAS/LAPACK file passed 445 tests. +Documentation checks passed 1,123 tests and whitespace validation passed. The +GNU toolchain compiled and ran the complete matrix suite; Intel `ifx` 2026.1.0 +compiled, linked, and ran the same generated ABI for `sequence` typed values, +mixed six-form input, module allocatable/pointer transactions, target-preserving +`MOVE_ALLOC`, and the accepted-input/rejected-reassociable pointer distinction. +The wrapper complexity checker, Ruff lint/format, Bandit, Vulture, explicit- +`origin/main` Radon policy, and advisory Radon complexity/maintainability runs +passed. The CI-only `--base-ref auto` Radon lookup was unavailable locally, so +the required explicit-base rerun was used. No LAPACK test was run locally. + +### Phase 8 Expansion Gate + +- [x] Inventory the live semantic contract, post-IR ownership policy, active + snapshot paths to remove, legacy binding/bridge paths, plan-route blockers, + public docs, checked `.pyi` fixtures, and real wrapper tests. +- [x] Separate origin, owned/borrowed lifetime, module address acquisition, input/result/field/module-state use, destruction, owner retention, and - recursive member cases found in the live contract. - -- [ ] Define derived-type handoff specs for wrapper address, owned instance, - borrowed instance, and snapshot copy. -- [ ] Represent destruction/release responsibility in the result/writeback plan. -- [ ] Add binding and bridge actions for derived-type input, result, field - access, and snapshot creation. -- [ ] Represent scalar and later non-scalar field getter/setter behavior only - after the owning derived wrapper and class lifecycle are available. -- [ ] Validate owner-retention and release expectations before emission. - -## Phase 9 — Classes, Constructors, Properties, And Methods - -Scope: generated Python classes, keyword constructors, explicit constructor -bindings, properties, methods, static methods, overloads, and direct -`@bind(func_name)` constructor cases. - -- [ ] Before implementation, expand this phase under the mandatory expansion - gate. Inventory class creation and destruction, constructor categories, - instance/static/type-bound methods, properties, overloads, inheritance or - type relationships, decorator effects, and module initialization needs - before defining the sub-lanes. - -- [ ] Represent class layout, constructor candidates, explicit constructor - bindings, default constructor behavior, and property/method plans. -- [ ] Keep `.pyi` constructor text, runtime constructor behavior, and bridge - calls aligned through the same class plan. -- [ ] Validate that direct constructor bindings are not confused with overload - dispatch. -- [ ] Validate property setter/getter exposure against bridge handoffs. + recursive member-path access into dependency-ordered Phase 8A-I sub-lanes. +- [x] Record the strict Phase 8/9/10 boundaries and identify reduced existing + native units that can prove opaque transfers without first migrating public + constructors, methods, inheritance, or callbacks. Public field descriptors + are part of Phase 8. +- [x] Begin Phase 8 implementation only from Phase 8A and keep every later + sub-lane blocked on its declared dependencies. + +## Phase 9 — Classes, Constructors, And Methods + +Expansion status: complete. Implementation status: complete. The direct class +path is covered by policy, plan-edit, artifact, compiled runtime, production +routing, and broad non-LAPACK wrapper-suite evidence below. + +Scope: generated Python class objects, namespace registration, default and +keyword constructors, explicit constructor bindings, constructor overloads, +instance and static methods, type-bound dispatch, method overloads, finalizer +attachment, inheritance, and the first supported scalar polymorphic input +dispatch. Phase 9 assembles those public class surfaces on the opaque storage, +field descriptors, handoffs, and lifetime rules completed in Phase 8. + +### Phase 9 Boundary And Explicit Non-Scope + +Phase 9 may compose completed Phase 8 records but must not revisit them. +Constructor and method policy may select how an instance is created or passed; +it may not change object origin, storage kind, field access, owner retention, +release, nullability, native setter assignment, or destruction. A class plan +references the namespace-owned `DerivedTypePlan` and its field plans rather +than copying or rendering them. + +The following surfaces are in Phase 9: + +- one generated Python type object for each public supported semantic class, + with stable native identity and explicit Python base identity; +- an explicitly present or deliberately absent public constructor surface; +- generated default/keyword field initialization for eligible public scalar + fields, including omitted-keyword preservation of native defaults; +- direct `@bind("native_name")` constructors and explicit constructor overload + candidates linked to concrete native procedures; +- passed-object type-bound instance methods, non-type-bound methods attached to + the class by the semantic contract, and supported `@staticmethod` methods; +- class-owned overload sets with exact candidate signatures and deterministic + runtime selection; +- owned-instance finalization through the Phase 8 destroy/release path and + borrowed-instance non-destruction; +- Python inheritance for supported Fortran extension types; and +- scalar, input-only polymorphic calls whose accepted runtime class set and + concrete native dispatch targets are fully enumerated before lowering. + +The following remain outside Phase 9: + +- callbacks, adapters, trampolines, and callable lifetime; these remain Phase + 10 even when a callback argument/result is a derived object; +- module-level generic/operator migration units that do not require a class + surface; those remain in Phase 11, although they may reuse the same overload + candidate and runtime-match vocabulary; +- arrays of derived or polymorphic values, elemental class dispatch, and + partial construction/destruction of array elements; +- polymorphic results, mutable polymorphic dummies, allocatable/pointer + polymorphic scalars, unlimited polymorphism, abstract instantiation, + deferred-binding execution, and runtime extension types not enumerated in + the semantic module; +- any unresolved Phase 8 storage blocker merely because a constructor or + method happens to use that type; Phase 9 reuses the completed allocatable and + pointer holders and must not invent a second storage path; +- generic constructor selection whose candidates are indistinguishable at the + Python boundary; and +- compatibility aliases, synthesized legacy entrypoints, string-built backend + method names, or a fallback from an incomplete class plan to legacy class + lowering. + +### Phase 9 Existing Oracle And Inventory + +The legacy route plus existing source/generated-`.pyi` runtime assertions are +the behavioral oracle. Capture complete binding, bridge, header, and runtime +evidence before each reduced direct-plan slice. Correct unsafe behavior only +when the documented contract says so; do not preserve legacy architecture. + +| Existing unit | Phase 9 behavior to preserve | Required reduced slice | +| --- | --- | --- | +| `derived_types/test_constructors_and_finalizers.py::test_fortran_default_constructor_keywords_and_finalization[*]` | default construction, keyword-only scalar fields, native defaults, invalid-call cleanup, and exactly-once finalization | default/keyword constructor plus owned destroy path | +| `derived_types/test_derived_type_methods.py::test_modern_fortran_derived_type_exposes_class_and_type_bound_methods[*]` | instance methods, explicit binding names, scalar arguments/results, class static factory, and Phase 7 handle fields | split `vector` methods from `vector_store` handle methods and static factory | +| `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component[*]` | borrowed child retains parent; only the owned parent finalizes | class assembly over the completed Phase 8 borrowed-field owner path | +| `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy[*]` | default class creation and methods coexist with opaque field access and typed native value copy | class surface only; Phase 8 retains layout and handoff ownership | +| `derived_types/test_inheritance.py::test_fortran_extension_types_generate_python_inheritance[*]` | Python subclass relationships, inherited field/method access, overridden methods, unbound base calls, and scalar polymorphic input dispatch | base/extension class graph first, polymorphic call second | +| `tests/semantics/conversion/pyi/test_classes_and_overloads.py` | generated versus bound constructors, removed constructors, direct constructor targets, explicit overload links, type-bound root targets, and invalid metadata diagnostics | semantic-policy fixtures before planner/backend work | +| `edit_pyi_contracts/test_surface_edit_contracts.py` | edited contracts can remove constructors/methods/candidates and add explicit bindings without resurrecting source declarations | absence/export validation and source/generated/edited parity | +| `naming/test_defined_operators.py` and `naming/test_generic_interfaces.py` | exact candidate matching and Python export naming | reuse candidate-match vocabulary; broad module/generic units remain Phase 11 | + +Inventory these legacy owners without importing them into the direct package: + +- `x2py/semantics/ir2ast.py` currently interprets constructor overloads, + passed-object positions, type-bound names, polymorphic variants, and class + insertion. Each semantic decision found there must move into post-IR class + policy before direct lowering. +- `x2py/codegen/bindings/c_to_python.py` currently assembles type objects, + constructors, methods, overloads, properties, inheritance, module exports, + and finalizers. Reuse emitted behavior as the oracle, not its broad control + flow or method-name synthesis. +- `x2py/codegen/bridges/fortran_to_c.py` currently supplies typed constructor + allocation, passed-object association, method calls, overload interfaces, + and finalization helpers. Direct bridge generation must consume completed + class/method actions and reuse Phase 8 native storage helpers. +- Generated semantic `.pyi` class declarations are a public contract. A + consumed default constructor still counts as a constructor surface and must + remain a whole-unit Phase 9 route requirement. + +### Phase 9 Plan Shape And Action Vocabulary + +Extend the existing namespace plan; do not introduce a rendered-class layer or +a second function plan. + +- Add one namespace-owned `ClassSurfacePlan` (name illustrative, not + prescriptive) that references exactly one `DerivedTypePlan`, its Python + exports, optional base-class identity, constructor plan, ordered methods, + ordered overload sets, type-object slots, and module-registration action. +- Add a `ConstructorPlan` with an explicit kind: `ABSENT`, + `DEFAULT_FIELDS`, `BOUND_PROCEDURE`, or `OVERLOAD_SET`. Record allocation + action, accepted Python parameters, native target/call slots, initialized + fields, omitted-field behavior, cleanup action, and success transition. +- Reuse `FunctionPlan` for each concrete method or constructor target. Add only + a class-call facet recording method kind, passed-object position, self + storage requirement, result attachment, public descriptor flags, and the + owning class identity. +- Add an `OverloadSetPlan` containing public export, overload kind, ordered + concrete candidate references, typed runtime predicates, ambiguity result, + no-match diagnostic, and selected native target. Candidate predicates use + exact dtype/rank/derived-class facts already completed by argument plans. +- Add an `InheritancePlan` containing canonical base identity, storage + compatibility, inherited/overridden method ownership, Python base type + symbol, and module initialization dependency order. +- Add a `PolymorphicDispatchPlan` only for supported scalar input calls. It + enumerates accepted concrete class identities and a concrete `FunctionPlan` + variant for each; it must not rediscover subclasses from runtime object names. +- Keep destructor selection on the referenced Phase 8 derived handoff/release + plan. Phase 9 records only which class slot invokes that existing action and + which constructor failure edges need cleanup. + +Stable semantic action names must describe behavior, not backend function +names. At minimum distinguish: + +- class registration: `CREATE_TYPE`, `SET_BASE`, `READY_TYPE`, `EXPORT_TYPE`; +- construction: `OMIT`, `ALLOCATE_DEFAULT`, `ALLOCATE_AND_ASSIGN_FIELDS`, + `CALL_BOUND_CONSTRUCTOR`, `DISPATCH_CONSTRUCTOR`, `REJECT_CONSTRUCTION`; +- method binding: `INSTANCE`, `STATIC`, and explicit unsupported class-method + policy until a real class-method contract exists; +- passed-object handoff: `WRAPPER_ADDRESS`, `BORROWED_ADDRESS`, or the exact + completed Phase 8 storage action; +- overload selection: `MATCH_EXACT`, `SELECT_CANDIDATE`, `NO_MATCH`, + `AMBIGUOUS`; and +- construction lifecycle: `ALLOCATE`, `INITIALIZE`, `COMMIT_OWNER`, + `CLEANUP_UNCOMMITTED`, `DESTROY_OWNED`. + +### Mandatory Phase 9 Migration Algorithm + +For every dependency-closed sub-lane: + +1. Capture one passing source/generated-`.pyi` legacy unit and its complete + class, binding, bridge, header, and runtime assertions. +2. Complete class export, constructor kind, method kind, passed-object policy, + overload candidates, inheritance, polymorphic accepted set, allocation, + commit, cleanup, and destruction before `ir2ast.py`. +3. Project those facts into the existing namespace, derived-type, function, + lifecycle, and native-slot plans plus the smallest class-specific facets. +4. Validate the complete class graph and every cross-backend symbolic role + before either backend emits source. +5. Lower through small named methods selected only by typed actions. Backend + local temporaries may implement a selected action but cannot choose policy. +6. Compare generated artifacts with the oracle and record intentional + differences before compiling. +7. Add focused policy, plan-edit, validation, printer, binding, bridge, + source/generated-`.pyi`, edited-contract, and compiled runtime tests. +8. Promote production routing only after the reduced unit passes direct-plan + runtime parity and no class-surface fallback remains. + +### Phase 9A — Semantic Class-Surface Completion + +- [x] Add completed post-IR policy records for public class identity, exports, + constructor kind, method kind, passed-object position, overload ownership, + base identity, type-object registration, and construction permissions. +- [x] Preserve explicit absence: an edited `.pyi` that removes `__init__`, a + method, or an overload candidate must produce an absent plan entry and cannot + resurrect source behavior. +- [x] Move any class-surface inference still in `ir2ast.py` into policy + completion. Readiness must report the owner path and exact missing decision. +- [x] Add policy tests for generated, bound, removed, overloaded, inherited, + abstract, and invalid class surfaces before planner changes. + +### Phase 9B — Typed Class Plan And Validation + +- [x] Add the namespace-owned class, constructor, method-call, overload, and + inheritance plan facets described above, each referencing existing Phase 8 + type/field/lifetime plans rather than copying them. +- [x] Project Python/native names and export aliases once. Do not synthesize + backend method names from strings or recover native targets by scanning + emitted functions. +- [x] Validate unique class/type identity, base-before-derived order, one + constructor kind, method ownership, passed-object position, native call-slot + agreement, field-plan identity, lifecycle roles, and module export symbols. +- [x] Add direct plan-edit tests proving invalid constructor, method, base, + overload, or lifecycle references fail before emission in both backends. + +### Phase 9C — Class Creation And Module Registration + +- [x] Emit one Python type object per supported class, attach the completed + Phase 8 field descriptors, set the validated base type, ready the type, and + export every completed Python name in dependency order. +- [x] Keep opaque instance storage identical to Phase 8 wrapper storage. Class + assembly must not add C aggregate layout, component offsets, or a second + native owner field. +- [x] Attach the Phase 8 destruction action only to owning classes; borrowed + proxies and nested objects retain owners and never gain independent destroy + slots. +- [x] Add artifact and compiled reduced tests for an opaque constructible class, + an intentionally nonconstructible class, a borrowed child, and exact module + export identity. + +### Phase 9D — Default And Keyword Field Constructors + +- [x] Build constructor parameters only from fields explicitly eligible in + completed constructor policy. Preserve keyword-only behavior and native + default component initialization for omitted fields. +- [x] Allocate the Phase 8 persistent native instance first, apply validated + field assignments through existing field setter actions, then commit wrapper + ownership only after every step succeeds. +- [x] On parse, conversion, allocation, or field-assignment failure, clean up + the uncommitted native instance exactly once. Failed `tp_init` must not leak, + double-finalize, or expose a partially initialized wrapper. +- [x] Replay `fconstructors_f90` for default, partial, complete, positional, + unknown-keyword, native-default, and finalization-count assertions through + both source and generated-`.pyi` contracts. + +### Phase 9E — Explicit And Overloaded Constructors + +- [x] Represent direct `@bind("native_name")` construction as one constructor + action linked to a concrete function plan. It replaces, rather than wraps or + falls back to, the generated field constructor. +- [x] Represent constructor overloads as an explicit constructor-owned overload + set. Do not combine `@overload` and `@native_call`, and do not reinterpret a + normal method overload as `tp_init`. +- [x] Complete allocation-before-call versus native-produced-instance policy, + result attachment, owner commit, failure cleanup, and exactly-once release for + every candidate before lowering. +- [x] Reject indistinguishable candidates, missing targets, incompatible self + types, mixed constructor kinds, or ambiguous edited declarations during + policy/readiness or plan validation, never from candidate trial calls. +- [x] Add isolated semantic and compiled fixtures for direct bound construction, + two distinguishable constructor candidates, no-match, ambiguity, target + failure cleanup, and source/generated/edited-contract parity. + +### Phase 9F — Instance And Static Methods + +- [x] Lower passed-object instance methods from the completed self position and + Phase 8 handoff. Preserve native argument order when `self` is not the first + native slot. +- [x] Support explicit binding names and type-bound root-target metadata without + exporting the private concrete target as a duplicate module function. +- [x] Lower supported static methods without fabricating `self`; attach them to + the type object with their completed export and descriptor flags. +- [x] Reuse ordinary function argument/result plans for scalar, string, array, + handle, and derived transfers. A method cannot widen an unsupported ordinary + call lane. +- [x] Replay reduced `fclasses_f90` vector methods first, then `vector_store` + handle methods and static factory, with exact source/generated-`.pyi` runtime + and artifact parity. + +### Phase 9G — Class-Owned Overload Dispatch + +- [x] Complete ordered candidates and exact runtime predicates for each + class-owned overload set. Candidate selection may inspect only typed Python + argument facts named by the plan, never invoke candidates speculatively. +- [x] Reuse one overload matching vocabulary for constructors, methods, + operators, and later Phase 11 module generics while keeping their owners and + call actions distinct. +- [x] Detect indistinguishable signatures before emission and produce stable + no-match diagnostics listing the public overload and accepted signatures. +- [x] Validate native target, Python export, argument/result plans, passed-object + position, and overload kind across binding and bridge views. +- [x] Add focused method-overload tests for primitive kinds, ranks, derived + subclasses, keyword normalization, exact no-match, and ambiguity; keep broad + defined-operator/module-generic promotion in Phase 11. + +### Phase 9H — Finalization And Constructor Failure Safety + +- [x] Route normal owned-instance deallocation, constructor failure, and + native-constructor failure through the same Phase 8 destroy/release action, + guarded by an explicit uncommitted/committed lifecycle state. +- [x] Prove finalization occurs exactly once for successfully constructed + owners, once for native storage allocated before a rejected constructor call, + and never for borrowed children or native-owned module objects. +- [x] Prove child-to-parent retention survives method/property access and that + deleting the parent first delays only the parent's owning finalizer. +- [x] Replay `fconstructors_f90` and `fborrowed_finalizer_f90`, including forced + Python argument failures and repeated garbage collection. + +### Phase 9I — Inheritance And Scalar Polymorphic Input Dispatch + +- [x] Complete canonical base/extension relationships, storage compatibility, + inherited fields, inherited methods, overrides, Python base symbols, and + module initialization order before planning. +- [x] Construct base and derived wrappers with the same Phase 8 opaque storage + contract while preserving exact runtime type identity and safe unbound base + method calls on derived instances. +- [x] For each supported scalar input-only polymorphic dummy, enumerate the + accepted concrete class identities and one concrete native call variant per + identity. Reject unknown or abstract runtime classes before the native call. +- [x] Keep polymorphic results, mutable dummies, arrays, descriptor-backed + polymorphic scalars, unlimited polymorphism, and unenumerated extensions on + exact blockers; inheritance must not silently widen them. +- [x] Replay `finheritance_f90` for `issubclass`, `isinstance`, inherited field + access, override dispatch, unbound base calls, and base/circle/box + polymorphic inputs through source and generated-`.pyi` routes. + +### Phase 9J — Production Routing, Documentation, And Closure + +- [x] Add support-report lanes for class registration, default constructors, + bound constructors, constructor overloads, instance methods, static methods, + class overloads, finalizers, inheritance, and scalar polymorphic input. +- [x] Add one reduced compiled direct-plan node per dependency-closed lane, then + update its migration-matrix row only after artifact and runtime parity. +- [x] Prove eligible class units select the production wrapper-plan route and + never call `semantic_ir_to_codegen_ast()`; an unsupported class decision must + keep the whole generation unit on one exact blocker without partial fallback. +- [x] Synchronize constructor/method/inheritance user docs, semantic `.pyi` + reference, source map, feature matrix, subject README, and checklist coverage + with the implemented class contract. +- [x] Run focused policy/plan/backend tests, all affected existing class wrapper + nodes through source/generated-`.pyi` modes, the wrapper suite excluding + LAPACK, the wrapper complexity checker, documentation checks, whitespace, + and the required static-analysis suite. +- [x] Close Phase 9 only when every supported constructor/method/inheritance + unit routes directly, all Phase 8 field/storage/lifecycle decisions remain + unchanged, and every remaining callback, derived-array, polymorphic, or + ambiguous-overload case has an exact Phase 10/11 or unsupported-policy + blocker. + +Closure evidence (2026-07-16): focused semantic, lowering, routing, and direct +Phase 8-10 plan tests passed 184 tests after the final policy refactor. The +complete local wrapper suite excluding LAPACK passed 449 tests in source and +generated-contract modes. The wrapper complexity checker, Ruff lint/format, +Bandit, Vulture, explicit-`origin/main` Radon policy, and advisory Radon +complexity/maintainability commands passed. The CI-only `--base-ref auto` +Radon lookup could not resolve outside CI, so the required explicit-base run +was used. No LAPACK test was run locally. + +### Phase 9 Expansion Gate + +- [x] Inventory class creation/destruction, constructor categories, + instance/static/type-bound methods, overloads, inheritance/polymorphism, + decorator effects, module initialization, legacy owners, semantic fixtures, + and passing runtime oracles. +- [x] Define the Phase 8/9/10/11 ownership boundaries and keep all class + implementation rows unchecked. +- [x] Split implementation into dependency-ordered Phase 9A-J sub-lanes with + explicit policy, plan, validation, lowering, artifact, compiled parity, + production routing, documentation, and closure gates. ## Phase 10 — Callbacks And Trampolines -Scope: callback argument conversion, callback result conversion, adapter -procedures, C trampolines, callback context setup/cleanup, and error/abort -paths. - -- [ ] Before implementation, expand this phase under the mandatory expansion - gate. First inventory callback signature categories, context lifetime, - re-entry/GIL behavior, exception propagation, abort paths, recursion, and the - scalar/string/array/derived argument-result combinations actually supported - by the product contract. - -- [ ] Define callback handoff specs for Python callable validation, callback - context, native adapter arguments, trampoline arguments, and callback results. -- [ ] Represent callback setup and cleanup as call-scoped plan phases. -- [ ] Add binding/bridge actions for scalar, array, string, and derived callback - arguments/results only after those lanes are stable for ordinary calls. -- [ ] Validate callback result and argument handoffs across binding, bridge, - adapter, and trampoline steps before emission. +Expansion status: complete. Implementation status: complete. Immediate +callbacks are covered by focused policy/plan/artifact tests, existing compiled +runtime oracles, production routing, and broad non-LAPACK wrapper-suite +evidence below. + +Scope: immediate callback argument validation, call-scoped context lifetime, +Fortran adapter procedures, C trampolines, scalar/string/array/derived +argument and result conversion, copy-back, same-thread re-entry and GIL +handling, callback cleanup, and the documented fatal error boundary. + +### Phase 10 Boundary And Explicit Non-Scope + +Phase 10 composes ordinary call transfers completed in Phases 2-9 but does not +reinterpret them. A callback signature is Fortran-facing: it describes the +procedure interface that native Fortran calls, including argument order, +value/reference access, intent-derived copy direction, rank, shape, character +length, and result representation. Normal wrapper projection and callback +adapter projection remain distinct completed records. + +The supported callback contract is deliberately call-scoped: + +- the Python callable is validated and retained before the native call, placed + in one thread-local context stack for that callback site, and released after + the native call returns; +- nested callback-taking calls on the same entering Python thread are allowed; +- each C trampoline validates the entering thread, acquires the GIL, converts + completed adapter arguments, invokes the current Python callable, converts + or copies back results, releases the GIL, and returns to its Fortran adapter; +- scalar values use value conversion; scalar reference storage, writable + fixed-length character storage, arrays, and derived objects use the exact + borrowed/copy-in/copy-out behavior already asserted by the legacy runtime + tests; and +- a Python exception, invalid callback return, missing context, or cross-thread + invocation prints the Python error and aborts the host process. The direct + path must not fabricate a fallback result or continue native execution. + +The following remain outside Phase 10: + +- stored callbacks, callback registration/unregistration, procedure-pointer + fields, callbacks invoked after the wrapped call, optional dummy procedures, + null procedure pointers, asynchronous callbacks, and cross-thread callback + execution; +- persistent callable ownership, callback teardown during object/library + destruction, and callback use as a synchronization mechanism; +- callbacks whose signature is incomplete, assumed-rank, has a runtime-only + character length, or otherwise lacks the exact ABI facts required by the + adapter and trampoline; +- callback-specific coercion, recovery, exception-result, or argument + reordering policies not present in the public contract or legacy tests; and +- module generic/operator orchestration that merely contains a callback-taking + candidate; its callback transfer may be reusable, but public generic routing + remains Phase 11. + +### Phase 10 Existing Oracle And Inventory + +The public callback guide/reference, generated semantic `.pyi` contracts, +legacy Python lowering/codegen, and existing source/generated-`.pyi` runtime +assertions are the behavioral oracle. Capture each reduced legacy artifact and +runtime assertion before implementing its direct-plan equivalent. + +| Existing unit | Phase 10 behavior to preserve | Required reduced slice | +| --- | --- | --- | +| `callbacks/test_scalar_callbacks.py::test_immediate_scalar_dummy_procedure_calls_python_callback[*]` | scalar result/void callbacks, callable validation, balanced references, nested same-thread re-entry, held-GIL wrapper envelope, and thread-local context | first context, trampoline, scalar-value, and cleanup slice | +| `callbacks/test_scalar_callbacks.py::test_callback_exception_prints_traceback_and_aborts_host_process[*]` | callback exception, wrong result, and wrong signature print a Python error and terminate the subprocess | fatal-boundary slice after scalar success | +| `callbacks/test_array_callbacks.py::test_immediate_dummy_procedure_converts_array_arguments_and_results[*]` | read-only array view, shaped array result, output identity, and copy-back | array argument/result slice | +| `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes[*]` | scalar value, rank-zero scalar storage, fixed strings, arrays, derived values, output/inout copy-back, and one combined call envelope | cross-kind closure slice | +| `callbacks/test_derived_callbacks.py::test_immediate_dummy_procedure_converts_derived_arguments_and_results[*]` | callback-local borrowed derived input plus wrapper-owned derived result conversion | derived slice after Phase 9 construction | +| `callbacks/test_callback_generated_pyi_contracts.py` | generated callable access, shape, character-storage, and result annotations round-trip exactly | semantic-contract parity slice | +| `semantics/conversion/pyi/test_types_and_values.py` callback cases | `Callable` ABI wrappers, inferred dimension names, writable-string validation, and forbidden `Addr(...)` callback forms | policy completion before planner work | + +Inventory these legacy Python owners without importing them into +`x2py.wrapper_codegen`: + +- `x2py/semantics/ir2ast.py` currently creates callback `FunctionAddress` + records, infers fallback argument records from callable metadata, and decides + callback result variables. Every required signature and ownership fact must + instead be complete before this lowering boundary. +- `x2py/codegen/bridges/fortran_to_c.py` currently constructs the typed + Fortran adapter, pointer/value ABI arguments, copy-in/copy-out storage, + callback result reconstruction, and adapter call ordering. Its behavior is + the oracle, not a reusable direct-path dependency. +- `x2py/codegen/bindings/c_to_python.py` currently validates the callable, + pushes/pops thread-local context, emits C trampolines, acquires/releases the + GIL, converts Python arguments/results, and implements traceback-plus-abort. + Direct lowering must consume completed callback actions through small named + methods rather than copying this broad control flow. + +### Phase 10 Plan Shape And Action Vocabulary + +Extend the existing argument/function plans; do not add a second function plan +or embed a legacy AST. + +- Add one `CallbackHandoffPlan` facet to each callable argument. It records the + callable owner, call-scoped lifetime, context symbol, context stack action, + entering-thread rule, GIL rule, Fortran adapter symbol, C trampoline symbol, + ordered callback argument plans, optional result plan, and fatal-error + action. +- Add a `CallbackTransferPlan` for each callback argument/result containing the + semantic type identity, object kind, value/reference ABI, rank/shape/length + roles, access direction, Python barrier action, adapter copy-in/copy-out + action, borrowed-owner retention, and exact C ABI roles. +- Reuse ordinary scalar, string, array, and derived plan vocabulary where the + representation is identical, but keep callback-direction roles explicit. + The native callback is the caller, so normal Python-to-native argument + projection cannot be silently reused in reverse. +- Add ordered function lifecycle phases `VALIDATE_CALLBACK`, `PUSH_CONTEXT`, + `ENTER_NATIVE`, `POP_CONTEXT`, and `RELEASE_CALLBACK`. Every failure edge + before native entry unwinds acquired references; the fatal trampoline edge + never returns. +- Keep backend-local adapter locals and temporary Python views inside the + selected implementation method. They are emitted-code details, not semantic + policy. + +Stable actions must describe behavior, not generated function names. At +minimum distinguish: + +- callable/context: `VALIDATE_CALLABLE`, `RETAIN_CALLABLE`, `PUSH_CONTEXT`, + `POP_CONTEXT`, `RELEASE_CALLABLE`; +- callback ABI: `VALUE`, `REFERENCE`, `DATA_AND_SHAPE`, + `DATA_AND_LENGTH`, and `DERIVED_ADDRESS`; +- adapter transfer: `COPY_IN`, `COPY_OUT`, `COPY_IN_OUT`, `BORROW_READ_ONLY`, + and `BORROW_WRITABLE`; +- trampoline runtime: `REQUIRE_ENTERING_THREAD`, `ACQUIRE_GIL`, `CALL_PYTHON`, + `RELEASE_GIL`, and `ABORT_WITH_PYTHON_ERROR`; and +- result handling: `RETURN_SCALAR`, `RETURN_ARRAY_ADDRESS`, + `RETURN_DERIVED_ADDRESS`, `RETURN_VOID`, and `REJECT_RESULT`. + +### Mandatory Phase 10 Migration Algorithm + +For every dependency-closed sub-lane: + +1. Capture the documented behavior, one passing source/generated-`.pyi` + legacy unit, and its callback-related binding, bridge, adapter, trampoline, + and runtime assertions. +2. Complete callable validity, signature order, ABI roles, copy direction, + shape/length dependencies, result handling, context lifetime, thread/GIL + rules, cleanup, and fatal behavior before `ir2ast.py`. +3. Project those facts into the existing function/argument/lifecycle plans plus + the smallest callback-specific facets. +4. Validate the binding, bridge, adapter, and trampoline role graph before + either backend emits source. +5. Lower through typed action dispatch and small named methods. Do not trial a + callback or infer shape/access from emitted locals. +6. Compare direct artifacts and behavior with the legacy oracle; document any + safety improvement before changing observable behavior. +7. Add focused policy, editable-plan, validation, binding, bridge, printer, + source/generated-`.pyi`, subprocess-failure, and compiled runtime tests. +8. Promote production routing only after the complete callback-taking + generation unit passes direct-plan parity with no callback fallback. + +### Phase 10A — Semantic Callback Completion + +- [x] Add completed post-IR callback records for callable signature order, + argument access, object kind, value/reference ABI, shape/length roles, result + representation, call scope, context lifetime, same-thread rule, GIL rule, + cleanup, and fatal-error behavior. +- [x] Preserve generated and edited `Callable` contracts exactly. Reject an + incomplete signature, invalid writable character form, forbidden `Addr`, + optional procedure, stored/procedure-pointer lifetime, or unsupported result + with the owner path and one exact reason. +- [x] Remove callback signature/result/ownership inference from `ir2ast.py`; + lowering may only project the completed callback record. +- [x] Add policy/readiness tests for every supported access form and retained + unsupported form before planner changes. + +### Phase 10B — Typed Callback Plan And Validation + +- [x] Add callback handoff, transfer, result, context, and lifecycle facets to + the existing function plan and reference ordinary datatype plans instead of + copying them. +- [x] Project adapter/trampoline symbols and ABI roles once. Do not synthesize + backend handler names or rediscover dimension/length dependencies from + emitted variables. +- [x] Validate unique callback sites, exact argument order, role availability, + copy direction, dtype/rank/shape/length agreement, derived type identity, + result compatibility, context balance, and validate/push/pop/release order. +- [x] Add direct plan-edit tests proving invalid callback roles, unbalanced + lifecycle, or cross-backend disagreement fail before emission. + +### Phase 10C — Context, Trampoline, GIL, And Scalar Values + +- [x] Emit one thread-local stack per callback site, callable validation and + strong-reference retention before native entry, reverse-order pop/release + after return, and cleanup on every ordinary pre-entry failure. +- [x] Emit one C trampoline and typed Fortran adapter from the completed ABI; + validate the entering thread and context before Python conversion. +- [x] Acquire/release the GIL inside the trampoline and keep the outer + callback-taking wrapper on the legacy-observed held-GIL envelope. +- [x] Lower void and scalar-value arguments/results first, then replay scalar + callback success, nested re-entry, non-callable rejection, and balanced + reference-count assertions in both build modes. + +### Phase 10D — Scalar Reference And Fixed-String Storage + +- [x] Lower missing-intent/inout scalar storage as copy-in/out rank-zero NumPy + storage, output storage as copy-out only, and explicit input references as + Python scalar values according to the completed access plan. +- [x] Lower read-only fixed strings as Python `str` and writable fixed strings + as rank-zero fixed-width bytes storage with the exact length, padding, and + copy-back behavior already asserted by the legacy tests. +- [x] Reject runtime-length or immutable writable string contracts before + emission; no adapter-local inference may change the representation. +- [x] Replay the scalar-storage and string-storage cases from the combined + callback fixture through source/generated-`.pyi` routes. + +### Phase 10E — Array Arguments And Results + +- [x] Lower array callback arguments from completed dtype, rank, shape, + ordering, contiguity, alignment, and access facts. Read-only inputs expose + read-only borrowed views; writable/output arrays expose writable storage and + copy back in adapter order. +- [x] Lower fixed-shape array results through one validated returned-address + ABI and assign them into the native adapter result. Reject incomplete shape + or unsupported ownership before emission. +- [x] Preserve output-array Python identity in the outer ordinary call and do + not add a detached-copy fallback. +- [x] Replay `fcallback_array_f90` plus the combined array-storage callback and + artifact assertions in both build modes. + +### Phase 10F — Derived Arguments And Results + +- [x] Reuse the exact Phase 8/9 type identity, opaque wrapper, owner-retention, + and destroy/release actions for callback-local derived wrappers. Do not + expose aggregate layout or introduce callback-specific storage ownership. +- [x] Borrow callback input wrappers only for the callback invocation; convert + supported callback results to the completed native result storage and + release temporary wrapper ownership exactly once. +- [x] Validate exact runtime class/type identity before using a returned + derived address. Polymorphic, descriptor-backed, or unsupported derived + callback forms retain exact blockers. +- [x] Replay `fcallback_derived_f90` and the combined derived callback after the + Phase 9 constructor/class route is green. + +### Phase 10G — Fatal Errors, Re-entry, And Cleanup + +- [x] Route Python exceptions, argument-call mismatch, invalid callback result, + missing context, and cross-thread entry through one + traceback-plus-`abort()` action. Never return a fabricated value. +- [x] Prove nested same-thread callback calls use stack discipline and restore + the previous callable/context after the inner call. +- [x] Prove ordinary validation or setup failures before native entry release + every retained reference, and successful calls leave the callable reference + count unchanged. +- [x] Run fatal cases in subprocesses for both source/generated-`.pyi` builds + and assert the documented traceback/error text plus nonzero termination. + +### Phase 10H — Production Routing, Documentation, And Closure + +- [x] Add support-report lanes for callback context, scalar value/storage, + fixed strings, arrays, derived values, result conversion, same-thread + re-entry, and fatal errors. +- [x] Add one reduced compiled direct-plan node per dependency-closed lane and + update its migration-matrix row only after artifact and runtime parity. +- [x] Prove eligible callback units select the production wrapper-plan route + and never call `semantic_ir_to_codegen_ast()`; unsupported callback policy + must keep the whole generation unit on one exact blocker. +- [x] Synchronize callback guide/reference, semantic `.pyi` reference, feature + matrix, callback README, source map, and checklist coverage with the direct + implementation. +- [x] Run focused policy/plan/backend tests, every callback wrapper node in + source/generated-`.pyi` modes, the wrapper suite excluding LAPACK, the + wrapper complexity checker, documentation checks, whitespace, and the + required static-analysis suite. +- [x] Close Phase 10 only when every supported immediate callback unit routes + directly, no callback plan falls back after generation starts, and every + stored/optional/asynchronous/cross-thread or incomplete callback form has an + exact retained blocker. Stop before Phase 11. + +Closure evidence (2026-07-16): callback policy, editable-plan validation, +binding/bridge artifacts, scalar/string/array/derived conversion, nested +same-thread re-entry, reference cleanup, and subprocess fatal-boundary tests +all passed through the direct route. The same 184-test focused batch and +449-test non-LAPACK wrapper replay used for Phase 9 closure cover the complete +immediate-callback matrix. Required static checks passed with the explicit +Radon base noted above, and implementation stopped before Phase 11. + +### Phase 10 Expansion Gate + +- [x] Inventory the public callback contract, semantic `Callable` records, + legacy lowering/codegen owners, source/generated-`.pyi` runtime fixtures, + context lifetime, re-entry/GIL behavior, exception/abort behavior, and every + supported scalar/string/array/derived argument-result combination. +- [x] Define the Phase 9/10/11 boundary and retain explicit blockers for stored, + optional, asynchronous, cross-thread, incomplete-signature, and unsupported + callback forms. +- [x] Split implementation into dependency-ordered Phase 10A-H sub-lanes with + policy, typed plan, validation, lowering, compiled parity, production + routing, documentation, and closure gates. ## Phase 11 — Cross-Cutting Wrapper Suite Completion diff --git a/docs/user/guide/allocatables.md b/docs/user/guide/allocatables.md index 9aaa8321a..4360355bd 100644 --- a/docs/user/guide/allocatables.md +++ b/docs/user/guide/allocatables.md @@ -74,17 +74,18 @@ rejected for `Allocatable[T[...]]` descriptor parameters because a NumPy array does not carry a native allocatable descriptor. `h.to_numpy()` is the explicit extraction operation. It returns `None` when the -handle is unallocated. When policy proves live aliasing is safe, it returns a -borrowed NumPy view; when live aliasing is unsafe but copying is supported, it -returns a read-only detached copy. Users can call `.copy()` on any returned -NumPy array when they need independent lifetime. +handle is unallocated. Otherwise, it returns a live mutable NumPy view of the +current native allocation. It never creates an automatic detached snapshot or +copy. Users who need independent storage must explicitly call `.copy()` on the +returned array. A borrowed view is a NumPy array that points at storage Python does not own. Mutating the view mutates the owner. Deallocating or reallocating the owner can -make existing views stale, so copy the view when Python needs an independent -lifetime. Each fresh extraction starts at the array's current native lower -bounds; changing lower bounds during native reallocation must not offset the -first element exposed to NumPy. +make an existing view stale. Accessing a stale view is unsupported and may +crash the process; discard it and call `to_numpy()` again after the native state +changes. Each fresh extraction inspects the current descriptor and starts at +the current native lower bounds. Changing lower bounds during native +reallocation must not offset the first element exposed to NumPy. An allocatable array returned by a function or hidden output is different from a borrowed module or field handle. x2py transfers the result into persistent @@ -171,7 +172,7 @@ Create `allocations.f90`: module storage implicit none real(8), allocatable, target :: shared_values(:) - real(8), allocatable :: snapshot_values(:) + real(8), allocatable :: plain_values(:) contains function make_values(count) result(values) integer(4), intent(in) :: count @@ -200,27 +201,27 @@ contains shared_values = [(1.0_8 * index, index = 1, count)] end subroutine allocate_shared - subroutine allocate_snapshot(count) + subroutine allocate_plain(count) integer(4), intent(in) :: count integer(4) :: index - if (allocated(snapshot_values)) deallocate(snapshot_values) - allocate(snapshot_values(count)) - snapshot_values = [(3.0_8 * index, index = 1, count)] - end subroutine allocate_snapshot + if (allocated(plain_values)) deallocate(plain_values) + allocate(plain_values(count)) + plain_values = [(3.0_8 * index, index = 1, count)] + end subroutine allocate_plain subroutine release_shared() if (allocated(shared_values)) deallocate(shared_values) end subroutine release_shared - subroutine scale_snapshot(scale) + subroutine scale_plain(scale) real(8), intent(in) :: scale - snapshot_values = scale * snapshot_values - end subroutine scale_snapshot + plain_values = scale * plain_values + end subroutine scale_plain - subroutine release_snapshot() - if (allocated(snapshot_values)) deallocate(snapshot_values) - end subroutine release_snapshot + subroutine release_plain() + if (allocated(plain_values)) deallocate(plain_values) + end subroutine release_plain real(8) function shared_sum() result(total) total = sum(shared_values) @@ -229,14 +230,15 @@ end module storage ``` Inspecting `allocations.f90` prints allocatable array handles for module -storage, descriptor results, and descriptor arguments. Metadata such as -`Aliased` can still wrap the handle to describe owner or transfer policy: +storage, descriptor results, and descriptor arguments. `Aliased` remains a +language-neutral fact that native storage may be externally aliased or +addressed. It does not change `to_numpy()` extraction semantics: ```python from x2py.contracts import Addr, Aliased, Allocatable, Annotated, Arg, Float64, Int32, Returns, native_call shared_values: Annotated[Allocatable[Float64[:]], Aliased] -snapshot_values: Allocatable[Float64[:]] +plain_values: Allocatable[Float64[:]] @native_call([Addr(Arg(0))]) def make_values( @@ -253,24 +255,26 @@ def allocate_shared( ) -> None: ... @native_call([Addr(Arg(0))]) -def allocate_snapshot( +def allocate_plain( count: Int32 ) -> None: ... def release_shared() -> None: ... -def scale_snapshot( +def scale_plain( scale: Float64 ) -> None: ... -def release_snapshot() -> None: ... +def release_plain() -> None: ... def shared_sum() -> Float64: ... ``` -`snapshot_values` is not written as `Snapshot[...]`. For arrays, detached -copies are an extraction policy of the allocatable handle, not a separate -public array type. Whole-object snapshots are a separate derived-object feature. +`plain_values` and `shared_values` have the same extraction behavior: a fresh +`to_numpy()` call returns a live view of the current allocation or `None`. +`Aliased` remains present on `shared_values` because the native declaration +supplies the corresponding addressability fact. It does not change the +allocatable-array extraction mode. Build it: @@ -303,21 +307,22 @@ view = shared.to_numpy() view[0] = np.float64(10.0) assert api.shared_sum() == np.float64(15.0) -api.allocate_snapshot(np.int32(3)) -snapshot = api.snapshot_values.to_numpy() -np.testing.assert_array_equal(snapshot, np.array([3.0, 6.0, 9.0], dtype=np.float64)) -assert not snapshot.flags.writeable +api.allocate_plain(np.int32(3)) +plain_view = api.plain_values.to_numpy() +plain_copy = plain_view.copy() +plain_view[0] = np.float64(12.0) -api.scale_snapshot(np.float64(2.0)) -np.testing.assert_array_equal(snapshot, np.array([3.0, 6.0, 9.0], dtype=np.float64)) +api.scale_plain(np.float64(2.0)) +np.testing.assert_array_equal(plain_copy, np.array([3.0, 6.0, 9.0], dtype=np.float64)) np.testing.assert_array_equal( - api.snapshot_values.to_numpy(), - np.array([6.0, 12.0, 18.0], dtype=np.float64), + api.plain_values.to_numpy(), + np.array([24.0, 12.0, 18.0], dtype=np.float64), ) ``` -Do not access `view` after `api.release_shared()`; native deallocation makes -the previous borrowed view stale. +Do not access `view` after `api.release_shared()`, or `plain_view` after +`api.release_plain()` or another reallocation. Native storage changes make the +previous views stale, and accessing a stale view is unsupported and may crash. ## Output And Function Results @@ -438,15 +443,18 @@ descriptor would not update the module handle reliably. An allocatable module array is native-owned. Reading the Python attribute returns an `Allocatable[T[...]]` handle, not `ndarray | None`. The module's allocation routines create and release the storage. `h.to_numpy()` returns the -current view, detached copy, or `None` according to completed policy and current -allocation state. When the Fortran declaration has `target`, the generated -`.pyi` marks the handle with `Aliased`. `Aliased` is not an ownership mode; it -says x2py may expose the native storage through an alias. - -A plain allocatable module array remains wrappable. Its handle can still report -unallocated state. If policy cannot expose a live view safely, `to_numpy()` -returns a read-only detached copy when that path is implemented, or wrapper -readiness blocks with a clear diagnostic. +current live view or `None` according to the current allocation state. When the +Fortran declaration has `target`, the generated `.pyi` marks the handle with +`Aliased`. `Aliased` is not an ownership mode or an extraction selector; it +records native addressability for pointer association, raw-address, foreign-pointer, +and related policy. + +A plain allocatable module array has the same extraction contract as an +`Aliased` one. The wrapper uses the completed descriptor mechanism to inspect +the current allocation without copying. If the backend cannot expose a live +view through a supported mechanism, wrapper readiness blocks with a clear +diagnostic. Call `.copy()` explicitly when independent Python-owned storage is +required. A supported allocatable component belongs to its containing native derived-type instance. The generated wrapper owns that native instance. The field exposes an @@ -465,10 +473,25 @@ independent = view.copy() ## Limitations -- Allocatable scalar derived-type argument replacement is blocked. +- A wrapper-owned allocatable scalar derived result can be passed to a + compatible ordinary, target, allocatable, allocatable-target, input-only + pointer, or value dummy. The generated typed holder preserves the same Python + object and writes allocation changes back to that holder. +- An allocatable scalar derived module variable is a live nullable field proxy, + and it can satisfy a compatible allocatable dummy through a scoped + `move_alloc` transaction. The allocation is moved into an exact typed local + holder, passed to the native procedure, and restored exactly once; no object + address substitutes for the module descriptor and no descriptor crosses the + interoperable boundary. +- The complete ordinary, `TARGET`, `ALLOCATABLE`, `ALLOCATABLE,TARGET`, + `POINTER`, and `VALUE` compatibility rules—including empty state, + multi-argument cleanup, and deliberate errors—are in the later Wrapping + Derived Types guide under “Scalar Actuals And Native Dummies.” - Mutable scalar deferred-length character storage is blocked. -- Borrowed views require a proved native or wrapper owner and `Aliased` - storage when the owner is a module variable. +- Plain derived module objects use typed module-specific member access; + `Aliased` is needed only for policies that require a direct native address. + Allocatable module-array handles use their standard descriptor path and have + the same live-view extraction contract with or without `Aliased`. - Borrowed module handles do not provide the persistent direct descriptor handoff required by projected writable descriptor arguments; use an owned result handle for that operation. diff --git a/docs/user/guide/callbacks.md b/docs/user/guide/callbacks.md index f123b9562..6eeb0184f 100644 --- a/docs/user/guide/callbacks.md +++ b/docs/user/guide/callbacks.md @@ -182,18 +182,22 @@ survive such failures. - asynchronous or cross-thread callback invocation; and - persistent callback ownership during object or library teardown. -## Future Contract Policy +## Adapter Policy -Future callback contract work should enrich adapter policy, not reshape the -callback call signature. A `Callable[[...], T]` must continue to describe the -Fortran procedure interface that Fortran calls: argument order, value/reference -passing, storage shape, character length, and result type. The Python callable -can adapt argument names or order itself, so callback contracts should not grow -normal wrapper features such as argument reordering or hidden native-call -projection. +The post-IR policy stage completes how the adapter crosses the +Fortran-to-Python-to-Fortran boundary before wrapper generation begins. It +records value versus reference ABI, copy-in/copy-out direction, array shape, +fixed character length, exact derived type identity, call-scoped context +lifetime, entering-thread enforcement, GIL entry, cleanup, and the fatal error +action. The Python binding and Fortran bridge only lower those completed actions. -The useful future work is explicit policy for how the adapter crosses the -Fortran-to-Python-to-Fortran boundary: +`Callable[[...], T]` continues to describe the Fortran procedure interface: +argument order, value/reference passing, storage shape, character length, and +result type. The Python callable can adapt argument names or order itself, so +callback contracts do not use normal wrapper features such as argument +reordering or hidden native-call projection. + +Future work may add user-selectable policy for: - copy-in, copy-out, borrowed-view, and zero-copy choices; - dtype conversion, overflow checks, and result coercion; @@ -205,8 +209,8 @@ Fortran-to-Python-to-Fortran boundary: - callback-specific error/result policy beyond the current fatal native callback boundary. -This is planned design work, not current support. The semantic `.pyi` wrapper -roadmap later tracks the callback-adapter policy work. +These choices are not currently user-selectable; unsupported forms remain +blocked instead of selecting a different backend behavior. ## Evidence And Troubleshooting diff --git a/docs/user/guide/editing-semantic-pyi-contracts.md b/docs/user/guide/editing-semantic-pyi-contracts.md index 72c04ec91..222502585 100644 --- a/docs/user/guide/editing-semantic-pyi-contracts.md +++ b/docs/user/guide/editing-semantic-pyi-contracts.md @@ -508,12 +508,13 @@ module_values: Annotated[ ``` Python receives a persistent `AllocatableArray` for the module descriptor. -`handle.to_numpy()` may expose a zero-copy view because `Aliased` proves the -required addressability. NumPy must not free the data. A native +`handle.to_numpy()` exposes a current live view for both plain and `Aliased` +module allocatables. `Aliased` preserves native addressability for other policy; +it is not the extraction switch. NumPy must not free the data. A native allocate/deallocate routine controls the allocation, and a later native -deallocation or reallocation makes previous views stale. The same handle then -reports `allocated is False`; the module attribute itself does not become -`None`. +deallocation or reallocation makes previous views stale. Accessing stale views +is unsupported and may crash. The same handle then reports `allocated is +False`; the module attribute itself does not become `None`. Lifecycle: diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 763d1a6f7..9fe3fcdf0 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -495,14 +495,14 @@ releases x2py-owned descriptor storage. A component becomes a borrowed | Term | Meaning | Typical example | | --- | --- | --- | -| Python-owned | Python or NumPy owns the value or data buffer and releases it normally. | Scalar results, strings, copy-return arrays, pointer detached copies. | +| Python-owned | Python or NumPy owns the value or data buffer and releases it normally. | Scalar results, strings, copy-return arrays, and explicitly copied pointer-scalar values. | | Caller-owned | The caller supplied the Python object and retains ownership. | A NumPy array passed as `intent(in)`, `intent(out)`, or `intent(inout)`. | | Wrapper-owned | A Python object owns or controls native storage. | A wrapped derived-type result or owned allocatable result handle. | | Native-owned | Fortran or an external library owns storage independently of Python. | A module allocatable array or external-library buffer. | | Descriptor handle | Python carries native allocation or association state and a completed owner policy. | An allocatable or pointer module variable, field, argument, or supported result. | | Borrowed view | Python references storage owned elsewhere and does not destroy it. | A NumPy array extracted from a borrowed descriptor handle. | | Copy-return | Native output is copied into a new Python-owned value before return. | An ordinary array result or immutable replacement. | -| Detached copy (`snapshot_copy` policy) | Python receives a copy of current native state, not a live view. | Scalar pointer copied values or detached handle extraction. | +| Detached copy (`snapshot_copy` policy) | Python receives a copy of current native state, not a live view. | Scalar pointer copied values or another explicit copy-result contract; native-array-handle `to_numpy()` never selects this behavior. | | Call-local association | Native code may use Python storage only during the wrapped call. | Pointer `intent(in)` array arguments. | | Blocked | Generation stops because a safe contract cannot be proven. | Pointer reassociation without owner and release policy. | @@ -530,7 +530,7 @@ X2PY_C_DOCS_END --> | Value | Who destroys it | When | | --- | --- | --- | | Python scalar or string | Python | When Python references are gone. | -| Copy-return or detached-copy NumPy array | NumPy or its generated base capsule | When Python references are gone. | +| Copy-return or explicitly detached NumPy array | NumPy or its generated base capsule | When Python references are gone. | | Caller-supplied NumPy array | The Python caller | According to normal Python lifetime. | | Wrapper-owned derived instance | Generated wrapper deallocator | When the owning wrapper is collected. | | Borrowed nested component | The parent wrapper | When parent and all borrowed children are gone. | @@ -984,19 +984,34 @@ np.testing.assert_array_equal(values.to_numpy(), [10.0, 20.0]) An allocatable derived-type field is owned by its containing native instance. Access returns an `AllocatableArray` that retains the wrapper owner. An allocatable module array also returns a handle whose descriptor storage remains -module-owned. `Aliased` permits borrowed view extraction; otherwise completed -policy may select a detached read-only extraction. The attribute itself remains -a handle when unallocated. Views returned by `to_numpy()` can become stale -after native reallocation; copy before reallocation when independent lifetime -is required. - -Allocatable scalar derived-type dummy replacement remains blocked because a -safe contract must define native construction, replacement, finalization, and -exactly-once destruction of the whole wrapped object. +module-owned. Plain and `Aliased` module handles both return a current live view +or `None`; the annotation records addressability for other native policy and +does not select extraction behavior. The attribute itself remains a handle when +unallocated. Views returned by `to_numpy()` can become stale after native +reallocation; accessing a stale view is unsupported and may crash. Copy +explicitly before reallocation when independent lifetime is required. + +Rank-zero allocatable and pointer derived module variables expose the same live +field surface, returning `None` while absent. A wrapper-owned allocatable +or pointer derived result uses persistent typed holder storage. Module +allocatables use scoped `move_alloc` transactions for allocatable dummies; +module pointers use typed local pointer transactions for reassociable pointer +dummies. Both restore final state exactly once without transporting a native +descriptor through the interoperable boundary. Reallocation, deallocation, +reassociation, or +nullification makes previously returned payload proxies stale; field access on +such a proxy raises `ReferenceError`. + +The full five-actual by six-dummy matrix, including `TARGET`, `VALUE`, empty +state, pointer `INTENT(IN)`, deliberate incompatibilities, and calls with many +derived objects, is maintained in +[Scalar Actuals And Native Dummies](wrapping-derived-types.md#scalar-actuals-and-native-dummies). Runtime tests: [`test_allocatable_views.py`](../../../tests/wrapper/fortran/module_state/test_allocatable_views.py) and +[`test_scalar_derived_actual_dummy_matrix.py`](../../../tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py) +and [`test_allocatable_replacement.py`](../../../tests/wrapper/fortran/module_state/test_allocatable_replacement.py). ## Pointer Arguments, Results, And Association @@ -1299,8 +1314,10 @@ Private components are omitted from Python descriptors. Allocatable fields use descriptor access; that retention does not make the wrapper owner of a pointer target. Arrays of derived types are blocked. -Runtime tests: [`test_derived_type_boundaries.py`](../../../tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py) -and [`test_derived_type_methods.py`](../../../tests/wrapper/fortran/derived_types/test_derived_type_methods.py). +Runtime tests: [`test_derived_type_boundaries.py`](../../../tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py), +[`test_derived_type_methods.py`](../../../tests/wrapper/fortran/derived_types/test_derived_type_methods.py), +and the direct Phase 8 object and field evidence in +[`test_phase8_derived_plan.py`](../../../tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py). ## Inheritance And Polymorphism @@ -1400,9 +1417,14 @@ class settings: The target method must have the same Python call shape and return type. A public target remains callable as a method; `@private` keeps the signature in the -standalone `.pyi` but exposes only construction to users. Fortran generic -constructor interfaces and overloaded runtime `tp_init` lowering are not yet -mapped; they report explicit blockers. +standalone `.pyi` but exposes only construction to users. + +An edited contract can instead declare multiple `__init__` overload links. +The wrapper allocates the ordinary Phase 8 native owner once, selects an exact +candidate from completed dtype/rank/class predicates, invokes that target, and +commits ownership only after it succeeds. Selection never calls candidates to +see which one works. Indistinguishable candidates fail during generation and a +runtime call with no match raises `TypeError` before native entry. ### Finalization @@ -1417,6 +1439,9 @@ native execution terminates the process. Runtime tests: [`test_constructors_and_finalizers.py`](../../../tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py) and [`test_borrowed_finalizers.py`](../../../tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py). +Bound and overloaded constructors are covered by +[`test_phase9_bound_constructors.py`](../../../tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py) +and [`test_phase9_class_overloads.py`](../../../tests/wrapper/fortran/naming/test_phase9_class_overloads.py). ## Module Variables, Constants, Saved State, And Common Blocks @@ -1678,10 +1703,11 @@ X2PY_C_DOCS_END --> | Persistent callbacks and procedure pointers | Unsupported | [Callback limitations](../guide/callbacks.md#unsupported-forms) | [Callback route](../../developer/source-map.md#common-change-routes) | [Callback tests](../../../tests/wrapper/fortran/callbacks/test_scalar_callbacks.py), [readiness tests](../../../tests/semantics/readiness/) | Callbacks are valid only during the wrapped call. | | Advanced multi-source dependency discovery and external-library integration | Unsupported | [Packaging limits](../guide/packaging.md#limitations) | [Build orchestration](../../developer/source-map.md#common-change-routes) | [Multi-source tests](../../../tests/wrapper/fortran/multiple_files/test_multi_source_builds.py) | x2py does not infer dependency graphs, prebuilt module paths, or external library discovery. | | Blocked array forms | Unsupported | [Unsupported array forms](../guide/arrays.md#unsupported-forms) | [Readiness route](../../developer/source-map.md#common-change-routes) | [Readiness tests](../../../tests/semantics/readiness/), [diagnostics](../reference/diagnostic-codes.md) | Assumed type `type(*)`, arrays of derived types, and character arrays not representable as fixed-width bytes need missing runtime contracts. | -| Unsupported polymorphic forms | Unsupported | [Inheritance limits](../guide/wrapping-derived-types.md#inheritance-and-polymorphism) | [Semantic readiness](../../developer/source-map.md#common-change-routes) | [Readiness tests](../../../tests/semantics/readiness/) | Results, mutable dummies, arrays, allocatable/pointer scalars, and `class(*)` are blocked. | -| Generic constructor interfaces and overloaded runtime initialization | Unsupported | [Constructor limitations](../guide/wrapping-derived-types.md#constructors) | [Constructor route](../../developer/source-map.md#common-change-routes) | [Constructor tests](../../../tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py), [readiness tests](../../../tests/semantics/readiness/) | Deterministic Python constructor selection and lowering is not complete. | +| Unsupported polymorphic forms | Unsupported | [Inheritance limits](../guide/wrapping-derived-types.md#inheritance-and-polymorphism) | [Semantic readiness](../../developer/source-map.md#common-change-routes) | [Readiness tests](../../../tests/semantics/readiness/) | Results, mutable dummies, arrays, polymorphic allocatable/pointer scalars, and `class(*)` are blocked. | +| Ambiguous or incomplete constructor overload sets | Unsupported | [Constructor limitations](../guide/wrapping-derived-types.md#constructors) | [Constructor route](../../developer/source-map.md#common-change-routes) | [Constructor overload tests](../../../tests/wrapper/fortran/naming/test_phase9_class_overloads.py), [class-plan validation tests](../../../tests/wrapper_codegen/test_phase9_class_surfaces.py) | Candidates must have distinguishable exact runtime signatures and compatible native-owner lifecycles. | | Character arrays and mutable deferred-length character storage | Partially supported | [Strings](../guide/data-types.md#strings) | [Character bridge route](../../developer/source-map.md#common-change-routes) | [Character edge tests](../../../tests/wrapper/fortran/strings/test_character_edge_cases.py), [readiness tests](../../../tests/semantics/readiness/) | Character arrays use fixed-width NumPy bytes dtype. Fixed and allocatable deferred element length maps to dtype itemsize; Unicode/object arrays and mutable scalar deferred-length storage are unsupported. | | Wider-than-supported real, complex, and logical storage | Unsupported | [Datatype limits](../guide/data-types.md#unsupported-widths-and-forms) | [Type probing](../../developer/source-map.md#hotspot-index) | [Scalar kind tests](../../../tests/wrapper/fortran/scalars/test_scalar_kinds.py), [readiness tests](../../../tests/semantics/readiness/) | x2py blocks rather than silently losing precision or Boolean storage semantics. | diff --git a/docs/user/reference/callbacks.md b/docs/user/reference/callbacks.md index 3cc1ad720..3e2aa0604 100644 --- a/docs/user/reference/callbacks.md +++ b/docs/user/reference/callbacks.md @@ -24,6 +24,12 @@ callable alive only for the active wrapped call, installs a callback context, passes a generated Fortran adapter to the native routine, and clears the context when the wrapped call returns. +Policy completion records the callback ABI, adapter copy direction, context +lifecycle, entering-thread rule, GIL actions, and fatal action before wrapper +planning. Direct wrapper-plan generation then emits one typed Fortran adapter +and one native trampoline per callback site. Neither backend infers callback access, +shape, or ownership from generated locals. + Native code must not store the callback or call it later. Stored procedure pointers, optional dummy procedures, asynchronous callbacks, and cross-thread callback invocation are unsupported. diff --git a/docs/user/reference/generated-classes.md b/docs/user/reference/generated-classes.md index ed97a011b..b10855b14 100644 --- a/docs/user/reference/generated-classes.md +++ b/docs/user/reference/generated-classes.md @@ -59,10 +59,12 @@ Omitted keywords preserve native default initialization. Private components, arrays, allocatables, pointers, strings, and nested derived components are not automatic constructor keywords. -An edited semantic `.pyi` may remove the generated constructor or bind one -concrete initializer. x2py does not recreate a constructor intentionally -removed from the contract. Generic constructor interfaces remain unsupported -when x2py cannot select one deterministic Python initializer. +An edited semantic `.pyi` may remove the generated constructor, bind one +concrete initializer, or replace it with an exact overload set. x2py does not +recreate a constructor intentionally removed from the contract. Overloaded +constructors select a concrete target from the completed scalar dtype, array +dtype/rank, or generated-class predicates before invoking native code. An +indistinguishable or incomplete set is rejected during generation. ## Fields And Methods @@ -76,8 +78,10 @@ its parent wrapper; `to_numpy()` performs explicit extraction when completed policy supports it. Arrays of derived types are unsupported. Whole-object snapshot classes are not part of the active generated contract. -Plain derived module variables need a completed live-borrow policy, such as -`Aliased`, before wrapper generation can expose them. +Plain and `Aliased` derived module variables both expose this normal live field +surface. An `Aliased` declaration permits a direct-address borrowed wrapper; +a plain declaration uses typed module-specific getter and setter bridge +operations without fabricating a whole-object address. Type-bound procedures become methods. The generated semantic contract uses `Pass()` on the concrete native-specific method when the native passed-object @@ -105,9 +109,11 @@ class accumulator: ) -> None: ... ``` -Method dispatch is exact. Indistinguishable overloads or unsupported generic -constructors block generation. The overload declaration is only a Python -dispatch link; it cannot also carry `@native_call(...)`. +Method and constructor dispatch is exact. Calls are normalized against each +candidate's declared positional and keyword parameters, then matched without +calling candidates speculatively. Indistinguishable overloads block generation; +a call with no match raises a stable `TypeError`. The overload declaration is +only a Python dispatch link; it cannot also carry `@native_call(...)`. ## Ownership And Finalization @@ -118,9 +124,10 @@ releases that native instance exactly once. Borrowed child wrappers, borrowed module objects, and borrowed component views do not destroy the storage they reference. They retain the owning wrapper or module reference needed for Python lifetime, but explicit native deallocation -or reallocation can still invalidate borrowed storage. Whole-object snapshots -are future-only, so the active wrapper blocks plain derived module variables -that lack addressability or another completed ownership policy. +or reallocation can still invalidate borrowed storage. Plain and `Aliased` +derived module objects are both live native-owned objects. An `Aliased` object +may use a proved native address; a plain object uses module-specific bridge +operations and must not fabricate addressability. Native finalizers do not provide a recoverable Python status channel during object destruction. Use ordinary wrapped procedures for recoverable cleanup @@ -148,6 +155,10 @@ Generated class behavior is covered by [`test_constructors_and_finalizers.py`](../../../tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py), [`test_borrowed_finalizers.py`](../../../tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py), and [`test_inheritance.py`](../../../tests/wrapper/fortran/derived_types/test_inheritance.py). +Exact class-method and constructor overloads are covered by +[`test_phase9_class_overloads.py`](../../../tests/wrapper/fortran/naming/test_phase9_class_overloads.py), +and explicit bound construction by +[`test_phase9_bound_constructors.py`](../../../tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py). When class behavior changes, update this page with the derived-type user guide, semantic `.pyi` reference, generated contract fixtures, and ownership evidence. diff --git a/docs/user/reference/generated-modules.md b/docs/user/reference/generated-modules.md index db107419b..4985fc97e 100644 --- a/docs/user/reference/generated-modules.md +++ b/docs/user/reference/generated-modules.md @@ -84,11 +84,19 @@ Python, but it does not mutate the native parameter. Supported module arrays, allocatables, pointers, and derived objects follow the ownership rules in [Memory Management](../guide/memory-management.md) and -the topic-specific user-guide pages. `Aliased` derived module variables return -live borrowed wrappers. Plain derived module variables without a completed -live-borrow policy block readiness; whole-object `Snapshot[T]` contracts are a -future feature, not an active generated module surface. Missing addressability, -ownership, release, mutability, or safe-copy facts block generation. +the topic-specific user-guide pages. Plain and `Aliased` derived module +variables return live native-owned objects. `Aliased` permits an address-backed +borrow; a plain declaration uses typed module-specific access. Missing +ownership, release, mutability, or a supported module-object mechanism blocks +generation. + +Rank-zero `Allocatable[Derived]` and `Pointer[Derived]` module variables are +nullable live proxies: the module attribute is `None` while native state is +absent and otherwise exposes the generated fields. Compatible allocatable +dummies use a reversible typed `move_alloc` transaction; reassociable pointer +dummies use a typed pointer transaction and restore the final association. +Payload-only calls use direct or synchronous scoped addresses. See the +[complete scalar-derived compatibility matrix](../guide/wrapping-derived-types.md#scalar-actuals-and-native-dummies). ## Visibility, Binding Names, And Imports diff --git a/docs/user/reference/python-api.md b/docs/user/reference/python-api.md index 7204c4e15..b93f36e60 100644 --- a/docs/user/reference/python-api.md +++ b/docs/user/reference/python-api.md @@ -130,14 +130,15 @@ projections remain ordinary `T | None` values and never produce an `AllocatableArray` or `PointerArray`. `to_numpy()` returns `None` when the descriptor is currently unallocated or -unassociated. Otherwise, the completed wrapper policy decides whether the result -is a live view, a detached copy, or an unsupported extraction. Once the handle -has reported allocated or associated state, generated extraction must return -array storage rather than `None`. Results must match the handle's declared dtype -and rank. Contiguous-view policy rejects non-contiguous storage, copy-only policy -returns detached NumPy storage, and pointer descriptor-view extraction can expose -positive or negative stride views when generated TS 29113 descriptor support is -available. +unassociated. Otherwise, it returns a live NumPy view of the current allocation +or pointer target and never an automatic detached copy. Results must match the +handle's declared dtype and rank. Contiguous-view policy rejects non-contiguous +storage, while descriptor-view extraction can expose positive or negative +strides when generated standard descriptor support is available. Unsupported +descriptor extraction fails explicitly. Reallocation, deallocation, pointer +reassociation, or nullification may make an older view stale; accessing a stale +view is unsupported and may crash. Call `.copy()` explicitly when independent +storage is required, and call `to_numpy()` again to inspect current state. When a generated wrapper accepts a handle for an ordinary `T[...]` argument, it uses an internal native array-actual handoff rather than an implicit diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index c52fd997a..df3f17a5a 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -817,6 +817,12 @@ The wrappers are valid only in `Callable[[...], T]` argument lists and are lowered internally to callback declaration access (`read`, `write`, `readwrite`, or `unspecified`). They are not general `.pyi` argument direction metadata. + +Before wrapper planning, x2py completes these declarations into explicit +callback ABI, copy direction, result, context-lifecycle, thread/GIL, cleanup, +and fatal-error actions. The generated C trampoline and Fortran adapter consume +that completed record directly; editing a `Callable` never causes a backend to +guess a different access or ownership mode. X2PY_C_DOCS_END --> -All specifics must have one compatible Python call shape. Parameter names and -keyword parsing use the first specific procedure's signature. A call that -matches no specific raises `TypeError`; duplicate dtype/rank signatures are a -deterministic generation error. +Each specific keeps its declared Python call shape. The generated dispatcher +normalizes positional and keyword arguments against each candidate, then uses +the candidate's exact typed predicates. A call that matches no specific raises +`TypeError`; duplicate runtime dtype/rank/class signatures are a deterministic +generation error. ## Defined Operators And Assignment @@ -1815,13 +1831,12 @@ exposed as `Allocatable[T[...]]` handles. The handle carries allocation state and descriptor operations. It is not a NumPy array, and unallocated state lives inside the handle. -`h.to_numpy()` returns `None` when the descriptor is unallocated. When completed -policy proves live aliasing is safe, it returns a borrowed NumPy view over -native storage. For derived-type fields, the extracted view retains the field +`h.to_numpy()` returns `None` when the descriptor is unallocated. Otherwise it +returns a live NumPy view over the current native storage and never an automatic +detached copy. For derived-type fields, the extracted view retains the field handle and the field handle retains the containing Python wrapper. For module -variables, the handle retains the generated module owner while the Fortran -module controls allocation. When live aliasing is unsafe but copying is -implemented, `to_numpy()` returns a read-only detached copy instead. +variables, the handle retains the generated module owner while the native +module controls allocation. Existing views are not invalidated, detached, or tracked. If a wrapped Fortran procedure reallocates or deallocates native storage while Python still holds an @@ -1829,7 +1844,7 @@ old view, that old view is stale; reading or writing it is unsupported and may crash the process. Users who need independent lifetime must copy explicitly: ```python -x = obj.values.to_numpy() # borrowed view, detached copy, or None +x = obj.values.to_numpy() # live view or None y = None if x is None else x.copy() obj.reset_values() # may invalidate x; y remains valid ``` @@ -1918,9 +1933,11 @@ class state: The generated keyword-only shape remains reserved: if undecorated `__init__` keeps the `self, *, ...` form and every keyword has a default, the loader treats it as the generated field constructor metadata. Constructor overload -declarations may still be used only when the generated field constructor is -present; overloaded `tp_init` runtime lowering is not implemented yet and code -generation reports an explicit blocker for that form. +declarations replace runtime field initialization with an exact constructor +overload set linked to concrete same-class targets. The direct wrapper allocates +one native owner, dispatches without candidate trial calls, and releases an +uncommitted owner on failure. Missing, incompatible, or indistinguishable +candidates are rejected before source emission. Module variables are declarations in the semantic contract. Allocatable array module variables expose handles; unallocated state is represented by the handle, @@ -1930,18 +1947,18 @@ not by making the module attribute `None`: from x2py.contracts import Aliased, Allocatable, Annotated, Float64 module_values: Annotated[Allocatable[Float64[:]], Aliased] -snapshot_values: Allocatable[Float64[:]] +plain_values: Allocatable[Float64[:]] ``` -`Aliased` says a native-owned borrowed view may be exposed through -`to_numpy()`. A plain allocatable module array remains wrappable as a handle; -if live aliasing is unsafe, `to_numpy()` uses a read-only detached copy when -that extraction policy is implemented. Fortran source declarations with -`target` are printed as `Aliased` because they prove that the current allocation -may be aliased by the wrapper. +Both declarations expose a stable native-owned handle whose `to_numpy()` call +returns a current live view or `None`. `Aliased` does not select view versus +copy extraction. It remains a language-neutral fact that native storage may be +externally aliased or addressed. Fortran source declarations with `target` are +printed as `Aliased` because they supply that native fact. -`Aliased` also controls borrowed access to an existing derived-type module -object: +`Aliased` also records addressability used by borrowed access to an existing +derived-type module object. Plain derived module objects remain live but use a +different bridge mechanism: ```python from x2py.contracts import Aliased, Allocatable, Annotated, Float64 @@ -1950,15 +1967,17 @@ class box: values: Allocatable[Float64[:]] live_current: Annotated[box, Aliased] +plain_current: box ``` The annotation belongs to the module variable, not to `box`. An x2py-created `box()` is addressable because its generated constructor allocates pointer-backed native storage. A native module declaration is a different object origin. `Annotated[box, Aliased]` lets the wrapper retain that object's native -address and return a live borrowed `box` wrapper. Without `Aliased` or another -completed policy, the derived module object blocks readiness because -whole-object snapshots are not part of the active contract. +address and return a live borrowed `box` wrapper. A plain `box` module variable +returns the same public wrapper type, backed by typed module-specific bridge +operations. Unsupported live lifetime or module-access policy blocks readiness; +the backend must not fall back to a detached object or an invented address. Public scalar Fortran module variables are emitted directly with their resolved semantic type: @@ -2082,9 +2101,10 @@ The handle carries association state and descriptor operations: When descriptor-backed extraction is enabled, `to_numpy()` builds NumPy shape and strides from descriptor metadata and can expose strided pointer targets. If that -path is unavailable, the completed policy must choose contiguous-only views, -an explicit copy fallback, or a readiness diagnostic. Pointer handle ownership -is descriptor or association access by default, not target ownership. +path is unavailable, the completed policy must choose a contiguous live view or +an explicit readiness diagnostic. It must not fall back to a copy. Pointer +handle ownership is descriptor or association access by default, not target +ownership. The generated descriptor-view path establishes portable descriptor storage, associates an `intent(out)` pointer dummy with the live target, and decodes the descriptor synchronously. It does not inspect a compiler-private Fortran diff --git a/tests/_shared/ownership_policy_support.py b/tests/_shared/ownership_policy_support.py index 80f3fc088..895a89e98 100644 --- a/tests/_shared/ownership_policy_support.py +++ b/tests/_shared/ownership_policy_support.py @@ -89,7 +89,6 @@ RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA, RESOLVED_OWNERSHIP_POLICY_METADATA, RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA, - RESOLVED_SNAPSHOT_FIELD_ACTION_METADATA, ProjectionMapping, SemanticArgument, SemanticArrayContract, @@ -286,7 +285,6 @@ def _native_array_policy( "RESOLVED_OWNERSHIP_POLICY_METADATA", "RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA", "RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA", - "RESOLVED_SNAPSHOT_FIELD_ACTION_METADATA", "ArrayInteropPolicy", "ArrayInteropPolicyDispatcher", "AssignmentMode", diff --git a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py b/tests/codegen/bindings/test_binding_handle_policy_dispatch.py index 0884856cf..d290d24d1 100644 --- a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py +++ b/tests/codegen/bindings/test_binding_handle_policy_dispatch.py @@ -250,12 +250,6 @@ def test_bridge_and_binding_generators_expose_ownership_action_maps(): ] == "_derived_module_variable" ) - assert ( - FortranToCBridgeGenerator._MODULE_VARIABLE_POLICY_DISPATCHER.handlers[ - (ObjectKind.DERIVED_TYPE, CodegenAction.SNAPSHOT_COPY) - ] - == "_snapshot_derived_module_variable" - ) assert ( CPythonBindingGenerator._ARRAY_RELEASE_POLICY_DISPATCHER.handlers[DestructionPolicy.PYTHON_REFCOUNT] == "_release_python_owned_array_memory" diff --git a/tests/codegen/printers/_support.py b/tests/codegen/printers/_support.py index e2a847fe6..27243e8f0 100644 --- a/tests/codegen/printers/_support.py +++ b/tests/codegen/printers/_support.py @@ -8,7 +8,6 @@ from x2py import parse_fortran_file as parse_fortran_source -from x2py.semantics.metadata import SNAPSHOT_TYPE_METADATA from x2py.codegen.binding_pipeline import BindingPipeline @@ -89,7 +88,6 @@ def normalize(text: str) -> str: "OPERATOR_F90_SOURCE", "RUNTIME_HOLD_GIL_METADATA", "RUNTIME_STATUS_ERROR_METADATA", - "SNAPSHOT_TYPE_METADATA", "BindingPipeline", "Codegen", "Path", diff --git a/tests/codegen/printers/test_classes_and_methods.py b/tests/codegen/printers/test_classes_and_methods.py index 9fe0bae03..d5c6dfc02 100644 --- a/tests/codegen/printers/test_classes_and_methods.py +++ b/tests/codegen/printers/test_classes_and_methods.py @@ -303,7 +303,7 @@ def test_emit_and_load_aliased_derived_module_variable_declaration(): assert codegen_module.variables[0].is_target is True -def test_emit_module_stubs_do_not_print_plain_derived_module_variable_as_snapshot(): +def test_emit_module_stubs_print_plain_derived_module_variable_as_live_object(): source = """ module derived_module_snapshot type :: box @@ -317,7 +317,6 @@ def test_emit_module_stubs_do_not_print_plain_derived_module_variable_as_snapsho code = emit_module_stubs(semantic_module)["derived_module_snapshot"] - assert "Snapshot[" not in code assert "current: box" in code loaded = parse_pyi_text(code, module_name="derived_module_snapshot") assert loaded.variables[0].semantic_type.name == "box" diff --git a/tests/codegen/printers/test_types_and_declarations.py b/tests/codegen/printers/test_types_and_declarations.py index 82c5c4c75..91e202c82 100644 --- a/tests/codegen/printers/test_types_and_declarations.py +++ b/tests/codegen/printers/test_types_and_declarations.py @@ -3,12 +3,10 @@ from tests.codegen.printers._support import ( ProjectionMapping, PyiPrinter, - SNAPSHOT_TYPE_METADATA, SemanticArgument, SemanticArrayContract, SemanticClass, SemanticConstraint, - SemanticField, SemanticFunction, SemanticMethod, SemanticModule, @@ -24,22 +22,6 @@ ) -def test_emit_snapshot_type_wrapper_is_not_active_contract(): - module = SemanticModule( - name="snapshot_mod", - classes=[SemanticClass(name="box", fields=[SemanticField("value", SemanticType("Float64"))])], - variables=[ - SemanticArgument( - "current", - SemanticType("box", dtype="box", metadata={SNAPSHOT_TYPE_METADATA: True}), - ) - ], - ) - - with pytest.raises(ValueError, match=r"Snapshot\[T\] is not an active semantic \.pyi contract"): - emit_module(module) - - def test_emit_basic_scalar_function(): source = """ module math_mod diff --git a/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 b/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 new file mode 100644 index 000000000..79ab1663d --- /dev/null +++ b/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 @@ -0,0 +1,346 @@ +module phase8_left_types + use iso_c_binding, only: c_int32_t + implicit none + + type :: item + integer(c_int32_t) :: value = 0_c_int32_t + end type item + +contains + + function make_item(initial) result(value) + integer(c_int32_t), intent(in) :: initial + type(item) :: value + value%value = initial + end function make_item + +end module phase8_left_types + +module phase8_right_types + use iso_c_binding, only: c_int32_t + implicit none + + type :: item + integer(c_int32_t) :: value = 0_c_int32_t + end type item + +contains + + function make_item(initial) result(value) + integer(c_int32_t), intent(in) :: initial + type(item) :: value + value%value = initial + end function make_item + +end module phase8_right_types + +module fscalar_derived_actual_dummy_matrix_f90 + use iso_c_binding, only: c_bool, c_int, c_int32_t + use phase8_left_types, only: left_item => item + use phase8_right_types, only: right_item => item + implicit none + + type :: item + integer(c_int32_t) :: value = 0_c_int32_t + end type item + + type :: sequence_item + sequence + integer(c_int32_t) :: value = 0_c_int32_t + end type sequence_item + + ! The five rank-zero module actual declaration forms. + type(item) :: ordinary_module + type(item) :: ordinary_module_two + type(item), target :: target_module + type(item), allocatable :: allocatable_module + type(item), allocatable, target :: allocatable_target_module + type(item), pointer :: pointer_module => null() + + ! Extra distinct origins let one native call exercise several transactions. + type(item), target :: pointer_target_one + type(item), target :: pointer_target_two + type(item), pointer :: pointer_module_two => null() + type(item), pointer :: allocation_follower => null() + integer(c_int32_t) :: writable_call_count = 0_c_int32_t + +contains + + subroutine reset_state() + ordinary_module%value = 10_c_int32_t + ordinary_module_two%value = 12_c_int32_t + target_module%value = 20_c_int32_t + pointer_target_one%value = 50_c_int32_t + pointer_target_two%value = 60_c_int32_t + if (allocated(allocatable_module)) deallocate(allocatable_module) + if (allocated(allocatable_target_module)) deallocate(allocatable_target_module) + allocate(allocatable_module) + allocate(allocatable_target_module) + allocatable_module%value = 30_c_int32_t + allocatable_target_module%value = 40_c_int32_t + pointer_module => pointer_target_one + pointer_module_two => pointer_target_two + nullify(allocation_follower) + writable_call_count = 0_c_int32_t + end subroutine reset_state + + subroutine clear_allocatable_module() + if (allocated(allocatable_module)) deallocate(allocatable_module) + end subroutine clear_allocatable_module + + subroutine clear_allocatable_target_module() + nullify(allocation_follower) + if (allocated(allocatable_target_module)) deallocate(allocatable_target_module) + end subroutine clear_allocatable_target_module + + subroutine clear_pointer_module() + nullify(pointer_module) + end subroutine clear_pointer_module + + subroutine associate_allocation_follower() + if (allocated(allocatable_target_module)) then + allocation_follower => allocatable_target_module + else + nullify(allocation_follower) + end if + end subroutine associate_allocation_follower + + function allocation_follower_value() result(value) + integer(c_int32_t) :: value + value = -1_c_int32_t + if (associated(allocation_follower)) value = allocation_follower%value + end function allocation_follower_value + + function get_writable_call_count() result(value) + integer(c_int32_t) :: value + value = writable_call_count + end function get_writable_call_count + + function make_item(initial) result(value) + integer(c_int32_t), intent(in) :: initial + type(item) :: value + value%value = initial + end function make_item + + function make_sequence_item(initial) result(value) + integer(c_int32_t), intent(in) :: initial + type(sequence_item) :: value + value%value = initial + end function make_sequence_item + + subroutine make_target_item(initial, value) + integer(c_int32_t), intent(in) :: initial + type(item), target, intent(out) :: value + value%value = initial + end subroutine make_target_item + + subroutine make_allocatable_item(initial, make_present, value) + integer(c_int32_t), intent(in) :: initial + logical(c_bool), intent(in) :: make_present + type(item), allocatable, intent(out) :: value + if (make_present) then + allocate(value) + value%value = initial + end if + end subroutine make_allocatable_item + + subroutine make_allocatable_target_item(initial, make_present, value) + integer(c_int32_t), intent(in) :: initial + logical(c_bool), intent(in) :: make_present + type(item), allocatable, target, intent(out) :: value + if (make_present) then + allocate(value) + value%value = initial + end if + end subroutine make_allocatable_target_item + + function make_pointer_item(selector) result(value) + integer(c_int32_t), intent(in) :: selector + type(item), pointer :: value + select case (selector) + case (1_c_int32_t) + value => pointer_target_one + case (2_c_int32_t) + value => pointer_target_two + case default + nullify(value) + end select + end function make_pointer_item + + ! The six exact native dummy forms. + function read_object(value) result(observed) + type(item), intent(in) :: value + integer(c_int32_t) :: observed + observed = value%value + end function read_object + + function read_target(value) result(observed) + type(item), target, intent(in) :: value + integer(c_int32_t) :: observed + observed = value%value + end function read_target + + function read_allocatable(value) result(observed) + type(item), allocatable, intent(in) :: value + integer(c_int32_t) :: observed + observed = -1_c_int32_t + if (allocated(value)) observed = value%value + end function read_allocatable + + function read_allocatable_target(value) result(observed) + type(item), allocatable, target, intent(in) :: value + integer(c_int32_t) :: observed + observed = -1_c_int32_t + if (allocated(value)) observed = value%value + end function read_allocatable_target + + function read_pointer_input(value) result(observed) + type(item), pointer, intent(in) :: value + integer(c_int32_t) :: observed + observed = -1_c_int32_t + if (associated(value)) observed = value%value + end function read_pointer_input + + function read_value(value) result(observed) + type(item), value :: value + integer(c_int32_t) :: observed + observed = value%value + end function read_value + + function read_sequence_value(value) result(observed) + type(sequence_item), value :: value + integer(c_int32_t) :: observed + observed = value%value + end function read_sequence_value + + subroutine increment_object(value, amount) + type(item), intent(inout) :: value + integer(c_int32_t), intent(in) :: amount + value%value = value%value + amount + end subroutine increment_object + + subroutine set_allocatable(value, new_value) + type(item), allocatable, intent(inout) :: value + integer(c_int32_t), intent(in) :: new_value + if (new_value < 0_c_int32_t) then + if (allocated(value)) deallocate(value) + return + end if + if (.not. allocated(value)) allocate(value) + value%value = new_value + end subroutine set_allocatable + + subroutine set_allocatable_target(value, new_value) + type(item), allocatable, target, intent(inout) :: value + integer(c_int32_t), intent(in) :: new_value + if (new_value < 0_c_int32_t) then + if (allocated(value)) deallocate(value) + return + end if + if (.not. allocated(value)) allocate(value) + value%value = new_value + end subroutine set_allocatable_target + + subroutine set_pointer(value, selector) + type(item), pointer, intent(inout) :: value + integer(c_int32_t), intent(in) :: selector + select case (selector) + case (1_c_int32_t) + value => pointer_target_one + case (2_c_int32_t) + value => pointer_target_two + case (3_c_int32_t) + nullify(value) + allocate(value) + value%value = 70_c_int32_t + case (4_c_int32_t) + if (associated(value)) deallocate(value) + nullify(value) + case default + nullify(value) + end select + end subroutine set_pointer + + function read_six_forms(object_value, target_value, allocatable_value, & + allocatable_target_value, pointer_value, value_value) result(total) + type(item), intent(in) :: object_value + type(item), target, intent(in) :: target_value + type(item), allocatable, intent(in) :: allocatable_value + type(item), allocatable, target, intent(in) :: allocatable_target_value + type(item), pointer, intent(in) :: pointer_value + type(item), value :: value_value + integer(c_int32_t) :: total + total = object_value%value + target_value%value + value_value%value + if (allocated(allocatable_value)) total = total + allocatable_value%value + if (allocated(allocatable_target_value)) total = total + allocatable_target_value%value + if (associated(pointer_value)) total = total + pointer_value%value + end function read_six_forms + + function read_qualified(left, right) result(total) + type(left_item), intent(in) :: left + type(right_item), intent(in) :: right + integer(c_int32_t) :: total + total = left%value * 100_c_int32_t + right%value + end function read_qualified + + subroutine mutate_three_descriptors(first, second, third, amount) + type(item), allocatable, intent(inout) :: first + type(item), allocatable, target, intent(inout) :: second + type(item), pointer, intent(inout) :: third + integer(c_int32_t), intent(in) :: amount + if (.not. allocated(first)) allocate(first) + if (.not. allocated(second)) allocate(second) + first%value = first%value + amount + second%value = second%value + amount + third => pointer_target_two + end subroutine mutate_three_descriptors + + function read_duplicate(first, second) result(total) + type(item), intent(in) :: first + type(item), intent(in) :: second + integer(c_int32_t) :: total + total = first%value + second%value + end function read_duplicate + + function read_optional(first, second) result(total) + type(item), intent(in), optional :: first + type(item), intent(in), optional :: second + integer(c_int32_t) :: total + total = 0_c_int32_t + if (present(first)) total = total + first%value + if (present(second)) total = total + second%value + end function read_optional + + subroutine mutate_duplicate(first, second) + type(item), intent(inout) :: first + type(item), intent(inout) :: second + writable_call_count = writable_call_count + 1_c_int32_t + first%value = first%value + 1_c_int32_t + second%value = second%value + 1_c_int32_t + end subroutine mutate_duplicate + + subroutine hold_allocatable(value, milliseconds) + type(item), allocatable, intent(inout) :: value + integer(c_int32_t), intent(in) :: milliseconds + integer(c_int) :: start_count, current_count, count_rate + call system_clock(start_count, count_rate) + do + call system_clock(current_count) + if ((current_count - start_count) * 1000_c_int / count_rate >= milliseconds) exit + end do + if (allocated(value)) value%value = value%value + 1_c_int32_t + end subroutine hold_allocatable + + subroutine hold_object(value, milliseconds) + type(item), intent(inout) :: value + integer(c_int32_t), intent(in) :: milliseconds + integer(c_int) :: start_count, current_count, count_rate + call system_clock(start_count, count_rate) + do + call system_clock(current_count) + if ((current_count - start_count) * 1000_c_int / count_rate >= milliseconds) exit + end do + value%value = value%value + 1_c_int32_t + end subroutine hold_object + +end module fscalar_derived_actual_dummy_matrix_f90 diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index 4e15b95c9..a2c6f0d03 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -36,12 +36,14 @@ C_DOCS_DISABLED = " -### Codegen Class Organization +### Wrapper Generator Class Organization @@ -281,7 +281,7 @@ User-visible `.pyi` syntax is first parsed to Python AST by `x2py/pyi_parser/parser.py`, loaded from text/files by `x2py/pipeline/pyi.py`, converted to semantic IR by `x2py/semantics/pyi2ir.py`, and printed by -`x2py/codegen/printers/pyi_printer.py`. The converter and printer operate on +`x2py/wrapper_codegen/printers/pyi_printer.py`. The converter and printer operate on `x2py/semantics/models.py`. Important implementation rules: @@ -304,7 +304,7 @@ Important implementation rules: When changing `.pyi` syntax: 1. Add or update parser tests in `tests/parsing/pyi/`. -2. Add or update printer tests in `tests/codegen/printers/`. +2. Add or update printer tests in `tests/wrapper_codegen/printers/`. 3. Update fixture tests only if the public generated contract changes. 4. Update [Basic wrapper tutorial](../user/tutorials/basic-wrapper.md) or [Verified examples cookbook](../user/examples/verified-cookbook.md) if users need to write or read the new syntax. @@ -828,8 +828,8 @@ ordered source paths -> compile-time expression and storage probes -> fortran_project_to_semantic_modules(...) -> merge public semantic modules - -> semantic_ir_to_codegen_ast(...) - -> Codegen and create_shared_library(...) + -> WrapperPlanner and WrapperCodeGenerator + -> create_shared_library(...) -> WrapperBuildResult ``` @@ -838,16 +838,18 @@ The main ownership boundaries are: - `x2py/pipeline/build.py`: source order, preprocessing/probing, semantic merge, `.pyi` entry-contract loading, native build plan assembly, output placement, direct-versus-Makefile mode, and artifact reporting; -- `x2py/semantics/ir2ast.py`: semantic contract validation and conversion to - codegen models; +- `x2py/wrapper_codegen/planner.py`: projection from completed semantic policy + into validated typed plans; +- `x2py/wrapper_codegen/generator.py`: direct bridge, binding, and source + artifact generation; - `x2py/compiling/`: compiler commands and shared-library linking; and - `x2py/stdlib/x2py_runtime/`: native runtime support copied into each build. Do not move semantic ownership or projection policy into printers. Do not infer @@ -926,7 +928,7 @@ X2PY_C_DOCS_END --> - `x2py/semantics/fortran2ir.py` maps Fortran procedures, derived types, module variables, kinds, shapes, storage contracts, visibility, imported references, and compile-time values. -- `x2py/codegen/printers/pyi_printer.py` emits editable user contracts. +- `x2py/wrapper_codegen/printers/pyi_printer.py` emits editable user contracts. - `x2py/pyi_parser/parser.py` parses edited contracts to Python AST. - `x2py/pipeline/pyi.py` converts edited contract text, files, and path sets. - `x2py/semantics/pyi2ir.py` converts parsed `.pyi` AST back into semantic IR. @@ -979,7 +981,7 @@ status-return policy, ownership conversion, or coercion execution. The test ownership is: - loader syntax and error behavior: `tests/parsing/pyi/`; -- printer round-trip shape: `tests/codegen/printers/`; +- printer round-trip shape: `tests/wrapper_codegen/printers/`; - readiness interpretation: `tests/semantics/readiness/`. 3. Keep the public semantic dtype names in `x2py/semantics/models.py` stable unless there is a deliberate schema decision. 4. If the emitted `.pyi` annotation changes, update - `tests/codegen/printers/` and `tests/parsing/pyi/`. + `tests/wrapper_codegen/printers/` and `tests/parsing/pyi/`. 5. Update the datatype tables in [Semantic IR reference](../user/reference/semantic-ir.md), and update [Basic wrapper tutorial](../user/tutorials/basic-wrapper.md) or [Verified examples cookbook](../user/examples/verified-cookbook.md) when a visible example changes. @@ -1199,13 +1201,13 @@ Focused verification: ```bash PYTHONPATH=. pytest -q tests/semantics/conversion/fortran/ -PYTHONPATH=. pytest -q tests/codegen/printers/ tests/parsing/pyi/ +PYTHONPATH=. pytest -q tests/wrapper_codegen/printers/ tests/parsing/pyi/ ``` @@ -1218,8 +1220,8 @@ Example target: add a new `Annotated[...]` metadata item or projection helper. when loading or cross-file reconciliation changes. Update `x2py/pyi_parser/parser.py` only when the raw Python AST parsing boundary changes. -3. Add printer tests in `tests/codegen/printers/`. -4. Update `x2py/codegen/printers/pyi_printer.py`. +3. Add printer tests in `tests/wrapper_codegen/printers/`. +4. Update `x2py/wrapper_codegen/printers/pyi_printer.py`. 5. Update semantic models in `x2py/semantics/models.py` only if the IR needs a new field or constraint. 6. Update readiness behavior if the new syntax resolves a blocker. @@ -1230,7 +1232,7 @@ Focused verification: ```bash PYTHONPATH=. pytest -q tests/parsing/pyi/ -PYTHONPATH=. pytest -q tests/codegen/printers/ +PYTHONPATH=. pytest -q tests/wrapper_codegen/printers/ PYTHONPATH=. pytest -q tests/semantics/readiness/ ``` @@ -1506,7 +1508,7 @@ Focused tests by concern: - Semantic readiness: `PYTHONPATH=. pytest -q tests/semantics/readiness/` - `.pyi` printer: - `PYTHONPATH=. pytest -q tests/codegen/printers/` + `PYTHONPATH=. pytest -q tests/wrapper_codegen/printers/` - `.pyi` loader and edited stub behavior: `PYTHONPATH=. pytest -q tests/parsing/pyi/` - Semantic and `.pyi` fixtures: @@ -1528,7 +1530,7 @@ python tests/pyi/generate_pyi_fixtures.py ``` Executable examples: `tests/semantics/readiness/`, -`tests/codegen/printers/`, and `tests/parsing/pyi/`. +`tests/wrapper_codegen/printers/`, and `tests/parsing/pyi/`. ### CLI diff --git a/docs/developer/feature-to-code-map.md b/docs/developer/feature-to-code-map.md index 19541a068..26678aacf 100644 --- a/docs/developer/feature-to-code-map.md +++ b/docs/developer/feature-to-code-map.md @@ -17,11 +17,11 @@ before documentation may call the behavior supported. | Feature or behavior | Public docs | Main implementation files | Focused tests | Support evidence | | --- | --- | --- | --- | --- | | Fortran parse output | `docs/developer/fortran-parser-reference.md` | `x2py/fortran_parser/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/parsing/fortran/`, `tests/parsing/fortran/test_fortran_fixture_suite.py` | Parser facts and diagnostics match fixtures | -| Semantic `.pyi` generation | `docs/user/reference/semantic-pyi-format.md` | `x2py/codegen/printers/pyi_printer.py` | `tests/codegen/printers/`, `tests/codegen/printers/test_modern_example.py` | Printed `.pyi` round-trips or matches fixtures | +| Semantic `.pyi` generation | `docs/user/reference/semantic-pyi-format.md` | `x2py/wrapper_codegen/printers/pyi_printer.py` | `tests/wrapper_codegen/printers/`, `tests/wrapper_codegen/printers/test_modern_example.py` | Printed `.pyi` round-trips or matches fixtures | | Semantic `.pyi` conversion and editing | `docs/user/guide/editing-semantic-pyi-contracts.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `models.py` | `tests/parsing/pyi/`, `tests/pipeline/pyi_builds/test_contract_fixtures.py` | Edited contracts parse to Python AST, then become semantic IR with preserved native facts | | Readiness blockers | `docs/user/reference/diagnostic-codes.md`, `docs/user/reference/semantic-pyi-format.md` | `x2py/semantics/readiness.py` | `tests/semantics/readiness/`, readiness fixture tests | Unsupported or incomplete contracts fail before codegen | | Fortran wrapper orchestration | `docs/user/guide/fortran-wrapper.md`, `docs/user/examples/recipes/build-and-import-cli.md`, `docs/user/examples/recipes/build-multiple-fortran-sources.md` | `x2py/pipeline/build.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, multi-source wrapper tests | Builds report artifacts and compile/link as documented | -| Semantic IR to codegen AST | `docs/user/guide/fortran-wrapper.md` | `x2py/semantics/ir2ast.py`, `x2py/semantics/ownership.py` | `tests/lowering/test_semantic_ir.py`, `tests/wrapper/fortran/` | Runtime policy is explicit and unsupported cases block | +| Completed semantic policy to wrapper artifacts | `docs/user/guide/fortran-wrapper.md` | `x2py/semantics/policy_completion.py`, `x2py/wrapper_codegen/plan.py`, `planner.py`, `generator.py` | `tests/semantics/policy/`, `tests/wrapper_codegen/`, `tests/wrapper/fortran/` | Runtime policy is explicit, the typed plan is complete, and generated artifacts compile and run | | Native compilation and runtime support | `docs/user/guide/fortran-wrapper.md`, `docs/user/examples/recipes/generate-editable-makefile.md`, `docs/developer/build-system.md`, `docs/developer/quality-assurance.md` | `x2py/compiling/`, `x2py/stdlib/x2py_runtime/` | `tests/wrapper/fortran/build_from_source/test_runtime_abi.py`, build-mode tests | Generated sources compile, link, import, and clean up correctly | | Source documentation structure | `docs/developer/source-map.md` | `docs/`, package README files, `tests/docs/test_structure.py` | documentation structure and example tests | Pages have metadata, audience separation, and source coverage checks | @@ -30,8 +30,8 @@ before documentation may call the behavior supported. | Compiler preprocessing | `docs/user/examples/recipes/compiler-preprocessing.md`, parser references | `x2py/pipeline/preprocessing.py`, parser CLI helpers | `tests/pipeline/preprocessing/`, `tests/pipeline/preprocessing/test_parser_boundaries.py`, C preprocessing tests | Preprocessed input and dependency facts are stable | | C parse output | `docs/developer/c-parser-reference.md`, `docs/user/examples/recipes/inspect-c-api.md` | `x2py/c_parser/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/parsing/c/test_c_declarations_and_declarators.py`, `tests/parsing/c/test_c_fixture_suite.py` | Parser facts and diagnostics match fixtures | | Semantic IR | `docs/user/reference/semantic-ir.md` | `x2py/semantics/models.py`, `fortran2ir.py`, `c2ir.py` | `tests/semantics/conversion/fortran/`, `tests/semantics/conversion/c/` | Source facts lower without losing wrapper-relevant meaning | -| Generated Fortran bridge | `docs/user/guide/fortran-wrapper.md` | `x2py/codegen/bridges/fortran_to_c.py`, `x2py/codegen/printers/fcode.py` | `tests/wrapper/fortran/` | Generated bridge compiles and preserves native calling contract | -| Generated CPython binding | `docs/user/guide/fortran-wrapper.md` | `x2py/codegen/bindings/c_to_python.py`, CPython and NumPy binding helpers | `tests/wrapper/fortran/` | Extension imports, validates Python inputs, and returns documented values | +| Generated Fortran bridge | `docs/user/guide/fortran-wrapper.md` | `x2py/wrapper_codegen/fortran/bridge.py`, `x2py/wrapper_codegen/printers/source_printers.py` | `tests/wrapper_codegen/`, `tests/wrapper/fortran/` | Generated bridge compiles and preserves native calling contract | +| Generated CPython binding | `docs/user/guide/fortran-wrapper.md` | `x2py/wrapper_codegen/c/binding.py`, `x2py/wrapper_codegen/printers/source_printers.py` | `tests/wrapper_codegen/`, `tests/wrapper/fortran/` | Extension imports, validates Python inputs, and returns documented values | | Public API exports | `README.md`, `docs/user/reference/python-api.md` | `x2py/__init__.py` | `tests/parsing/fortran/test_public_entrypoints.py`, C public API tests | Import paths are intentional and documented | X2PY_C_DOCS_END --> @@ -41,9 +41,9 @@ X2PY_C_DOCS_END --> For a feature change, start with the implementation file named in the feature map and read only the downstream files that the change actually crosses. For example, a CLI output change normally starts and ends in `x2py/cli.py`, while a -wrapper output-projection change must move through -`x2py/semantics/ir2ast.py`, `x2py/semantics/ownership.py`, the bridge generator, -and the CPython binding generator. +wrapper output-projection change must move through semantic policy completion, +the typed wrapper planner, and the selected bridge and binding implementation +methods. X2PY_C_DOCS_END --> When the user-visible behavior changes, update the public docs in the same row @@ -54,9 +54,9 @@ this routing page tied to the source hotspots and package README files. | User workflow | Start in code | Do not mark supported until | | --- | --- | --- | -| Wrapping functions and subroutines | `x2py/semantics/fortran2ir.py`, `x2py/semantics/ir2ast.py`, bridge and binding generators | Runtime tests compile, import, call, and verify return and failure behavior | +| Wrapping functions and subroutines | `x2py/semantics/fortran2ir.py`, policy completion, `x2py/wrapper_codegen/planner.py`, bridge and binding generators | Runtime tests compile, import, call, and verify return and failure behavior | | Wrapping modules and module variables | parser module facts, semantic module conversion, naming policy, wrapper generators | Python-visible names, accessors, and unsupported module constructs are tested | -| Arrays and allocatables | semantic array contracts, `ir2ast`, ownership policy, bridge/binding array handlers | dtype, shape, rank, contiguity, mutation, returned arrays, and failure paths are tested | +| Arrays and allocatables | semantic array contracts, ownership policy, typed wrapper plans, bridge/binding array handlers | dtype, shape, rank, contiguity, mutation, returned arrays, and failure paths are tested | | Pointer arguments | semantic metadata, ownership policy, bridge/binding pointer handlers | Owner, lifetime, association, and blocked cases are explicit and tested | | Optional arguments | parser optional attributes, semantic arguments, binding argument parsing | Present/absent calls and unsupported combinations are tested | | Generic interfaces | parser interface facts, semantic overload sets, `FunctionOverloadSet`, binding dispatch | Overload selection and ambiguity failures are tested at runtime | diff --git a/docs/developer/repository-structure.md b/docs/developer/repository-structure.md index f46eb4156..ad8dbbebf 100644 --- a/docs/developer/repository-structure.md +++ b/docs/developer/repository-structure.md @@ -22,7 +22,8 @@ artifacts used by tests. Navigate by ownership boundary first, then by file. | `x2py/runtime/` | Python runtime objects used by generated extension modules. | | `x2py/types/` | Cross-layer mappings from resolved semantic types to Python ecosystem types. | | `x2py/fortran_parser/` | Fortran parser frontend and Fortran parse report helpers. | -| `x2py/semantics/` | Semantic IR, source-to-IR conversion, `.pyi` parsing, readiness, and codegen lowering. | +| `x2py/semantics/` | Semantic IR, source-to-IR conversion, `.pyi` parsing, policy completion, and readiness. | +| `x2py/wrapper_codegen/` | Typed wrapper plans, direct native bridge/binding lowering, and source and semantic `.pyi` printers. | | `x2py/compiling/` | Native compile objects, compiler command orchestration, runtime support installation, and linking. | | `x2py/stdlib/` | Native runtime support copied into generated wrapper builds. | | `x2py/naming/` | Unified public-name and generated-symbol policy. | @@ -30,7 +31,6 @@ artifacts used by tests. Navigate by ownership boundary first, then by file. The major source packages have local README files under `x2py/` for @@ -48,11 +48,13 @@ contract syntax. | Path | Purpose | | --- | --- | -| `tests/parser/` | Parser, preprocessing, CLI, and parser fixture tests. | -| `tests/semantics/` | Semantic IR, readiness, type mapping, and lowering tests. | -| `tests/pyi/` | Semantic `.pyi` parser and fixture tests. | +| `tests/parsing/` | Parser and parser fixture tests grouped by source language. | +| `tests/pipeline/` | Preprocessing and semantic `.pyi` build-orchestration tests. | +| `tests/semantics/` | Semantic conversion, completed policy, and readiness tests. | +| `tests/wrapper_codegen/` | Typed planning, direct bridge/binding generation, and source-printer tests. | | `tests/wrapper/fortran/` | Runtime wrapper tests that compile, import, call, and check failure paths. | -| `tests/tools/` | Tooling tests, including documentation example and structure checks. | +| `tests/docs/` | Documentation example and structure checks. | +| `tests/tools/` | Repository tooling tests. | ## Package Map @@ -78,8 +78,8 @@ X2PY_C_DOCS_END --> | `x2py/probes/c_types.py` | Compiler-derived target ABI facts for C inspection workflows | `c_types.py` | C target probe tests | | `x2py/c_parser/` | C lexer, parser, models, preprocessing metadata, and C parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `preprocessor.py`, `type_resolver.py`, `cli.py` | `tests/parser/c/`, `docs/developer/c-parser-reference.md` | | `x2py/pyi_parser/` | Semantic `.pyi` text/file parsing to Python AST. | `parser.py` | `tests/parsing/pyi/`, `docs/user/reference/semantic-pyi-format.md` | -| `x2py/semantics/` | Language-neutral semantic IR, source-to-IR conversion, `.pyi` AST conversion, policy completion, readiness, and codegen lowering | `models.py`, `fortran2ir.py`, `c2ir.py`, `pyi2ir.py`, `policy_completion.py`, `readiness.py`, `ir2ast.py` | `tests/semantics/`, `tests/pyi/`, `docs/user/reference/semantic-ir.md`, `docs/user/reference/semantic-pyi-format.md` | -| `x2py/codegen/` | Codegen AST, bridge and binding generation, printers, and semantic `.pyi` printing | `models/`, `bridges/fortran_to_c.py`, `bindings/c_to_python.py`, `printers/`, `binding_pipeline.py` | `tests/wrapper/`, `tests/codegen/printers/`, `docs/user/guide/fortran-wrapper.md` | +| `x2py/semantics/` | Language-neutral semantic IR, source-to-IR conversion, `.pyi` AST conversion, policy completion, and readiness | `models.py`, `fortran2ir.py`, `c2ir.py`, `pyi2ir.py`, `policy_completion.py`, `readiness.py` | `tests/semantics/`, `tests/pyi/`, `docs/user/reference/semantic-ir.md`, `docs/user/reference/semantic-pyi-format.md` | +| `x2py/wrapper_codegen/` | Canonical wrapper planning, C/Fortran generation, source printing, and semantic `.pyi` printing | `plan.py`, `planner.py`, `generator.py`, `printers/` | `tests/wrapper_codegen/`, `tests/wrapper/`, `docs/user/guide/fortran-wrapper.md` | | `x2py/naming/` | Unified public-name and generated-symbol policy for Python, C, and Fortran targets | `policy.py` | naming, visibility, and wrapper runtime tests | X2PY_C_DOCS_END --> @@ -105,15 +105,17 @@ update this table, the package README files, and the mechanical checks in | `x2py/pyi_parser/parser.py` | Minimal `.pyi` text/file parsing to Python AST. | | `x2py/pipeline/pyi.py` | Semantic `.pyi` text/file/path-set conversion and external-type reconciliation. | | `x2py/semantics/pyi2ir.py` | Semantic `.pyi` AST conversion and validation. | -| `x2py/semantics/policy_completion.py` | Post-IR semantic policy completion before readiness and lowering. | +| `x2py/semantics/policy_completion.py` | Post-IR semantic policy completion before readiness and wrapper planning. | | `x2py/semantics/readiness.py` | Support blockers and readiness reporting. | -| `x2py/semantics/ir2ast.py` | Semantic IR to codegen AST lowering. | -| `x2py/codegen/binding_pipeline.py` | Ordered bridge and binding generation. | -| `x2py/codegen/printers/fcode.py` | Fortran source printing. | -| `x2py/codegen/printers/pyi_printer.py` | Semantic `.pyi` printing. | +| `x2py/wrapper_codegen/plan.py` | Typed, policy-complete wrapper plan records. | +| `x2py/wrapper_codegen/planner.py` | Semantic policy to wrapper-plan conversion. | +| `x2py/wrapper_codegen/generator.py` | Ordered direct bridge, binding, header, and source generation. | +| `x2py/wrapper_codegen/fortran/bridge.py` | Direct Fortran bridge lowering from typed plans. | +| `x2py/wrapper_codegen/c/binding.py` | Direct Python-extension binding lowering from typed plans. | +| `x2py/wrapper_codegen/printers/source_printers.py` | Native binding, header, and Fortran source printing. | +| `x2py/wrapper_codegen/printers/pyi_printer.py` | Semantic `.pyi` printing. | | `x2py/compiling/basic.py` | Native compile object model. | | `x2py/compiling/compilers.py` | Compiler command execution and tool lookup. | -| `x2py/compiling/python_wrapper.py` | Generated wrapper compilation and shared-library linking. | | `x2py/compiling/runtime_support.py` | Runtime support installation for generated wrappers. | | `x2py/naming/policy.py` | Public wrapper names and generated target-language symbols. | | `x2py/stdlib/` | Runtime support payload copied into generated builds. | @@ -123,12 +125,6 @@ update this table, the package README files, and the mechanical checks in | `x2py/c_parser/parser.py` | C parser project model and diagnostics. | | `x2py/c_parser/cli.py` | C parser report formatting and preprocessing integration. | | `x2py/semantics/c2ir.py` | C parser facts to semantic modules. | -| `x2py/codegen/bridges/fortran_to_c.py` | Fortran bind(C) bridge generation. | -| `x2py/codegen/bindings/c_to_python.py` | CPython extension binding generation. | -| `x2py/codegen/bindings/cpython_api.py` | CPython C API helper nodes. | -| `x2py/codegen/bindings/numpy_cpython_api.py` | NumPy C API helper nodes. | -| `x2py/codegen/printers/ccode.py` | C source printing. | -| `x2py/codegen/printers/cpythoncode.py` | CPython C source printing. | X2PY_C_DOCS_END --> ## Layer-To-Layer Route @@ -145,10 +141,11 @@ x2py/cli.py -> x2py/semantics/fortran2ir.py -> x2py/semantics/policy_completion.py -> x2py/semantics/readiness.py - -> x2py/semantics/ir2ast.py - -> x2py/codegen/bridges/fortran_to_c.py - -> x2py/codegen/bindings/c_to_python.py - -> x2py/compiling/python_wrapper.py + -> x2py/wrapper_codegen/planner.py + -> x2py/wrapper_codegen/generator.py + -> x2py/wrapper_codegen/fortran/bridge.py + -> x2py/wrapper_codegen/c/binding.py + -> x2py/compiling/compilers.py -> tests/wrapper/fortran/ ``` X2PY_C_DOCS_END --> @@ -161,7 +158,8 @@ x2py/pyi_parser/parser.py -> x2py/semantics/pyi2ir.py -> x2py/semantics/policy_completion.py -> x2py/semantics/readiness.py - -> x2py/semantics/ir2ast.py + -> x2py/wrapper_codegen/planner.py + -> x2py/wrapper_codegen/generator.py ``` Verbose wrapper builds should print the exact compiler command lines they run, diff --git a/docs/maintainer/internal-architecture/pipeline-map.md b/docs/maintainer/internal-architecture/pipeline-map.md index 8a70b49bd..94ad3d729 100644 --- a/docs/maintainer/internal-architecture/pipeline-map.md +++ b/docs/maintainer/internal-architecture/pipeline-map.md @@ -26,7 +26,7 @@ CLI request -> semantic policy completion -> ownership/transfer/destruction policy completion -> readiness blockers - -> codegen AST + -> typed wrapper plan -> generated Fortran bind(C) bridge -> generated C/CPython binding -> native compile, runtime support install, and link @@ -43,15 +43,14 @@ X2PY_C_DOCS_END --> | Parser project model | `x2py/fortran_parser/parser.py` | preprocessed Fortran source | parser project with modules, procedures, types, visibility | Fortran parser fixture tests | | Target probes | `x2py/probes/fortran_types.py` | semantic type requirements and compiler flags | resolved kind/storage facts | Fortran type probe tests | | Semantic IR | `x2py/semantics/fortran2ir.py` | parser project and target facts | `SemanticModule` objects | semantic Fortran tests | -| Semantic policy completion | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py` | full semantic modules with signatures and `.pyi` overrides | semantic modules annotated with completed ownership, transfer, and destruction decisions | ownership-policy, readiness, and lowering tests | +| Semantic policy completion | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py` | full semantic modules with signatures and `.pyi` overrides | semantic modules annotated with every ownership, transfer, destruction, mutability, storage, accessor, and projection decision needed by wrapper generation | ownership-policy and readiness tests | | Readiness | `x2py/semantics/readiness.py` | prepared semantic modules | blockers and support status | readiness tests and fixtures | -| Codegen lowering | `x2py/semantics/ir2ast.py` | policy-completed semantic modules | codegen AST consuming completed policy decisions | `tests/lowering/test_semantic_ir.py`, wrapper tests | -| Printing | `x2py/codegen/printers/` | generated ASTs | wrapper source files | generated build artifacts and wrapper tests | +| Wrapper planning | `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/plan.py` | policy-completed semantic modules | typed wrapper plans consuming completed decisions without re-inferring policy | `tests/wrapper_codegen/`, wrapper tests | +| Direct bridge and binding lowering | `x2py/wrapper_codegen/fortran/bridge.py`, `x2py/wrapper_codegen/c/binding.py`, `x2py/wrapper_codegen/generator.py` | validated typed wrapper plans | Fortran, C, and header syntax nodes | `tests/wrapper_codegen/`, wrapper tests | +| Wrapper and semantic-contract printing | `x2py/wrapper_codegen/printers/` | wrapper syntax nodes or semantic IR | wrapper source files or semantic `.pyi` text | printer, generated-contract, and wrapper artifact tests | | Compile and link | `x2py/compiling/` | user objects, wrapper sources, runtime support | shared library | wrapper runtime tests | ## Concept Ownership Rules @@ -79,12 +78,12 @@ cross-cutting infrastructure. | --- | --- | --- | --- | | Parser facts | parser packages | Source syntax, native declaration structure, source locations, and parser diagnostics | Wrapper policy, Python API projection, generated names, and compile/link decisions | | Readiness and ownership policy | `x2py/semantics/readiness.py`, `x2py/semantics/policy_completion.py`, and `x2py/semantics/ownership.py` | Semantic policy completion, support blockers, and policy choices for ownership, lifetime, output projection, replacement, and ABI safety | Raw parser syntax, backend-specific statement trees, and hidden lowering-time policy decisions | -| Core codegen AST | `x2py/codegen/models/` and `x2py/semantics/ir2ast.py` outputs | The implementation plan after a semantic contract is accepted: generated functions, variables as storage locations, statements, expressions, control flow, temporaries, scopes, and imports/includes | Source-contract authority, `.pyi` persistence, and readiness-only facts | -| Printers and compilation | `x2py/codegen/printers/`, `x2py/compiling/`, and wrapper orchestration | Text emission, generated artifact layout, compiler commands, native objects, libraries, include directories, and link inputs | Semantic support decisions and generated-AST rewriting policy | +| Typed wrapper plan | `x2py/wrapper_codegen/plan.py` and `x2py/wrapper_codegen/planner.py` | A validated, backend-neutral implementation plan projected from completed semantic decisions | Source-contract authority, policy inference, and target-language statement details | +| Printers and compilation | `x2py/wrapper_codegen/printers/`, `x2py/compiling/`, and wrapper orchestration | Text emission, generated artifact layout, compiler commands, native objects, libraries, include directories, and link inputs | Semantic support decisions and plan rewriting policy | @@ -97,14 +96,14 @@ Use these rules when adding a new notion: than a source fact: for example borrowed versus copied data, visible versus hidden native outputs, replacement rules, destructor ownership, or unsupported ABI combinations. If the decision depends on full signature context, complete - it in `policy_completion.py` before readiness or `ir2ast.py`. + it in `policy_completion.py` before readiness or wrapper planning. - Put it in compiling or wrapping when it describes build inputs or build execution: sources, objects, libraries, library directories, include directories, compiler flags, link items, runtime support files, and generated artifact paths. | --- | --- | --- | | CLI and output routing | `x2py/cli.py`, parser CLI helpers | `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md` | | Source loading and preprocessing | `x2py/pipeline/preprocessing.py` | `docs/developer/source-map.md`, parser references | -| Editable semantic contracts | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | +| Editable semantic contracts | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | | Readiness | `x2py/semantics/readiness.py` | `docs/user/reference/diagnostic-codes.md` | -| Wrapper policy and lowering | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py`, `x2py/semantics/ir2ast.py` | `docs/user/guide/fortran-wrapper.md`, ownership docs | -| Native build | `x2py/compiling/python_wrapper.py`, `x2py/compiling/runtime_support.py` | compiling package README and build-system docs | +| Wrapper policy and lowering | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py`, `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/generator.py` | `docs/user/guide/fortran-wrapper.md`, ownership docs | +| Native build | `x2py/pipeline/build.py`, `x2py/compiling/compilers.py`, `x2py/compiling/runtime_support.py` | compiling package README and build-system docs | ## Semantic `.pyi` Wrapper Pipeline @@ -183,15 +182,16 @@ the Python API. -> x2py/semantics/native_contract.py -> x2py/semantics/policy_completion.py -> x2py/semantics/readiness.py - -> x2py/semantics/ir2ast.py - -> bridge, binding, compile, and link pipeline + -> x2py/wrapper_codegen/planner.py + -> x2py/wrapper_codegen/generator.py + -> compile and link pipeline ``` The `.pyi` path must preserve native ABI facts in the semantic contract. Missing native build inputs or contradictory contract facts fail before bridge emission or native compilation. Ownership, transfer, and destruction policy is completed -from the full `.pyi` signature before lowering; `ir2ast.py` consumes that -completed policy and must not invent a different one. +from the full `.pyi` signature before planning; the wrapper planner and backend +generators consume that completed policy and must not invent a different one. ## Shared Semantic Policy Boundary diff --git a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md index edfa1269e..114999d3c 100644 --- a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md +++ b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md @@ -8,15 +8,14 @@ status: maintained # Wrapper Generation Pipeline -This page describes the direct wrapper-plan route through Phase 6. It covers -primitive scalars, scalar strings, ordinary arrays, and the supported -module-variable surface. Native allocatable and pointer handles belong to Phase -7 and are intentionally outside this contract. +This page describes the canonical wrapper-plan generation route. It covers the +completed scalar, string, array, native-handle, derived-type, class, callback, +module-state, generic, and build surfaces. ## Architectural Boundary -All semantic policy must be complete before wrapper planning or -`x2py/semantics/ir2ast.py` lowering begins. Post-IR policy completion owns +All semantic policy must be complete before wrapper planning begins. Post-IR +policy completion owns object kind, ownership, transfer, destruction, mutability, writeback, nullability, output projection, release responsibility, storage mode, getter behavior, native setter assignment, and Python setter exposure. @@ -39,11 +38,12 @@ artifacts = WrapperCodeGenerator().generate(plan) runs both backend preflight checks, lowers recursively to C and Fortran syntax nodes, and asks the source printers to render those nodes. Build integration compiles the rendered sources; it does not own datatype transfer policy. +Wrapper C/Fortran source printers and the semantic `.pyi` printer share +`x2py/wrapper_codegen/printers/`; no compatibility printer remains under the +legacy codegen package. -During the migration, route selection may still choose the legacy generators -for unsupported functions. The legacy mappings in `x2py/codegen/` consume the -same completed semantic action enums, but they are not dependencies of -`x2py/wrapper_codegen/`. They remain only until the final route cutover. +Wrapper builds have no legacy route or fallback. An unsupported completed plan +fails with its exact owner path before either backend emits source. ## Stable Tree and Datatype-Varying Records @@ -80,6 +80,13 @@ value. They own export names, call order, result order, runtime/GIL envelopes, and aggregation, but not datatype policy. +Python-facing documentation is also a plan projection. The shared docstring +builder consumes completed namespace, module-variable, class, overload, +argument, result, and lifecycle records and stores the rendered text on the +owning plan nodes. C method-table emission and generated Python class assembly +only attach that text; neither backend infers signatures, ownership, mutation, +nullability, or exception behavior while rendering source. + `NativeCallSlotPlan` and `LifecycleActionPlan` are subordinate transfer details. Native slots stay indexed on `FunctionPlan` because native ABI order can interleave argument slots, result slots, literals, and helpers. Lifecycle @@ -153,11 +160,12 @@ whether the transfer itself is a scalar, string, or array. Inspect the real records directly with normal Python prints. The primary path is `complete_semantic_policies()` -> `WrapperPlanner.build()` -> `WrapperCodeGenerator.generate()`. Generated artifacts from real passing -`tests/wrapper` cases are the behavioral oracle; plan unit tests cover selector -and graph invariants, while dual-route runtime tests prove legacy/direct-plan -parity. +`tests/wrapper` cases are the behavioral oracle; plan unit tests cover action +and graph invariants. Production source and semantic-`.pyi` builds both use +this one path; unsupported completed policy is an error before lowering, not a +request to retry a legacy generator. -A Phase 5 or Phase 6 change is acceptable when: +A wrapper-generation change is acceptable when: - semantic decisions are complete before planning; - datatype variation is confined to transfer, result, lifecycle, or diff --git a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md b/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md index 574eba301..1558a0364 100644 --- a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md +++ b/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md @@ -81,7 +81,7 @@ projection language. Python-extension extraction and native handoff are separate completed policy actions. Policy completion records a Python barrier action and a native barrier action -on each `OwnershipDecision` before `ir2ast.py`. Python binding generation +on each `OwnershipDecision` before wrapper planning. Python binding generation dispatches from the Python barrier action; the native bridge dispatches argument handoff from the native barrier action. @@ -99,7 +99,7 @@ argument handoff from the native barrier action. `x2py.utilities.visitor.ClassVisitor` and configured `_` handlers instead of parallel visitor implementations or local `isinstance` dispatch ladders. -- [x] Structural evidence lives in `tests/lowering/test_visitor_protocol.py` +- [x] Structural evidence lives in `tests/architecture/test_visitor_protocol.py` and `tests/semantics/policy/`; runtime evidence covers scalar value/address projection, rank-0 scalar storage, arrays, strings, raw addresses, and wrapper instances through focused `tests/wrapper/fortran/` @@ -404,13 +404,13 @@ X2PY_C_DOCS_END --> dummies update the supplied wrapper object. Runtime evidence lives in `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py`. - [x] Ownership, transfer, and destruction policy is completed after full - signatures are known and before readiness or `ir2ast.py`. The shared post-IR + signatures are known and before readiness or wrapper planning. The shared post-IR entrypoint is `complete_semantic_policies(...)` in `x2py/semantics/policy_completion.py`; direct ownership subpasses stay behind that entrypoint. Readiness and lowering consume completed policy metadata instead of recomputing policy from raw datatypes. Evidence: `tests/semantics/policy/`, - `tests/lowering/test_semantic_ir.py`, + `tests/wrapper_codegen/`, `tests/semantics/readiness/`, and `x2py/semantics/README.md`. - [x] `.pyi` parsing and `.pyi` semantic conversion are separate stages: @@ -519,36 +519,13 @@ X2PY_C_DOCS_END --> `tests/wrapper/CHECKLIST_COVERAGE.md`. bridge and binding code are local emitted-code, ABI, documentation, or object-model mechanics rather than semantic policy selection. Evidence: `x2py/semantics/ownership.py`, - `x2py/codegen/bridges/fortran_to_c.py`, - `x2py/codegen/bindings/c_to_python.py`, + `x2py/wrapper_codegen/fortran/bridge.py`, + `x2py/wrapper_codegen/c/binding.py`, `tests/semantics/policy/`, + `tests/wrapper_codegen/`, `tests/wrapper/fortran/derived_types/test_derived_layout.py`, and `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py`. X2PY_C_DOCS_END --> diff --git a/docs/maintainer/roadmap/test-suite-organization-checklist.md b/docs/maintainer/roadmap/test-suite-organization-checklist.md index 5a3771ca5..28ab807e8 100644 --- a/docs/maintainer/roadmap/test-suite-organization-checklist.md +++ b/docs/maintainer/roadmap/test-suite-organization-checklist.md @@ -98,7 +98,7 @@ all original cases are accounted for in its destination modules. | `tests/semantics/test_ownership_policy.py` | completed decisions in `tests/semantics/policy/`; generator dispatch cases in `tests/codegen/bridges/` and `tests/codegen/bindings/` | | `tests/semantics/test_c_semantic_readiness.py`, `test_semantic_wrap_readiness.py`, `test_wrap_readiness_fixture_suite.py` | `tests/semantics/readiness/`, with the oversized module split by readiness boundary | | `tests/semantics/test_ir2ast.py`, `test_visitor_protocol.py` | `tests/lowering/` | -| `tests/semantics/test_pyi_printer*.py` | `tests/codegen/printers/`, with the oversized printer module split by emitted concept | +| `tests/semantics/test_pyi_printer*.py` | `tests/wrapper_codegen/printers/`, with the oversized printer module split by emitted concept | | `tests/test_runtime_handles.py` | split under `tests/runtime/handles/` | | `tests/test_naming_policy.py` | `tests/naming/test_policy.py` | | `tests/tools/test_documentation_examples.py`, `test_documentation_structure.py` | `tests/docs/` | @@ -122,7 +122,7 @@ trees are not part of this map and must not move. | `tests/semantics/test_ownership_policy.py` | `tests/semantics/policy/test_accessor_and_storage_policy.py`, `test_native_array_ownership.py`, `test_policy_defaults_and_validation.py`; `tests/lowering/test_array_interop_policy.py`; `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py`; `tests/codegen/bindings/test_binding_handle_policy_dispatch.py` | | `tests/semantics/test_fortran2ir.py` | `tests/semantics/conversion/fortran/test_compile_time_values.py`, `test_fortran_conversion_procedures_and_interfaces.py`, `test_modules_and_imports.py`, `test_types_and_storage.py` | | `tests/parser/test_preprocessing_cli.py` | `tests/pipeline/preprocessing/test_cli.py`, `test_configuration_and_adapters.py`, `test_dependencies_and_includes.py`, `test_execution.py` | -| `tests/semantics/test_pyi_printer.py` | `tests/codegen/printers/test_calls_and_policy_metadata.py`, `test_classes_and_methods.py`, `test_pyi_printer_imports_and_packages.py`, `test_types_and_declarations.py` | +| `tests/semantics/test_pyi_printer.py` | `tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py`, `test_classes_and_methods.py`, `test_pyi_printer_imports_and_packages.py`, `test_types_and_declarations.py` | | `tests/test_runtime_handles.py` | `tests/runtime/handles/test_array_actual_abi.py`, `test_descriptor_abi.py`, `test_factories_and_lifecycle.py`, `test_handle_protocols.py` | | `tests/semantics/test_c2ir.py` | `tests/semantics/conversion/c/test_functions_and_callbacks.py`, `test_projects_and_diagnostics.py`, `test_records_and_enums.py`, `test_types_and_constants.py` | | `tests/semantics/test_semantic_wrap_readiness.py` | `tests/cli/test_wrap_readiness.py`; `tests/semantics/readiness/test_policy_blockers.py`, `test_pyi_readiness.py`, `test_reports.py` | @@ -259,8 +259,8 @@ row therefore accounts for two of the 34 normalized-ID differences. | `tests/semantics/test_c_semantic_readiness.py` | `tests/semantics/readiness/test_c_readiness.py` | | `tests/semantics/test_fortran2ir.py` | `tests/semantics/conversion/fortran/` | | `tests/semantics/test_ir2ast.py` | `tests/lowering/test_semantic_ir.py` | -| `tests/semantics/test_pyi_printer.py` | `tests/codegen/printers/` | -| `tests/semantics/test_pyi_printer_modern_example.py` | `tests/codegen/printers/test_modern_example.py` | +| `tests/semantics/test_pyi_printer.py` | `tests/wrapper_codegen/printers/` | +| `tests/semantics/test_pyi_printer_modern_example.py` | `tests/wrapper_codegen/printers/test_modern_example.py` | | `tests/semantics/test_semantic_wrap_readiness.py` | `tests/semantics/readiness/` | | `tests/tools/test_documentation_examples.py` | `tests/docs/test_examples.py` | | `tests/tools/test_documentation_structure.py` | `tests/docs/test_structure.py` | @@ -304,3 +304,13 @@ only fixture/generator trees remain under `tests/parser/`, `tests/pyi/`, and - [x] Final tree, mapping, split rationale, helper moves, collection evidence, focused results, wrapper verification, static checks, retained locations, and any product failures are recorded here and in the handoff. + +## Post-Cutover Supersession + +The earlier inventory and execution record above documents the historical test +move and intentionally retains its old paths as evidence. After the canonical +wrapper-plan cutover, `tests/codegen/`, `tests/lowering/`, `x2py.codegen`, and +the adjacent legacy lowering/build entrypoints are removed. Still-required +contracts belong to completed semantic-policy tests, `tests/wrapper_codegen/` +plan and direct-generation tests, or compiled public behavior under +`tests/wrapper/fortran/`. diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 7d71f3b2a..a0ae2e09f 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -593,11 +593,11 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 236 | -| `dual-route` | 5 | -| `legacy` | 113 | -| `not-applicable` | 95 | -| `deferred-real-library` | 2 | +| `wrapper-plan` | 346 | +| `dual-route` | 0 | +| `legacy` | 0 | +| `not-applicable` | 75 | +| `deferred-real-library` | 0 | #### Recorded Route Progression @@ -629,6 +629,8 @@ blockers. | Phase 8 scalar-derived object lifetimes | 106 | 5 | 123 | 95 | 2 | 331 | | Phase 8 complete scalar-derived actual/dummy matrix | 213 | 5 | 123 | 95 | 2 | 438 | | Phase 8H failure, qualified-type, and typed-value closure | 222 | 5 | 123 | 95 | 2 | 447 | +| Phase 11 cross-cutting suite completion | 344 | 0 | 0 | 95 | 2 | 441 | +| Phase 12 canonical cutover | 346 | 0 | 0 | 95 | 0 | 441 | Migration is complete only when `legacy`, `dual-route`, and `deferred-real-library` are all zero. At that point every runtime-generating @@ -645,146 +647,145 @@ already covered by the new generator. | Pytest selector | Generation unit | Feature lanes / blockers | Status | | --- | --- | --- | --- | -| `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `legacy` | +| `tests/wrapper/fortran/arrays/test_array_contracts.py::*` | source/generated-.pyi parity or parametrized route | ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/arrays/test_array_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts[*]` | production plan route in source/generated-.pyi parity modes | fixed/runtime-shape ordinary array results; owned allocatable descriptor results; namespace preservation | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_ordinary_array_results_match_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape ordinary array results; ranks one through fifteen; Fortran order; zero-sized results; allocation/copy/release failure paths | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_array_results.py::test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes` | reduced owned-result contract with deliberate legacy rollback comparison | allocated and zero-sized wrapper-owned `CFI_CDESC_T` function-result handles; extraction and release | `wrapper-plan` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing assumed-rank native unit | runtime ranks one through fifteen; mutable storage; rank validation; native-handle actuals deferred to Phase 7 | `dual-route` | -| `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::*` | non-generating: legacy model/printer/policy unit coverage | legacy model/printer mechanics; ordinary arrays; native handles/descriptors | `not-applicable` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `legacy` | -| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing multidimensional native unit | dense/explicit extents; positive-strided views; zero-sized axes; projected output identity; native-handle actuals deferred to Phase 7 | `dual-route` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_ordinary_array_results_use_canonical_plan` | canonical production output-only plan route | fixed/runtime-shape ordinary array results; ranks one through fifteen; Fortran order; zero-sized results; allocation/copy/release failure paths | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state` | canonical reduced owned-result contract | allocated and zero-sized wrapper-owned `CFI_CDESC_T` function-result handles; extraction and release | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arrays_use_explicit_plan_branches` | reduced semantic `.pyi` entry over the existing assumed-rank native unit | runtime ranks one through fifteen; mutable storage; rank validation; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views[*]` | source/generated-.pyi parity | ordinary arrays; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::test_dense_strided_and_projected_arrays_use_canonical_plan` | reduced semantic `.pyi` entry over the existing multidimensional native unit | dense/explicit extents; positive-strided views; zero-sized axes; projected output identity; native-handle actuals deferred to Phase 7 | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_init_entry_uses_resolved_parent_name_from_inside_package` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_output_name_override_replaces_entry_inference` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support_imports` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_cycles_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_recursive_graph_reports_missing_relative_contract_before_native_validation` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_source_build_preserves_modules_and_root_externals` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_both_routes[*]` | production plan route with deliberate legacy rollback comparison | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::test_complete_general_source_preserves_namespaces_through_canonical_plan[*]` | canonical production plan route | Python namespace hierarchy; native import aliases; scalar inputs/results; void calls | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_can_alias_one_module_procedure_at_the_root` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_rejects_colliding_wildcard_exports` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_entry_wildcard_import_explicitly_flattens_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_matches_checked_in_fixture` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mixed_entry_exposes_externals_at_root_and_modules_as_children` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_leaf_can_be_the_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_variable_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mutable_module_variable_default_initializes_native_storage` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_matches_checked_in_fixture` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mixed_entry_exposes_externals_at_root_and_modules_as_children` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_leaf_can_be_the_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_module_variable_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_mutable_module_variable_default_initializes_native_storage` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_one_entry_preserves_multiple_native_module_namespaces` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_preserves_explicit_ordered_link_items` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_cli_requires_a_native_link_input` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_makefile_manifest_and_replay_workflows` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_manifest_records_pointer_descriptor_interop_requirements` | non-generating: manifest serialization unit | completed native-array build requirements and local standard-descriptor headers | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_accepts_exactly_one_entry_contract` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_a_missing_native_artifact` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_address_contracts_before_codegen[*]` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_invalid_projection_before_codegen` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_pyi_python_api_rejects_python_suffix_as_semantic_contract` | non-generating: validation/failure-path assertion | semantic .pyi generation/parsing; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_extension_beside_source` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_source_build_result_records_structured_native_plan` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_custom_wrapper_flags` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_verbose_mode_prints_full_direct_build_commands` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_empty_source_list` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_makefile_verbose_combination` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_wrapper_build_rejects_missing_source` | non-generating: validation/failure-path assertion | build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/build_from_source/test_compiler_verbose.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | -| `tests/wrapper/fortran/build_from_source/test_runtime_abi.py::*` | direct wrapper/build route | build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/build_from_source/test_compiler_verbose.py::*` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_runtime_abi.py::*` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_source_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines | `wrapper-plan` | | `tests/wrapper/fortran/callbacks/test_array_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/callbacks/test_callback_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/callbacks/test_derived_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; derived types/object lifetimes | `wrapper-plan` | | `tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::*` | source/generated-.pyi parity or parametrized route | callbacks/trampolines; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `legacy` | -| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `legacy` | +| `tests/wrapper/fortran/derived_types/test_borrowed_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_constructors_and_finalizers.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_derived_layout.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_derived_type_boundaries.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_derived_type_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `legacy` | +| `tests/wrapper/fortran/derived_types/test_derived_type_methods.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::*` | reduced passing legacy/source artifacts compared with direct typed-plan generation; plain non-target module objects intentionally use the safer member-proxy correction described in Phase 8 | scalar derived arguments/results; optional and by-value inputs; projected identity; owned/borrowed lifecycle; plain/`Aliased` module objects; scalar/string/array/nested/native-handle fields; production routing | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::*` | reduced direct-plan bound-constructor runtime and artifact proof | explicit bound construction; shared method plan; allocation and owner commit | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py::*` | complete source/generated-contract and direct-plan proof over the canonical scalar-derived matrix fixture; replaces the former isolated descriptor rejection unit; final Phase 8H cross-suite verification remains a separate closure gate | all five actual declarations from module and wrapper origins; all six dummy forms; exact action/error selection; holder, scoped-address, allocation and pointer transactions; mixed multi-argument acquisition and reverse cleanup | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes` | reduced module-only contract with deliberate legacy rollback comparison | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/object lifetimes; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; raw array addresses completed by Phase 6G; derived result remains Phase 8 | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entries over existing vector/matrix native routines | raw numeric addresses; visible scalar-storage extents; rank one/two; default C and explicit Fortran orientation; mutation; integer-only conversion | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan` | canonical reduced module-only contract | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifetime` | direct wrapper/build route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call` | direct wrapper/build route | semantic .pyi generation/parsing; raw array addresses completed by Phase 6G; derived result remains Phase 8 | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_use_canonical_plan` | reduced edited semantic `.pyi` entries over existing vector/matrix native routines | raw numeric addresses; visible scalar-storage extents; rank one/two; default C and explicit Fortran orientation; mutation; integer-only conversion | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_copy_f_preserves_logical_axes_through_binding_owned_temporary` | reduced edited semantic `.pyi` entries over the existing matrix native routine | explicit C-to-Fortran representation copy; native-input and inout calls; projected original identity; binding-owned copyback and cleanup | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fnative_call_examples_f90` native unit | fixed mutable rank-zero NumPy bytes storage; raw fixed-string addresses; in-place mutation; rank/dtype/itemsize/writability/type validation | `wrapper-plan` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::test_fixed_string_storage_and_raw_address_use_canonical_plan` | reduced edited semantic `.pyi` entry over the existing `fnative_call_examples_f90` native unit | fixed mutable rank-zero NumPy bytes storage; raw fixed-string addresses; in-place mutation; rank/dtype/itemsize/writability/type validation | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_ownership_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation` | non-generating: policy validation before bridge generation | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `not-applicable` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `legacy` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements` | direct wrapper/build route | semantic .pyi generation/parsing; native handles/descriptors; derived types/object lifetimes; optional/presence/writeback | `wrapper-plan` | +| `tests/wrapper/fortran/edit_pyi_contracts/test_surface_edit_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; classes/methods/properties/overloads; naming/visibility/dispatch | `wrapper-plan` | | `tests/wrapper/fortran/edit_pyi_contracts/test_visibility_contracts.py::*` | direct wrapper/build route | semantic .pyi generation/parsing; scalar module visibility and namespace projection | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | direct wrapper/build route | scalar external symbol; explicit bridge interface; renamed export | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_external_bridge_uses_explicit_interface_and_no_module_use` | direct wrapper/build route | scalar external symbol; explicit bridge interface | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_fixed_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_free_form_standalone_external_runtime_parity[*]` | source/generated-.pyi parity or parametrized route | scalar external symbol; explicit bridge interface | `wrapper-plan` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_generated_external_contracts_are_non_empty_root_fragments` | direct wrapper/build route | external symbols/native linkage | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `legacy` | -| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | direct wrapper/build route | external symbols/native linkage | `legacy` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_generated_external_contracts_are_non_empty_root_fragments` | direct wrapper/build route | external symbols/native linkage | `wrapper-plan` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | direct wrapper/build route | external symbols/native linkage; ordinary arrays | `wrapper-plan` | +| `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | direct wrapper/build route | external symbols/native linkage | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_namespace_imported_module_rejects_external_marker_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root[*]` | source/generated-.pyi parity or parametrized route | scalar external symbols; explicit bridge interfaces | `wrapper-plan` | | `tests/wrapper/fortran/external_routines/test_external_procedures.py::test_package_entry_rejects_non_external_root_declaration_before_codegen` | non-generating: validation/failure-path assertion | external symbols/native linkage | `not-applicable` | | `tests/wrapper/fortran/function_calls/test_function_call_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `legacy` | -| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `legacy` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | direct wrapper/build route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection[*]` | source/generated-.pyi parity or parametrized route | native-call projections; ordinary arrays; strings; derived types/object lifetimes | `wrapper-plan` | | `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states` | production plan route with deliberate legacy rollback comparison | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | production plan route with deliberate legacy rollback comparison | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes` | reduced optional descriptor contract with deliberate legacy rollback comparison | omitted/`None` absence; present unallocated/unassociated and allocated/associated handle states; kind/dtype validation | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `legacy` | -| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `dual-route` | -| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes` | production plan route with deliberate legacy rollback comparison | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/object lifetimes; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes` | production output-only plan route with deliberate legacy rollback comparison | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_fixed_optional_scalar_plan_matches_all_presence_states` | canonical production plan route | optional/presence; scalar inputs/results; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value` | canonical production plan route | optional/presence; nullable scalar descriptor; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_descriptors_preserve_presence_and_storage_state` | canonical reduced optional descriptor contract | omitted/`None` absence; present unallocated/unassociated and allocated/associated handle states; kind/dtype validation | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior[*]` | source/generated-.pyi parity or parametrized route | optional/presence/writeback | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_array_buffers_preserve_omission_and_identity` | reduced semantic `.pyi` entry over the existing optional native unit | omitted/`None`/present ordinary array storage; mutation; projected identity; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement` | canonical production plan route | scalar copy-in/native mutation/copy-out/cleanup; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules[*]` | source/generated-.pyi parity | mixed-type multiple-result aggregation; ordinary arrays; strings; derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/function_calls/test_output_arguments.py::test_hidden_ordinary_array_output_uses_canonical_plan` | canonical production output-only plan route | fixed/runtime-shape hidden ordinary array output; zero-sized output; allocation/copy failure paths | `wrapper-plan` | | `tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py::*` | non-generating: wrapper docs/test layout | test/docs layout | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_replacement_has_no_native_memory_errors[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `legacy` | -| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes` | reduced owned-result plus projected-descriptor contract with deliberate legacy rollback comparison | direct persistent descriptor mutation; allocation/reallocation/deallocation; same-handle result identity | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | source/generated-.pyi parity with one mixed generation unit | derived class/field handles and parent retention remain Phase 8/9 blockers | `legacy` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_allocatable_replacement_has_no_native_memory_errors[*]` | source/generated-.pyi parity route | module variables/state; native handles/descriptors | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity` | canonical reduced owned-result plus projected-descriptor contract | direct persistent descriptor mutation; allocation/reallocation/deallocation; same-handle result identity | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles[*]` | source/generated-.pyi parity with one mixed generation unit | derived class/field handles and parent retention remain Phase 8/9 blockers | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | production plan route in source/generated-.pyi parity modes | rank-zero allocatable/pointer arguments, writeback, results, and copied nullable module values | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view[*]` | production plan route after the Phase 7 contract correction | plain and `Aliased` module handles return a current live view or `None`; explicit `.copy()` is independent and a fresh extraction follows current native state | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_common_blocks.py::*` | source/generated-.pyi parity or parametrized route | scalar calls with internal common-block storage | `wrapper-plan` | -| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/object lifetimes | `legacy` | -| `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `legacy` | +| `tests/wrapper/fortran/module_state/test_module_state.py::test_aliased_derived_module_object_borrows_native_state[*]` | source/generated-.pyi parity or parametrized route | module variables/state; derived types/object lifetimes | `wrapper-plan` | +| `tests/wrapper/fortran/module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter[*]` | source/generated-.pyi parity or parametrized route | module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/module_state/test_module_state_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route[*]` | production plan route with deliberate legacy rollback comparison | scalar inputs/results; scalar module variables/state; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | +| `tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_uses_canonical_plan` | canonical production plan route | scalar inputs/results; scalar module variables/state; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_makefile_mode_reproduces_multi_source_build` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_modules_build_one_merged_extension` | direct wrapper/build route | scalar multi-source build/link orchestration; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_file_standalone_procedures_build_one_merged_extension` | direct wrapper/build route | scalar multi-source external symbols and link orchestration | `wrapper-plan` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_generated_contract_build_matches_source_runtime_and_link_order` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_modified_entry_preserves_modules_and_adds_documented_alias` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `legacy` | -| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_pyi_out_writes_one_flat_combined_package` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `legacy` | -| `tests/wrapper/fortran/naming/test_defined_operators.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; operators; generic dispatch | `legacy` | -| `tests/wrapper/fortran/naming/test_generic_interfaces.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; generic dispatch | `legacy` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_generated_contract_build_matches_source_runtime_and_link_order` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `wrapper-plan` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_modified_entry_preserves_modules_and_adds_documented_alias` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py::test_multi_source_pyi_out_writes_one_flat_combined_package` | direct wrapper/build route | build/compile/link orchestration; external symbols/native linkage; module variables/state; semantic .pyi generation/parsing | `wrapper-plan` | +| `tests/wrapper/fortran/naming/test_defined_operators.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; operators; generic dispatch | `wrapper-plan` | +| `tests/wrapper/fortran/naming/test_generic_interfaces.py::*` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads; generic dispatch | `wrapper-plan` | | `tests/wrapper/fortran/naming/test_naming_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | | `tests/wrapper/fortran/naming/test_phase9_class_overloads.py::*` | reduced direct-plan constructor and method overload runtime proof | class-owned exact predicates; constructor ownership; no speculative calls | `wrapper-plan` | -| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `legacy` | -| `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::*` | full BLAS/LAPACK wrapper generation unit | external symbols/native linkage; build/compile/link orchestration; broad wrapper corpus | `deferred-real-library` | +| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_strict_wrapper_names_reject_python_name_fixes` | direct wrapper/build route | naming/visibility/dispatch; classes/methods/properties/overloads | `wrapper-plan` | +| `tests/wrapper/fortran/naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy[*]` | source/generated-.pyi parity or parametrized route | naming/visibility/dispatch; classes/methods/properties/overloads | `wrapper-plan` | +| `tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::*` | canonical full BLAS/LAPACK wrapper generation; BLAS runs locally and both exact nodes run in the dedicated GitHub Actions matrix | external symbols/native linkage; build/compile/link orchestration; broad wrapper corpus | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_duplicate_native_definitions_report_linker_error` | direct wrapper/build route | scalar external symbols; linker failure propagation | `wrapper-plan` | -| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_imported_contracts_resolve_from_one_archive_or_shared_library[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_imported_contracts_resolve_from_one_archive_or_shared_library[*]` | source/generated-.pyi parity or parametrized route | external symbols/native linkage; build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_incompatible_native_artifact_reports_linker_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_module_directory_reports_compile_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_missing_symbol_reports_native_link_or_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | @@ -793,35 +794,35 @@ already covered by the new generator. | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_dependency_order_resolves_transitive_library` | direct wrapper/build route | scalar external symbol; ordered archive linkage | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_static_archive_groups_resolve_cyclic_archive_dependencies` | direct wrapper/build route | scalar external symbol; archive-group linkage | `wrapper-plan` | | `tests/wrapper/fortran/real_libraries/test_stage7_native_bundles.py::test_unavailable_dependent_shared_library_reports_loader_error` | non-generating: validation/failure-path assertion | external symbols/native linkage; build/compile/link orchestration | `not-applicable` | -| `tests/wrapper/fortran/runtime_behavior/test_openmp_runtime.py::*` | direct wrapper/build route | runtime policies/errors/GIL; build/compile/link orchestration | `legacy` | +| `tests/wrapper/fortran/runtime_behavior/test_openmp_runtime.py::*` | direct wrapper/build route | runtime policies/errors/GIL; build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/runtime_behavior/test_runtime_behavior_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::*` | source and edited-.pyi production plan route with deliberate legacy rollback comparison | runtime policies/errors/GIL | `wrapper-plan` | +| `tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::*` | source and edited-.pyi canonical production plan route | runtime policies/errors/GIL | `wrapper-plan` | | `tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::*` | source/generated-.pyi parity or parametrized route | runtime policies/errors/GIL; scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `legacy` | -| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | direct wrapper/build route | scalar inputs/results; module variables/state | `legacy` | +| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; module variables/state | `wrapper-plan` | +| `tests/wrapper/fortran/scalars/test_fortran_enums.py::test_fortran_enums_preserve_values_in_generated_pyi_contract` | direct wrapper/build route | scalar inputs/results; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::*` | scalar-only copied native routines with deliberate legacy/direct-plan parity | primitive scalar kinds; value and `Addr(Arg(i))` inputs; hidden output; copy-in/copy-out; rank-zero storage; raw `Addr(T)`; native slot reordering; direct-plus-hidden result tuple assembly | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_scalar_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | -| `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `legacy` | -| `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `legacy` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes[*]` | production plan route with deliberate legacy rollback comparison using the existing `fmath.f` and `fmath_f90.f90` generation units | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/scalars/test_scalar_kinds.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | +| `tests/wrapper/fortran/scalars/test_value_and_bind_c.py::*` | source/generated-.pyi parity or parametrized route | scalar inputs/results; native-call projections | `wrapper-plan` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fmath_scalar_sources_use_canonical_wrapper_plan[*]` | canonical production plan route using the existing fixed- and free-form generation units | scalar inputs/results; native-call projections; Python namespaces; build/artifact integration | `wrapper-plan` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results; ordinary arrays | `wrapper-plan` | | `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension[*]` | source/generated-.pyi parity or parametrized route | scalar inputs/results | `wrapper-plan` | -| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_required_array_buffers_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fmath_arrays_f90` native unit | required rank-one dense buffers; exact dtype/rank/order/alignment/writeability; zero length; native-handle actuals deferred to Phase 7 | `dual-route` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | edited semantic `.pyi` contract | strings; fixed/assumed inputs; arrays; mutable string storage | `legacy` | +| `tests/wrapper/fortran/scalars/test_verified_baseline.py::test_required_array_buffers_use_canonical_wrapper_plan` | reduced semantic `.pyi` entry over the existing `fmath_arrays_f90` native unit | required rank-one dense buffers; exact dtype/rank/order/alignment/writeability; zero length; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_edited_modern_string_contract_wraps_full_axis_spelling_set` | edited semantic `.pyi` contract | strings; fixed/assumed inputs; arrays; mutable string storage | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results[*]` | production plan route from source/generated-.pyi parity | fixed-form strings; fixed/assumed inputs; fixed results | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity | strings; fixed/assumed inputs; fixed/deferred results; arrays; writeback | `legacy` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes` | reduced scalar descriptor result contract with deliberate legacy rollback comparison | runtime length; nullable copy-out; UTF-8 data; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes` | reduced descriptor-result and projected-descriptor contract with deliberate legacy rollback comparison | hidden/direct owned deferred-character arrays; runtime `S3`/`S4`/`S5` width; projected identity; nullable rank-zero result | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | fixed-width `NPY_STRING` array itemsize; rank/dtype/zero-size validation; native-handle actuals deferred to Phase 7 | `dual-route` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fstrings_f90` native unit | raw fixed-width character array address; literal shape; element length; integer-only conversion | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | required fixed/assumed scalar string inputs; default/kind-1/`c_char`; UTF-8 length and NUL validation; scalar results | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_string_results_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | direct fixed string results; trailing blanks; default/`c_char`; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results[*]` | source/generated-.pyi parity | strings; fixed/assumed inputs; fixed/deferred results; arrays; writeback | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan` | canonical reduced scalar descriptor result contract | runtime length; nullable copy-out; UTF-8 data; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan` | canonical reduced descriptor-result and projected-descriptor contract | hidden/direct owned deferred-character arrays; runtime `S3`/`S4`/`S5` width; projected identity; nullable rank-zero result | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_width_character_arrays_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | fixed-width `NPY_STRING` array itemsize; rank/dtype/zero-size validation; native-handle actuals deferred to Phase 7 | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_use_canonical_plan` | reduced edited semantic `.pyi` entry over the existing `fstrings_f90` native unit | raw fixed-width character array address; literal shape; element length; integer-only conversion | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_required_scalar_string_inputs_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | required fixed/assumed scalar string inputs; default/kind-1/`c_char`; UTF-8 length and NUL validation; scalar results | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_arguments.py::test_fixed_string_results_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fstrings_f90` native unit | direct fixed string results; trailing blanks; default/`c_char`; allocation failure | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy[*]` | production plan route from source/generated-.pyi parity | strings; fixed/assumed input/output; optional presence; Unicode/NUL handling | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed hidden string output; trailing blanks; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes` | reduced edited semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed immutable replacement and discarded identity; exact length; trailing blanks; allocation failure | `wrapper-plan` | -| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | assumed-length and optional immutable replacement; empty/omitted/`None`/concrete states; NUL rejection; concrete-only allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_hidden_string_output_uses_canonical_plan` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed hidden string output; trailing blanks; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_fixed_string_replacement_and_identity_use_canonical_plan` | reduced edited semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | fixed immutable replacement and discarded identity; exact length; trailing blanks; allocation failure | `wrapper-plan` | +| `tests/wrapper/fortran/strings/test_character_edge_cases.py::test_assumed_and_optional_string_replacements_use_canonical_plan` | reduced semantic `.pyi` entry over the existing `fcharacter_edges_f90` native unit | assumed-length and optional immutable replacement; empty/omitted/`None`/concrete states; NUL rejection; concrete-only allocation failure | `wrapper-plan` | | `tests/wrapper/fortran/strings/test_string_generated_pyi_contracts.py::*` | non-generating: generated semantic .pyi fixture parity | semantic .pyi generation/parsing | `not-applicable` | ## Incremental Protocol @@ -1300,6 +1301,14 @@ one hidden primitive scalar output, assembled into a Python tuple in declared result order. Keep it separate from arrays, strings, derived types, and native handles before widening the plan route. +For source-derived contracts, an ordinary non-descriptor `intent(out)` scalar +hidden by Python result projection still selects `PASS_CALL_LOCAL_ADDRESS` +even when no edited `.pyi` `Addr(...)` spelling exists. The hidden-result +projection is itself the completed semantic fact that requires writable +call-local native storage; the binding and bridge must not rediscover that ABI +rule. Rank-zero allocatable/pointer descriptor outputs retain the distinct +Phase 7H descriptor transport and are not rewritten as ordinary addresses. + The completed representation is an ordered `FunctionWrapperPolicy.results` tuple and an ordered `FunctionPlan.results` tuple. Each Python-visible result has its own `ResultPolicy` and `ResultPlan`, including its binding consumer and @@ -2865,13 +2874,13 @@ Phase 7 rows were split, proved through both routes, and then recorded as | Completed sub-lane | Dependency-closed compiled evidence | | --- | --- | -| Phase 7A, 7B, 7F, and 7G | `derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes` | -| Phase 7C | `function_calls/test_optional_arguments.py::test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes` | -| Phase 7D | `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes` | -| Phase 7E numeric | `arrays/test_array_results.py::test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes` | -| Phase 7E deferred character | `strings/test_character_arguments.py::test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes` | +| Phase 7A, 7B, 7F, and 7G | `derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan` | +| Phase 7C | `function_calls/test_optional_arguments.py::test_optional_array_descriptors_preserve_presence_and_storage_state` | +| Phase 7D | `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity` | +| Phase 7E numeric | `arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state` | +| Phase 7E deferred character | `strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan` | | Phase 7H numeric | `scalars/test_scalar_boundary_plan.py::test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_route` | -| Phase 7H deferred scalar character | `strings/test_character_arguments.py::test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes` and the nullable case in the deferred-character handle test | +| Phase 7H deferred scalar character | `strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan` and the nullable case in the deferred-character handle test | | Phase 7H source/default projection | `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values[*]` | Required focused intermediate coverage includes: @@ -4357,6 +4366,12 @@ value/reference access, intent-derived copy direction, rank, shape, character length, and result representation. Normal wrapper projection and callback adapter projection remain distinct completed records. +Native `VALUE` and callback access are independent facts. A callback dummy may +be both `VALUE` and `INTENT(IN)`; the completed ABI selects value passing while +the completed access selects read-only behavior, and the typed adapter must +emit both attributes. `INTENT(IN)` without `VALUE` remains a reference-passed +dummy even though Python observes a converted scalar value. + The supported callback contract is deliberately call-scoped: - the Python callable is validated and retained before the native call, placed @@ -4629,108 +4644,192 @@ Radon base noted above, and implementation stopped before Phase 11. ## Phase 11 — Cross-Cutting Wrapper Suite Completion +Implementation status: complete. The pre-Phase-11 ledger contained 236 +wrapper-plan nodes, five dual-route array parity nodes, 113 passing legacy-route +nodes, 95 non-generating nodes, and two deferred real-library nodes. The final +forced-plan sweep passed 435 of 449 non-real-library nodes before obsolete +dual-route artifact assertions were removed; its two shared implementation +gaps were Fortran-ordered strided ndarray validation and static `nopass` method +dispatch, both now resolved through existing policy/runtime paths. + +The ordered output aggregator now combines direct and hidden native results +with visible scalar, string, array, and derived writeback. It converts each +value once in public result order and releases every earlier Python reference +if a later conversion or tuple allocation fails; the former single-result and +"native result plus writeback" blockers are removed. + Scope: existing wrapper tests whose generation units combine completed semantic lanes or exercise build and runtime behavior rather than introducing one new datatype lane. -- [ ] Reconcile every remaining `legacy` or `dual-route` matrix row by owning +Implement in these dependency-ordered waves: + +1. reconcile the five reduced array dual-route nodes and remove stale Phase 7 + exclusion bookkeeping where their completed actual-source policy now permits + production routing; +2. migrate source/semantic-`.pyi` build modes, edited contracts, external + symbols, multiple-source linkage, and independent native bundles through one + shared route and planner; +3. migrate mixed scalar/string/array/handle/derived/module/class generation + units without adding per-test or per-datatype fallback; +4. migrate naming, generic interfaces, defined operators, OpenMP/runtime policy, + and remaining public-surface orchestration; and +5. require the live nondeferred ledger to contain only `wrapper-plan` or + justified `not-applicable` nodes before Phase 12 begins. + +- [x] Reconcile every remaining `legacy` or `dual-route` matrix row by owning test area: `build_from_source`, `build_from_pyi`, `edit_pyi_contracts`, `external_routines`, `multiple_files`, `naming`, `runtime_behavior`, and `real_libraries`. -- [ ] Group remaining rows into dependency-ordered waves by their actual +- [x] Group remaining rows into dependency-ordered waves by their actual unsupported owner paths. Do not implement a broad test directory as one special case and do not add per-test backend fallbacks. -- [ ] For every newly discovered semantic or backend gap, expand the applicable +- [x] For every newly discovered semantic or backend gap, expand the applicable earlier lane or add an explicit sub-lane here, then follow the complete policy -> plan -> backend -> emission -> compiled parity -> route sequence. -- [ ] Prove source-driven and semantic-`.pyi`-driven builds use the same route +- [x] Prove source-driven and semantic-`.pyi`-driven builds use the same route selector and wrapper planner while retaining their existing build assertions. -- [ ] Prove edited-policy contracts, external symbols, multiple-source builds, +- [x] Prove edited-policy contracts, external symbols, multiple-source builds, naming/generic interfaces, runtime policies, recursion, OpenMP, and real library-independent native bundles preserve their existing assertions through the wrapper-plan route. -- [ ] Keep non-wrapper-generating tests, including layout and generated-`.pyi` +- [x] Keep non-wrapper-generating tests, including layout and generated-`.pyi` checks, marked `not-applicable` to route selection but passing in the same suite. -- [ ] Run every `tests/wrapper` test except +- [x] Run every `tests/wrapper` test except `test_real_blas_lapack.py` locally and in CI as the pre-cutover gate. -- [ ] Finish this phase only when every nondeferred matrix row is either +- [x] Finish this phase only when every nondeferred matrix row is either `wrapper-plan` or justified `not-applicable`; no nondeferred row may remain `legacy` or `dual-route`. BLAS/LAPACK rows remain `deferred-real-library` until Phase 12. +Closure evidence (2026-07-16): the Phase 11 ledger contains 344 canonical +wrapper-plan nodes, 95 justified non-generating nodes, two deferred +real-library nodes, and no legacy or dual-route node. The complete local +pre-cutover suite outside the shared BLAS/LAPACK file passed all 439 collected +tests. Mixed outputs use the ordered aggregator, Fortran-ordered strided array +validation reuses the shared array-actual runtime path, and static `nopass` +methods reuse the completed class invocation path; no per-test route or +backend fallback was added. + ## Phase 12 — Cutover And Removal -- [ ] Re-audit collected Python test nodes under `tests/wrapper` and reconcile +Implementation status: complete. Local BLAS evidence is recorded below; +LAPACK execution remains intentionally CI-only. + +Local verification boundary: run the BLAS generation unit locally. Do not run +the LAPACK generation unit locally; make its wrapper-plan invocation runnable +in GitHub Actions and use that job for LAPACK parity and cutover evidence. + +External-interface parameter lists preserve native ABI order, while their +declarations may be topologically ordered from the plan's explicit array +extent-reference roles. This permits a later scalar extent dummy to be +declared before an earlier array dummy without reordering the native call. + +Cutover contract: source builds, semantic-`.pyi` builds, Makefile generation, +manifest replay, and strict-name validation all use completed policy -> +`WrapperPlan` -> `WrapperCodeGenerator`. The build API has no route selector, +rollback flag, or silent fallback; an unsupported owner path fails before any +backend or legacy lowering runs. + +- [x] Re-audit collected Python test nodes under `tests/wrapper` and reconcile them with the migration matrix. No test may be missing from the matrix. -- [ ] After every other migration row is complete, restore the full +- [x] After every other migration row is complete, restore the full `test_real_blas_lapack.py` run and any required native-cache preparation in local opt-in verification and GitHub Actions. -- [ ] Run both BLAS and LAPACK generation units through legacy and wrapper-plan - routes using their existing assertions. Resolve parity before changing their - matrix rows from `deferred-real-library` to `wrapper-plan`. -- [ ] Require every wrapper-generating test row to be `wrapper-plan`; no row may - remain `legacy` or `dual-route`. Confirm route diagnostics show that every - runtime wrapper generation unit uses the new route. -- [ ] Run the complete `tests/wrapper` suite in CI, including restored BLAS and - LAPACK coverage, and require every test to pass before legacy deletion. -- [ ] Track which lanes still use the old `semantic_ir_to_codegen_ast()` path. -- [ ] Track route support at whole-generation-unit granularity and keep - unsupported owner-path diagnostics stable until the corresponding lane is - migrated. -- [ ] Keep completed legacy lanes available for deliberate rollback until all - live lanes have parity evidence and the final cutover is approved; do not - delete old handlers incrementally merely because one fixture uses the plan - route. -- [ ] Do not move modified isolated nodes or printers back into the legacy +- [x] Run BLAS locally through the canonical route using its existing contract, + import, ABI, and runtime assertions. Run the equivalent exact LAPACK node in + the dedicated GitHub Actions real-library matrix; do not run it locally. +- [x] Require every wrapper-generating test row to be `wrapper-plan`; no row + remains `legacy`, `dual-route`, or `deferred-real-library`. +- [x] Configure the complete `tests/wrapper` suite in CI with ordinary tests in + the main matrix and the full BLAS/LAPACK nodes in the cached real-library + matrix. +- [x] Confirm no wrapper build lane uses the old + `semantic_ir_to_codegen_ast()` path. The old lowering is no longer a supported + test owner and receives no focused compatibility coverage. +- [x] Remove route support tracking and fallback diagnostics; whole-generation + units now either validate and generate one plan or fail on exact owner-path + support diagnostics before emission. +- [x] Retain rollback only until the live ledger is reconciled, then remove it + in one cutover without compatibility flags or per-function fallback. +- [x] Do not move modified isolated nodes or printers back into the legacy package during migration. After final cutover, remove the legacy package pieces proven unused and keep `x2py.wrapper_codegen` as the canonical generator rather than performing a second package rename. -- [ ] Remove old lowering/codegen code only after every live wrapper lane has a - wrapper-plan route and focused verification. -- [ ] Remove the temporary legacy route and its route diagnostics after every +- [x] Keep semantic `.pyi` emission under `x2py.wrapper_codegen.printers` and + retire focused tests of the old semantic AST, bridge, binding, and printer + implementation before deleting the legacy package. +- [x] Remove the temporary legacy route and its route diagnostics after every live generation unit is supported; do not replace it with compatibility shims or per-function fallback. -- [ ] Remove migration-only dual-route orchestration after the complete existing +- [x] Remove migration-only dual-route orchestration after the complete existing wrapper suite proves the wrapper-plan route and legacy rollback is no longer supported. Keep the existing behavioral fixtures and assertions. -- [ ] Keep source printers only for the remaining generated source fragments they +- [x] Keep source printers only for the remaining generated source fragments they still own, or replace them with narrower emitters once the model layer is no longer needed. +Closure evidence (2026-07-16): the final live ledger contains 346 canonical +wrapper-plan nodes, 75 justified non-generating nodes, and zero legacy, +dual-route, or deferred nodes. The complete local suite outside the shared +real-library file passed 419 tests; the exact BLAS full-library node passed +locally; and the exact BLAS and LAPACK nodes are runnable as independent legs +of the cached GitHub Actions real-library matrix. LAPACK was intentionally not +run locally, so its runtime result remains CI evidence. Focused semantic and +compiled class/module policy tests passed 80 tests, all wrapper-codegen tests +passed 352 tests, and documentation plus structural layout checks passed 1,142 +tests. Ruff lint/format, Bandit, Vulture, the wrapper-codegen complexity check, +the Radon policy against explicit base `main`, advisory Radon complexity and +maintainability reports, and `git diff --check` all passed. + ## Verification -- [ ] Documentation-only changes run +- [x] Documentation changes run `python3 -m pytest -q tests/docs/test_examples.py tests/docs/test_structure.py` and `git diff --check`. -- [ ] Wrapper-plan code changes run the affected existing `tests/wrapper` nodes, +- [x] Wrapper-plan code changes run the affected existing `tests/wrapper` nodes, the minimal intermediate contract tests required above, and the required static-analysis suite from `AGENTS.md`. -- [ ] Wrapper-codegen implementation changes pass +- [x] Wrapper-codegen implementation changes pass `python3 tools/check_wrapper_codegen_complexity.py` with no handler waiver. -- [ ] Runtime wrapper tests are required when generated behavior changes. -- [ ] Every migrated lane runs eligible existing fixtures and assertions through - both routes. Compare behavior and ABI-relevant call mapping; do not require - byte-identical source when mechanical organization differs. -- [ ] Structural dependency tests prove complete generator isolation: no +- [x] Runtime wrapper tests cover every changed generated behavior. +- [x] Every migrated lane completed legacy-oracle comparison before cutover; + final tests now exercise only the canonical wrapper-plan route and retain the + existing behavior and ABI-relevant call assertions. +- [x] Structural dependency tests prove complete generator isolation: no imports from `x2py.wrapper_codegen` to `x2py.codegen` or in the reverse direction. -- [ ] BLAS and LAPACK full-library wrapper tests remain excluded locally and in - GitHub Actions throughout Phases 0-11. Re-enable both only at the explicit - Phase 12 gate after every other migration row is complete. +- [x] BLAS and LAPACK full-library wrapper tests remained excluded locally and in + GitHub Actions throughout Phases 0-11. At the explicit Phase 12 gate, enable + BLAS locally and in GitHub Actions, enable LAPACK only in GitHub Actions, and + keep local LAPACK execution disabled. ## Completion Record -- [ ] The final report for each lane names the plan actions added, the binding +- [x] The final report for each lane names the plan actions added, the binding and bridge handlers they dispatch to, and the handoff specs validated. -- [ ] The final report lists old lowering/codegen paths still used by unsupported - lanes. -- [ ] The final cutover report includes the completed `tests/wrapper` migration +- [x] No unsupported wrapper lane uses old lowering/codegen; focused tests now + target completed policy, `WrapperPlan`, `WrapperCodeGenerator`, or compiled + public behavior rather than `ir2ast.py` and `x2py.codegen` internals. +- [x] The final cutover report includes the completed `tests/wrapper` migration matrix and confirms every wrapper-generating row uses the wrapper-plan route. -- [ ] The final report includes focused verification commands and results. -- [ ] The final report includes the changed-stage breakdown required by +- [x] The final report includes focused verification commands and results. +- [x] The final report includes the changed-stage breakdown required by `AGENTS.md` and names every test file added or updated with the behavior it covers. + +## Post-Cutover Legacy Codegen Removal + +The legacy `x2py.codegen` package, `x2py/semantics/ir2ast.py`, and the obsolete +`x2py/compiling/python_wrapper.py` pipeline are removed together. No alias, +fallback, compatibility import, or rejection-only test preserves that route. + +Required behavior remains with its current owner: completed semantic policy +tests for semantic decisions, `tests/wrapper_codegen/` for plans and direct +source generation, and compiled `tests/wrapper/` cases for public Python +behavior and native ABI outcomes. Static-analysis baselines cover only source +that remains in the repository. ## Session Continuation Protocol The stable continuation prompt is: diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 9fe3fcdf0..5295e92e6 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -834,6 +834,24 @@ use ordinary return annotations; hidden allocatable array outputs use `Allocatable[T[...]]` handles whose unallocated state remains inside the handle. +### Generated Docstrings + +Generated modules, functions, classes, constructors, methods, overloads, and +properties expose compact NumPy-style docstrings derived from the same completed +wrapper plan as the executable code. `help(module.function)` therefore reports +the Python-visible signature rather than the native dummy list, including +hidden outputs, ordered tuple results, optional omission versus a present +`None`, constrained array shape and layout, handle ownership, and native-status +exceptions. + +Module docstrings index their public functions, module attributes, and classes. +Class docstrings index the public constructor, fields, methods, and overloads; +the individual constructor, method, overload, and property descriptors also +carry focused docstrings. Private wrapper helper names and internal bridge roles +are never shown. Module attributes are documented in the module docstring +because Python extension modules do not provide portable per-attribute +descriptor docstrings. + Runtime tests: [`test_output_arguments.py`](../../../tests/wrapper/fortran/function_calls/test_output_arguments.py), [`test_native_call_examples.py`](../../../tests/wrapper/fortran/function_calls/test_native_call_examples.py). diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index df3f17a5a..8d79cdf8e 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -33,7 +33,8 @@ reference and summarized later in Language Support. Status terms used below: -- **Generated**: emitted today by `--pyi` or `codegen.printers.pyi_printer`. +- **Generated**: emitted today by `--pyi` or + `wrapper_codegen.printers.pyi_printer`. - **Loaded**: accepted today by `x2py.pyi_parser` and converted back to semantic IR. - **Readiness**: understood by the semantic readiness checker. diff --git a/tests/README.md b/tests/README.md index 1ad3252ee..c2a517ef9 100644 --- a/tests/README.md +++ b/tests/README.md @@ -21,10 +21,7 @@ user-visible feature that a contributor is changing. | `.pyi` AST-to-semantic conversion | `tests/semantics/conversion/pyi/` | `python3 -m pytest -q tests/semantics/conversion/pyi` | | Completed semantic policy | `tests/semantics/policy/` | `python3 -m pytest -q tests/semantics/policy` | | Wrap readiness and blockers | `tests/semantics/readiness/` | `python3 -m pytest -q tests/semantics/readiness` | -| Semantic IR-to-codegen lowering | `tests/lowering/` | `python3 -m pytest -q tests/lowering` | -| Bridge generation | `tests/codegen/bridges/` | `python3 -m pytest -q tests/codegen/bridges` | -| Python binding generation | `tests/codegen/bindings/` | `python3 -m pytest -q tests/codegen/bindings` | -| Source and `.pyi` printers | `tests/codegen/printers/` | `python3 -m pytest -q tests/codegen/printers` | +| Wrapper planning, bridge/binding generation, and source/`.pyi` printing | `tests/wrapper_codegen/` | `python3 -m pytest -q tests/wrapper_codegen` | | Naming policy | `tests/naming/` | `python3 -m pytest -q tests/naming` | | NumPy and semantic type mapping | `tests/types/` | `python3 -m pytest -q tests/types` | | Runtime handles | `tests/runtime/handles/` | `python3 -m pytest -q tests/runtime/handles` | @@ -48,10 +45,7 @@ empty directory merely to mirror this table. | `x2py.semantics.c2ir`, `fortran2ir`, `pyi2ir` | matching language under `tests/semantics/conversion/` | | semantic ownership and policy completion | `tests/semantics/policy/` | | `x2py.semantics.readiness` | `tests/semantics/readiness/` | -| `x2py.semantics.ir2ast` | `tests/lowering/` | -| `x2py.codegen.bridges` | `tests/codegen/bridges/` | -| `x2py.codegen.bindings` | `tests/codegen/bindings/` | -| `x2py.codegen.printers` | `tests/codegen/printers/` | +| `x2py.wrapper_codegen` | `tests/wrapper_codegen/` plus compiled behavior under `tests/wrapper/` | | `x2py.compiling` | compiled build and runtime feature evidence under `tests/wrapper/fortran/` | | `x2py.naming` | `tests/naming/` | | `x2py.types` | `tests/types/` | @@ -73,6 +67,12 @@ know roadmap wording but not the feature module. Source-build, generated-`.pyi`, and modified-`.pyi` scenarios for one feature stay together; identical source/generated behavior uses one shared assertion body. +The legacy lowering AST and `x2py.codegen` implementation are removed. Their +handler maps and emitted-source details have no compatibility tests. Behavior +still required by users belongs either in `tests/wrapper_codegen/` against the +canonical plan/generator or in compiled `tests/wrapper/` coverage against the +public build APIs. + During the wrapper-plan migration, do not run the full BLAS or LAPACK real-library wrapper tests locally or in GitHub Actions. Exclude `wrapper/fortran/real_libraries/test_real_blas_lapack.py`; keep the general diff --git a/tests/_shared/fixture_outputs.py b/tests/_shared/fixture_outputs.py index edaf38605..4ee3ce936 100644 --- a/tests/_shared/fixture_outputs.py +++ b/tests/_shared/fixture_outputs.py @@ -11,7 +11,7 @@ from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source from x2py.semantics.c2ir import c_project_to_semantic_module from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules, fortran_module_to_semantic_module -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from x2py.semantics.readiness import assess_semantic_wrap_readiness from x2py.cli import _fortran_contract_files, _semantic_report diff --git a/tests/_shared/ownership_policy_support.py b/tests/_shared/ownership_policy_support.py index 895a89e98..5b6f8b2c7 100644 --- a/tests/_shared/ownership_policy_support.py +++ b/tests/_shared/ownership_policy_support.py @@ -1,5 +1,3 @@ -from dataclasses import replace - import pytest from x2py.contracts import CONTRACT_SYMBOLS @@ -12,47 +10,7 @@ SCALAR_STORAGE_CATEGORY, ) -from x2py.codegen.bind_c import ( - BindCArrayType, - BindCFunctionDef, - BindCNativeArrayDescriptorType, - BindCNativeArrayHandleProperty, - BindCNativeArrayHandleVariable, - BindCPointer, - BindCAccessorModuleVariable, - native_array_descriptor_argument_type, -) - -from x2py.codegen.bindings.c_concepts import CFIDescriptorType - -from x2py.codegen.bindings.c_to_python import CPythonBindingGenerator - -from x2py.codegen.bindings.cpython_api import PythonObjectType - -from x2py.codegen.bridges.fortran_to_c import FortranToCBridgeGenerator - -from x2py.codegen.models.core import ( - Declare, - FunctionCall, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - IndexedElement, - Return, - Variable, -) - -from x2py.codegen.models.datatypes import NIL, NumpyFloat64Type, NumpyNDArrayType, convert_to_literal - -from x2py.codegen.printers.ccode import CCodePrinter - -from x2py.codegen.printers.cpythoncode import CPythonCodePrinter - -from x2py.codegen.printers.fcode import FCodePrinter - -from x2py.codegen.printers.pyi_printer import PyiPrinter - -from x2py.codegen.scope import Scope +from x2py.wrapper_codegen.printers import PyiPrinter from x2py.semantics.ownership import ( AssignmentMode, @@ -71,13 +29,10 @@ SetterAction, StorageMode, TransferMode, - codegen_action_for_variable, default_ownership_policy, set_ownership_metadata, ) -from x2py.semantics.ir2ast import semantic_ir_to_codegen_ast as _semantic_ir_to_codegen_ast - from x2py.semantics.models import ( POLICY_COMPLETION_PREPARED_METADATA, MODULE_VARIABLE_INITIALIZER_UNSUPPORTED_BLOCKER, @@ -125,35 +80,10 @@ def parse_pyi_text(source: str, *args, **kwargs): return _parse_pyi_text(f"{CONTRACT_IMPORT}{source}", *args, **kwargs) -def _model_call_names(node, seen=None): - """Collect function-call names from a generated model subtree.""" - if seen is None: - seen = set() - node_id = id(node) - if node_id in seen: - return - seen.add(node_id) - if isinstance(node, FunctionCall): - yield str(node.func_name) - for attr in getattr(node, "_attribute_nodes", ()): - value = getattr(node, attr) - if isinstance(value, tuple | list): - for item in value: - yield from _model_call_names(item, seen) - elif value is not None: - yield from _model_call_names(value, seen) - - def _scalar_type(name: str = "Int32") -> SemanticType: return SemanticType(name=name, dtype=name) -def semantic_ir_to_codegen_ast(node, *args, **kwargs): - if isinstance(node, SemanticModule): - complete_semantic_policies(node) - return _semantic_ir_to_codegen_ast(node, *args, **kwargs) - - def _string_type() -> SemanticType: return SemanticType(name="String", dtype="String") @@ -274,7 +204,6 @@ def _native_array_policy( "ADDRESS_ROLE_PROJECTION", "ADDRESS_ROLE_RAW", "MODULE_VARIABLE_INITIALIZER_UNSUPPORTED_BLOCKER", - "NIL", "POLICY_COMPLETION_PREPARED_METADATA", "PROJECTED_OUTPUT_METADATA", "PYTHON_EXPORTS_METADATA", @@ -288,32 +217,12 @@ def _native_array_policy( "ArrayInteropPolicy", "ArrayInteropPolicyDispatcher", "AssignmentMode", - "BindCAccessorModuleVariable", - "BindCArrayType", - "BindCFunctionDef", - "BindCNativeArrayDescriptorType", - "BindCNativeArrayHandleProperty", - "BindCNativeArrayHandleVariable", - "BindCPointer", - "CCodePrinter", - "CFIDescriptorType", - "CPythonBindingGenerator", - "CPythonCodePrinter", "CodegenAction", - "Declare", "DestructionPolicy", - "FCodePrinter", - "FortranToCBridgeGenerator", - "FunctionDef", - "FunctionDefArgument", - "FunctionDefResult", - "IndexedElement", "NativeArrayBuildRequirement", "NativeArrayHandlePolicyDispatcher", "NativeBarrierAction", "NativeBarrierDispatcher", - "NumpyFloat64Type", - "NumpyNDArrayType", "ObjectKind", "OwnershipContext", "OwnershipDecision", @@ -324,9 +233,6 @@ def _native_array_policy( "PyiPrinter", "PythonBarrierAction", "PythonBarrierDispatcher", - "PythonObjectType", - "Return", - "Scope", "SemanticArgument", "SemanticClass", "SemanticConstraint", @@ -338,30 +244,22 @@ def _native_array_policy( "SetterAction", "StorageMode", "TransferMode", - "Variable", "_address_type", "_array_type", "_derived_type", "_hidden_output_context", - "_model_call_names", "_native_array_policy", "_read_only_argument_context", "_scalar_storage_type", "_scalar_type", - "_semantic_ir_to_codegen_ast", "_string_storage_type", "_string_type", "_writable_argument_context", - "codegen_action_for_variable", "complete_semantic_policies", - "convert_to_literal", "default_ownership_policy", - "native_array_descriptor_argument_type", "native_array_descriptor_kind", "native_array_handle_build_requirements", "parse_pyi_text", "pytest", - "replace", - "semantic_ir_to_codegen_ast", "set_ownership_metadata", ) diff --git a/tests/_shared/parser_property_support.py b/tests/_shared/parser_property_support.py index 33bdbc91f..5d7dd1e28 100644 --- a/tests/_shared/parser_property_support.py +++ b/tests/_shared/parser_property_support.py @@ -30,7 +30,7 @@ from x2py.pipeline.pyi import pyi_text_to_semantic_module as parse_pyi_text -from x2py.codegen.printers.pyi_printer import emit_module_stubs +from x2py.wrapper_codegen.printers import emit_module_stubs from x2py import FortranParseError, parse_fortran_file diff --git a/tests/_shared/pyi_conversion_support.py b/tests/_shared/pyi_conversion_support.py index 9502fd839..3f858a661 100644 --- a/tests/_shared/pyi_conversion_support.py +++ b/tests/_shared/pyi_conversion_support.py @@ -59,19 +59,13 @@ from x2py.pyi_parser import parse_pyi_text as parse_pyi_ast_text -from x2py.semantics.ir2ast import semantic_ir_to_codegen_ast as _semantic_ir_to_codegen_ast - from x2py.semantics.native_contract import native_contract_issues from x2py.semantics.policy_completion import complete_semantic_policies from x2py.semantics.readiness import assess_semantic_wrap_readiness -from x2py.codegen.bindings.c_to_python import CPythonBindingGenerator - -from x2py.codegen.printers.pyi_printer import emit_module - -from x2py.codegen.scope import Scope +from x2py.wrapper_codegen.printers import emit_module from tests._shared.fixture_outputs import FORTRAN_DATA_DIR, FORTRAN_SUFFIXES @@ -87,12 +81,6 @@ ) -def semantic_ir_to_codegen_ast(node, *args, **kwargs): - if isinstance(node, SemanticModule): - complete_semantic_policies(node) - return _semantic_ir_to_codegen_ast(node, *args, **kwargs) - - def _sample_pyi_compare_fixtures(paths: list[Path]) -> list[Path]: by_dir: dict[str, list[Path]] = {} for path in paths: @@ -150,10 +138,8 @@ def _semantic_modules_for_source(path: Path): "PYTHON_VALUE_MUTABILITY_METADATA", "SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA", "USER_PRIVATE_METADATA", - "CPythonBindingGenerator", "Path", "ProjectionMapping", - "Scope", "SemanticArgument", "SemanticConstraint", "SemanticField", @@ -187,5 +173,4 @@ def _semantic_modules_for_source(path: Path): "pyi_text_to_semantic_module", "pytest", "re", - "semantic_ir_to_codegen_ast", ) diff --git a/tests/architecture/test_dependency_boundaries.py b/tests/architecture/test_dependency_boundaries.py index 72f653b86..3bb297de8 100644 --- a/tests/architecture/test_dependency_boundaries.py +++ b/tests/architecture/test_dependency_boundaries.py @@ -1,4 +1,4 @@ -"""Structural contracts for navigable codegen classes.""" +"""Structural contracts for navigable wrapper-generation boundaries.""" from __future__ import annotations @@ -6,32 +6,25 @@ from tests.wrapper.fortran._support import REPO_ROOT -CODEGEN_ROOT = REPO_ROOT / "x2py" / "codegen" -BOUNDARY_DIRS = ("bridges", "bindings", "printers") +WRAPPER_CODEGEN_ROOT = REPO_ROOT / "x2py" / "wrapper_codegen" +BOUNDARY_MODULES = ( + ("c", WRAPPER_CODEGEN_ROOT / "c" / "binding.py"), + ("fortran", WRAPPER_CODEGEN_ROOT / "fortran" / "bridge.py"), + ("printers", WRAPPER_CODEGEN_ROOT / "printers" / "pyi_printer.py"), + ("printers", WRAPPER_CODEGEN_ROOT / "printers" / "source_printers.py"), +) PUBLIC_MODULE_FUNCTIONS = { - ("bindings", "cpython_api.py", "C_to_Python"), - ("bindings", "numpy_cpython_api.py", "get_numpy_max_acceptable_version_file"), ("printers", "pyi_printer.py", "emit_module"), ("printers", "pyi_printer.py", "emit_module_stubs"), ("printers", "pyi_printer.py", "opaque_dependency_modules"), } -SHARED_PRIVATE_FUNCTIONS = { - ("bindings", "c_concepts.py", "_is_string_literal"), -} - - -def _boundary_modules(): - """Yield each Python module in the codegen boundaries under review.""" - for directory in BOUNDARY_DIRS: - for path in sorted((CODEGEN_ROOT / directory).glob("*.py")): - yield directory, path -def test_codegen_boundary_callables_are_documented(): - """Require every boundary function and method to state its contract.""" +def test_wrapper_codegen_boundary_entrypoints_and_visitors_are_documented(): + """Require public entrypoints and dispatched model visitors to state their contract.""" missing = [] - for _, path in _boundary_modules(): - tree = ast.parse(path.read_text(), filename=str(path)) + for _, path in BOUNDARY_MODULES: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) for node in tree.body: if isinstance(node, ast.FunctionDef) and ast.get_docstring(node) is None: missing.append(f"{path.name}:{node.lineno}:{node.name}") @@ -39,17 +32,19 @@ def test_codegen_boundary_callables_are_documented(): missing.extend( f"{path.name}:{method.lineno}:{node.name}.{method.name}" for method in node.body - if isinstance(method, ast.FunctionDef) and ast.get_docstring(method) is None + if isinstance(method, ast.FunctionDef) + and (not method.name.startswith("_") or method.name.startswith("_visit_")) + and ast.get_docstring(method) is None ) - assert not missing, "Undocumented codegen callables:\n" + "\n".join(missing) + assert not missing, "Undocumented wrapper-codegen callables:\n" + "\n".join(missing) -def test_codegen_uses_one_model_visitor_protocol(): - """Prevent legacy printer and extractor dispatch protocols from returning.""" +def test_wrapper_codegen_uses_one_model_visitor_protocol(): + """Prevent alternate printer and extractor dispatch protocols from appearing.""" invalid = [] lowercase_model_names = {"int", "str", "tuple"} - for _, path in _boundary_modules(): - tree = ast.parse(path.read_text(), filename=str(path)) + for _, path in BOUNDARY_MODULES: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) for node in ast.walk(tree): if isinstance(node, ast.FunctionDef) and node.name.startswith(("_print_", "_extract_")): invalid.append(f"{path.name}:{node.lineno}:{node.name}") @@ -60,31 +55,13 @@ def test_codegen_uses_one_model_visitor_protocol(): assert not invalid, "Use _visit_* handlers or named helpers:\n" + "\n".join(invalid) -def test_public_methods_precede_internal_methods(): - """Keep each class's real public API above its visitors and helpers.""" - misplaced = [] - for _, path in _boundary_modules(): - tree = ast.parse(path.read_text(), filename=str(path)) - for class_node in (node for node in tree.body if isinstance(node, ast.ClassDef)): - private_seen = False - for method in (node for node in class_node.body if isinstance(node, ast.FunctionDef)): - is_public = not method.name.startswith("_") - if is_public and private_seen: - misplaced.append(f"{path.name}:{method.lineno}:{class_node.name}.{method.name}") - is_dunder = method.name.startswith("__") and method.name.endswith("__") - if method.name.startswith("_") and not is_dunder: - private_seen = True - assert not misplaced, "Public methods below internal methods:\n" + "\n".join(misplaced) - - -def test_module_functions_are_deliberate_boundary_apis_or_shared_utilities(): +def test_module_functions_are_deliberate_boundary_apis(): """Keep stateful generation logic on its owning class.""" unexpected = [] - allowed = PUBLIC_MODULE_FUNCTIONS | SHARED_PRIVATE_FUNCTIONS - for directory, path in _boundary_modules(): - tree = ast.parse(path.read_text(), filename=str(path)) + for area, path in BOUNDARY_MODULES: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) for node in (node for node in tree.body if isinstance(node, ast.FunctionDef)): - key = (directory, path.name, node.name) - if key not in allowed: + key = (area, path.name, node.name) + if key not in PUBLIC_MODULE_FUNCTIONS: unexpected.append(f"{path.name}:{node.lineno}:{node.name}") - assert not unexpected, "Unexpected module-level codegen functions:\n" + "\n".join(unexpected) + assert not unexpected, "Unexpected module-level wrapper-codegen functions:\n" + "\n".join(unexpected) diff --git a/tests/architecture/test_package_structure.py b/tests/architecture/test_package_structure.py index 0036cebaf..2a829350d 100644 --- a/tests/architecture/test_package_structure.py +++ b/tests/architecture/test_package_structure.py @@ -4,7 +4,7 @@ PACKAGE_ROOT = Path(__file__).parents[2] / "x2py" -ROOT_PYTHON_MODULES = {"__init__.py", "__main__.py", "cli.py"} +ROOT_PYTHON_MODULES = {"__init__.py", "__main__.py", "cli.py", "stage_values.py"} def test_x2py_root_contains_only_public_entrypoint_modules(): diff --git a/tests/architecture/test_test_suite_layout.py b/tests/architecture/test_test_suite_layout.py index f7123844c..bfed2b928 100644 --- a/tests/architecture/test_test_suite_layout.py +++ b/tests/architecture/test_test_suite_layout.py @@ -2,8 +2,8 @@ from __future__ import annotations -from pathlib import Path import re +from pathlib import Path REPO_ROOT = Path(__file__).parents[2] @@ -15,9 +15,7 @@ "architecture", "benchmarks", "cli", - "codegen", "docs", - "lowering", "naming", "parsing", "pipeline", @@ -26,6 +24,7 @@ "semantics", "tools", "types", + "wrapper_codegen", } NON_STAGE_DIRECTORIES = {"wrapper"} DOCUMENTED_SOURCE_OWNERS = { @@ -34,10 +33,7 @@ "x2py.pyi_parser": "tests/parsing/pyi/", "x2py.probes": "tests/probes/", "x2py.pipeline": "tests/pipeline/", - "x2py.semantics.ir2ast": "tests/lowering/", - "x2py.codegen.bridges": "tests/codegen/bridges/", - "x2py.codegen.bindings": "tests/codegen/bindings/", - "x2py.codegen.printers": "tests/codegen/printers/", + "x2py.wrapper_codegen": "tests/wrapper_codegen/", "x2py.naming": "tests/naming/", "x2py.types": "tests/types/", "x2py.runtime.handles": "tests/runtime/handles/", @@ -55,6 +51,7 @@ "test_dependency_boundaries.py", "test_package_structure.py", "test_test_suite_layout.py", + "test_visitor_protocol.py", } DEPRECATED_PYTEST_ROOTS = { TEST_ROOT / "parser", diff --git a/tests/lowering/test_visitor_protocol.py b/tests/architecture/test_visitor_protocol.py similarity index 50% rename from tests/lowering/test_visitor_protocol.py rename to tests/architecture/test_visitor_protocol.py index 05ee7e8b3..d8d1e3cdb 100644 --- a/tests/lowering/test_visitor_protocol.py +++ b/tests/architecture/test_visitor_protocol.py @@ -1,4 +1,4 @@ -"""Structural contract for class-based parser, semantics, and codegen visitors.""" +"""Structural contract for the active parser, semantic, and wrapper visitors.""" from __future__ import annotations @@ -7,63 +7,60 @@ from tests.wrapper.fortran._support import REPO_ROOT from x2py.c_parser.parser import CParser -from x2py.codegen.bindings.c_to_python import CPythonBindingGenerator -from x2py.codegen.bridges.fortran_to_c import FortranToCBridgeGenerator -from x2py.codegen.generator import _Generator -from x2py.codegen.printers.codeprinter import CodePrinter -from x2py.codegen.printers.pyi_printer import PyiPrinter from x2py.fortran_parser.parser import FortranParser, SourceUnit, _SOURCE_UNIT_TYPES from x2py.semantics.c2ir import CToIRConverter from x2py.semantics.fortran2ir import FortranToIRConverter, _FortranVariableContextVisitor -from x2py.semantics.ir2ast import _SemanticIrToCodegenAstVisitor from x2py.semantics.pyi2ir import _ClassBodyVisitor, _ModuleVisitor -from x2py.utilities.visitor import ClassVisitor +from x2py.utilities.visitor import ClassVisitor as SemanticClassVisitor +from x2py.wrapper_codegen.c.binding import CBindingGenerator +from x2py.wrapper_codegen.fortran.bridge import FortranBridgeGenerator +from x2py.wrapper_codegen.planner import WrapperPlanner +from x2py.wrapper_codegen.printers import PyiPrinter +from x2py.wrapper_codegen.support import WrapperPlanSupportAnalyzer +from x2py.wrapper_codegen.visitor import ClassVisitor as WrapperClassVisitor -VISITOR_CLASSES = ( +SEMANTIC_VISITORS = ( FortranParser, CToIRConverter, FortranToIRConverter, _FortranVariableContextVisitor, - _SemanticIrToCodegenAstVisitor, _ClassBodyVisitor, _ModuleVisitor, - _Generator, - CodePrinter, PyiPrinter, - CPythonBindingGenerator, - FortranToCBridgeGenerator, ) - +WRAPPER_VISITORS = ( + WrapperPlanSupportAnalyzer, + WrapperPlanner, + CBindingGenerator, + FortranBridgeGenerator, +) VISITOR_IMPLEMENTATION_PATHS = ( REPO_ROOT / "x2py" / "fortran_parser" / "parser.py", REPO_ROOT / "x2py" / "semantics" / "c2ir.py", REPO_ROOT / "x2py" / "semantics" / "fortran2ir.py", - REPO_ROOT / "x2py" / "semantics" / "ir2ast.py", REPO_ROOT / "x2py" / "semantics" / "pyi2ir.py", - REPO_ROOT / "x2py" / "codegen" / "generator.py", - REPO_ROOT / "x2py" / "codegen" / "bindings" / "c_to_python.py", - REPO_ROOT / "x2py" / "codegen" / "bridges" / "fortran_to_c.py", - REPO_ROOT / "x2py" / "codegen" / "printers" / "codeprinter.py", - REPO_ROOT / "x2py" / "codegen" / "printers" / "pyi_printer.py", + REPO_ROOT / "x2py" / "wrapper_codegen" / "planner.py", + REPO_ROOT / "x2py" / "wrapper_codegen" / "support.py", + REPO_ROOT / "x2py" / "wrapper_codegen" / "c" / "binding.py", + REPO_ROOT / "x2py" / "wrapper_codegen" / "fortran" / "bridge.py", + REPO_ROOT / "x2py" / "wrapper_codegen" / "printers" / "pyi_printer.py", ) -def test_model_visitors_share_one_class_visitor_base(): - """Route every model visitor through the shared MRO dispatcher.""" - assert all(issubclass(visitor, ClassVisitor) for visitor in VISITOR_CLASSES) - +def test_active_model_visitors_use_their_owned_dispatch_protocol(): + assert all(issubclass(visitor, SemanticClassVisitor) for visitor in SEMANTIC_VISITORS) + assert all(issubclass(visitor, WrapperClassVisitor) for visitor in WRAPPER_VISITORS) -def test_class_visitor_supports_configured_handler_prefix(): - """Allow specialized visitors to use names such as ``_print_``.""" +def test_semantic_class_visitor_supports_configured_handler_prefix(): class Node: pass class SpecificNode(Node): pass - class ParserVisitor(ClassVisitor): + class ParserVisitor(SemanticClassVisitor): visitor_method_prefix = "_parse" @staticmethod @@ -73,11 +70,25 @@ def _parse_Node(node): assert ParserVisitor()._visit(SpecificNode()) == "SpecificNode" -def test_model_visitor_handlers_use_configured_class_names(): - """Reject the stdlib-style ``visit_Class`` protocol and lowercase handlers.""" +def test_wrapper_class_visitor_supports_configured_handler_prefix(): + class Node: + pass + + class SpecificNode(Node): + pass + + class PlanVisitor(WrapperClassVisitor): + @staticmethod + def _plan_Node(node): + return type(node).__name__ + + assert PlanVisitor(method_prefix="_plan").visit(SpecificNode()) == "SpecificNode" + + +def test_active_visitor_handlers_use_configured_class_names(): invalid = [] lowercase_model_names = {"int", "str", "tuple"} - for visitor in VISITOR_CLASSES: + for visitor in (*SEMANTIC_VISITORS, *WRAPPER_VISITORS): handler_prefix = f"{visitor.visitor_method_prefix}_" for name, _method in inspect.getmembers(visitor, predicate=inspect.isfunction): if name.startswith("visit_"): @@ -90,26 +101,7 @@ def test_model_visitor_handlers_use_configured_class_names(): assert not invalid, "Use configured _ handlers:\n" + "\n".join(invalid) -def test_ir2ast_visitor_methods_own_their_conversion_bodies(): - """Keep semantic lowering in the visitor instead of one-line module-private shims.""" - invalid = [] - for name in ( - "_visit_SemanticModule", - "_visit_ProcedureOverloadSet", - "_visit_SemanticFunction", - "_visit_SemanticClass", - "_visit_SemanticArgument", - "_visit_SemanticVariable", - ): - source = inspect.getsource(getattr(_SemanticIrToCodegenAstVisitor, name)) - for forbidden in ("_convert_", "_codegen_callback_argument("): - if forbidden in source: - invalid.append(f"_SemanticIrToCodegenAstVisitor.{name} uses {forbidden}") - assert not invalid, "Move visitor conversion bodies onto _SemanticIrToCodegenAstVisitor:\n" + "\n".join(invalid) - - def test_parser_entrypoints_are_not_misnamed_as_visitors(): - """Keep source parsing under ``parse_*`` and reserve visitors for model nodes.""" invalid = [ f"{parser.__name__}.{name}" for parser in (FortranParser, CParser) @@ -120,7 +112,6 @@ def test_parser_entrypoints_are_not_misnamed_as_visitors(): def test_fortran_source_unit_classes_have_matching_handlers(): - """Require every sliced grammar-unit class to have one matching visitor.""" assert all(issubclass(unit_type, SourceUnit) for unit_type in _SOURCE_UNIT_TYPES.values()) assert { kind: f"_visit_{unit_type.__name__}" @@ -129,12 +120,11 @@ def test_fortran_source_unit_classes_have_matching_handlers(): } == {} -def test_shared_visitor_is_the_only_mro_dispatch_implementation(): - """Prevent local visitor loops from bypassing ``ClassVisitor``.""" +def test_visitors_do_not_reimplement_mro_dispatch(): invalid = [] for path in VISITOR_IMPLEMENTATION_PATHS: tree = ast.parse(path.read_text(), filename=str(path)) for node in ast.walk(tree): if isinstance(node, ast.Attribute) and node.attr in {"__mro__", "mro"}: invalid.append(f"{path.relative_to(REPO_ROOT)}:{node.lineno}") - assert not invalid, "Use x2py.utilities.visitor.ClassVisitor instead of local MRO dispatch:\n" + "\n".join(invalid) + assert not invalid, "Use the owned ClassVisitor instead of local MRO dispatch:\n" + "\n".join(invalid) diff --git a/tests/benchmarks/test_parser_benchmarks.py b/tests/benchmarks/test_parser_benchmarks.py index c4cf209ea..27f0ef5a3 100644 --- a/tests/benchmarks/test_parser_benchmarks.py +++ b/tests/benchmarks/test_parser_benchmarks.py @@ -8,7 +8,7 @@ from x2py.c_parser import parse_c_file from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules -from x2py.codegen.printers.pyi_printer import emit_module_stubs +from x2py.wrapper_codegen.printers import emit_module_stubs from x2py import parse_fortran_file pytestmark = pytest.mark.skip(reason="Benchmarks are parked until benchmark adoption resumes.") diff --git a/tests/cli/test_readiness_reports.py b/tests/cli/test_readiness_reports.py index 5597ef11c..d4c6ecd85 100644 --- a/tests/cli/test_readiness_reports.py +++ b/tests/cli/test_readiness_reports.py @@ -319,7 +319,7 @@ def serialize(received): monkeypatch.setattr(x2py_cli, "_parse_c_project", parse_project) monkeypatch.setattr(x2py_cli, "c_project_to_semantic_modules", convert) monkeypatch.setattr(x2py_cli, "expand_c_paths", expand) - monkeypatch.setattr("x2py.codegen.printers.pyi_printer.emit_module_stubs", emit) + monkeypatch.setattr("x2py.wrapper_codegen.printers.emit_module_stubs", emit) monkeypatch.setattr(x2py_cli, "asdict", serialize) assert x2py_cli._semantic_report( @@ -437,7 +437,7 @@ def serialize(module): monkeypatch.setattr(x2py_cli, "_fortran_wrapped_derived_types", wrapped) monkeypatch.setattr(x2py_cli, "_fortran_compile_time_values", compile_values) monkeypatch.setattr(x2py_cli, "fortran_file_to_semantic_modules", convert) - monkeypatch.setattr("x2py.codegen.printers.pyi_printer.emit_module_stubs", emit) + monkeypatch.setattr("x2py.wrapper_codegen.printers.emit_module_stubs", emit) monkeypatch.setattr(x2py_cli, "asdict", serialize) assert x2py_cli._semantic_report(["api"], config) == { diff --git a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py b/tests/codegen/bindings/test_binding_handle_policy_dispatch.py deleted file mode 100644 index d290d24d1..000000000 --- a/tests/codegen/bindings/test_binding_handle_policy_dispatch.py +++ /dev/null @@ -1,538 +0,0 @@ -"""Tests split by stable ownership concept from `test_handle_policy_dispatch.py`.""" - -from tests._shared.ownership_policy_support import ( - BindCArrayType, - BindCFunctionDef, - BindCNativeArrayDescriptorType, - BindCNativeArrayHandleVariable, - BindCPointer, - CCodePrinter, - CFIDescriptorType, - CPythonBindingGenerator, - CPythonCodePrinter, - CodegenAction, - DestructionPolicy, - FortranToCBridgeGenerator, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - IndexedElement, - NIL, - NativeBarrierAction, - NumpyFloat64Type, - NumpyNDArrayType, - ObjectKind, - PythonBarrierAction, - PythonObjectType, - Scope, - SetterAction, - Variable, - _model_call_names, - _native_array_policy, - _semantic_ir_to_codegen_ast, - complete_semantic_policies, - convert_to_literal, - parse_pyi_text, - pytest, -) - - -def test_bridge_and_binding_generators_expose_ownership_action_maps(): - assert ( - CPythonBindingGenerator._RESULT_DETAIL_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY) - ] - == "_snapshot_copy_result_detail_lines" - ) - assert ( - CPythonBindingGenerator._RESULT_POLICY_DISPATCHER.handlers[(ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY)] - == "_convert_snapshot_policy_scalar_result" - ) - assert ( - CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER.handlers[PythonBarrierAction.SCALAR_STORAGE] - == "_convert_python_scalar_storage_argument" - ) - assert ( - CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER.handlers[PythonBarrierAction.STRING_VALUE] - == "_convert_python_string_value_argument" - ) - assert ( - CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER.handlers[PythonBarrierAction.STRING_STORAGE] - == "_convert_python_string_storage_argument" - ) - assert ( - FortranToCBridgeGenerator._NATIVE_BARRIER_DISPATCHER.handlers[NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS] - == "_convert_native_call_local_address_argument" - ) - assert ( - FortranToCBridgeGenerator._NATIVE_BARRIER_DISPATCHER.handlers[NativeBarrierAction.PASS_RAW_ADDRESS] - == "_convert_native_raw_address_argument" - ) - assert ( - CPythonBindingGenerator._ARGUMENT_CAST_GUARD_DISPATCHER.handlers[ - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT) - ] - == "_append_unchecked_argument_cast" - ) - assert ( - CPythonBindingGenerator._ARGUMENT_CAST_GUARD_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT) - ] - == "_append_replacement_argument_cast" - ) - assert ( - CPythonBindingGenerator._RESULT_NOTE_DISPATCHER.handlers[(ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT)] - == "_copy_return_result_notes" - ) - assert FortranToCBridgeGenerator._NDARRAY_RESULT_DISPATCHER.handlers == { - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_build_snapshot_copy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_build_borrowed_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_build_copy_return_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_build_copy_return_array_result", - } - assert ( - FortranToCBridgeGenerator._ALLOCATABLE_RESULT_HELPER_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT) - ] - == "_uses_heap_allocatable_result_helper" - ) - native_array_handle_keys = { - ("allocatable", "argument_descriptor"), - ("allocatable", "borrowed_field_descriptor"), - ("allocatable", "borrowed_module_descriptor"), - ("allocatable", "optional_absent_handle"), - ("allocatable", "owned_result_descriptor"), - ("pointer", "argument_descriptor"), - ("pointer", "borrowed_field_descriptor"), - ("pointer", "borrowed_module_descriptor"), - ("pointer", "optional_absent_handle"), - } - assert set(FortranToCBridgeGenerator._NATIVE_ARRAY_HANDLE_DISPATCHER.handlers) == native_array_handle_keys - assert set(CPythonBindingGenerator._NATIVE_ARRAY_HANDLE_DISPATCHER.handlers) == native_array_handle_keys - assert FortranToCBridgeGenerator._ARRAY_INTEROP_POLICY_DISPATCHER.handlers == { - ("argument", "data_buffer"): "_bridge_data_buffer_argument", - ("argument", "descriptor"): "_bridge_descriptor_argument", - ("module_variable", "data_buffer"): "_bridge_data_buffer_module_variable", - ("module_variable", "descriptor"): "_bridge_descriptor_module_variable", - ("result", "data_buffer"): "_bridge_data_buffer_result", - ("result", "descriptor"): "_bridge_descriptor_result", - } - assert CPythonBindingGenerator._ARRAY_INTEROP_POLICY_DISPATCHER.handlers == { - ("argument", "data_buffer"): "_bind_data_buffer_argument", - ("argument", "descriptor"): "_bind_descriptor_argument", - ("result", "data_buffer"): "_bind_data_buffer_result", - ("result", "descriptor"): "_bind_descriptor_result", - } - assert ( - FortranToCBridgeGenerator._NATIVE_ARRAY_HANDLE_DISPATCHER.handlers[ - ("allocatable", "borrowed_module_descriptor") - ] - == "_bridge_borrowed_native_array_module_handle" - ) - assert ( - CPythonBindingGenerator._NATIVE_ARRAY_HANDLE_DISPATCHER.handlers[("pointer", "borrowed_module_descriptor")] - == "_bind_borrowed_native_array_module_handle" - ) - dispatchers = ( - (FortranToCBridgeGenerator, "_NATIVE_BARRIER_DISPATCHER"), - (FortranToCBridgeGenerator, "_FUNCTION_ARGUMENT_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_RESULT_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_REPLACEMENT_RESULT_DISPATCHER"), - (FortranToCBridgeGenerator, "_NDARRAY_RESULT_DISPATCHER"), - (FortranToCBridgeGenerator, "_ALLOCATABLE_RESULT_HELPER_DISPATCHER"), - (FortranToCBridgeGenerator, "_FIELD_SETTER_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_FIELD_GETTER_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_MODULE_VARIABLE_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_MODULE_ARRAY_GETTER_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_NATIVE_ARRAY_HANDLE_DISPATCHER"), - (FortranToCBridgeGenerator, "_CALLBACK_ARGUMENT_POLICY_DISPATCHER"), - (FortranToCBridgeGenerator, "_CALLBACK_RESULT_POLICY_DISPATCHER"), - (CPythonBindingGenerator, "_PYTHON_BARRIER_DISPATCHER"), - (CPythonBindingGenerator, "_ARGUMENT_DETAIL_DISPATCHER"), - (CPythonBindingGenerator, "_ARGUMENT_CAST_GUARD_DISPATCHER"), - (CPythonBindingGenerator, "_RESULT_POLICY_DISPATCHER"), - (CPythonBindingGenerator, "_RESULT_DETAIL_DISPATCHER"), - (CPythonBindingGenerator, "_RESULT_NOTE_DISPATCHER"), - (CPythonBindingGenerator, "_PROPERTY_SETTER_POLICY_DISPATCHER"), - (CPythonBindingGenerator, "_BORROWED_GETTER_POLICY_DISPATCHER"), - (CPythonBindingGenerator, "_NATIVE_ARRAY_HANDLE_DISPATCHER"), - (CPythonBindingGenerator, "_NATIVE_ARRAY_DESCRIPTOR_ARGUMENT_DISPATCHER"), - (CPythonBindingGenerator, "_ARGUMENT_RETURN_PROJECTION_DISPATCHER"), - (CPythonBindingGenerator, "_PROJECTED_ARGUMENT_OBJECT_DISPATCHER"), - (CPythonBindingGenerator, "_ARRAY_ACCESS_VALIDATION_DISPATCHER"), - (CPythonBindingGenerator, "_ARRAY_RELEASE_POLICY_DISPATCHER"), - ) - for generator, dispatcher_name in dispatchers: - dispatcher = getattr(generator, dispatcher_name) - assert dispatcher.handlers - assert all(hasattr(generator, handler_name) for handler_name in dispatcher.handlers.values()) - - assert set(FortranToCBridgeGenerator._NATIVE_BARRIER_DISPATCHER.handlers) == { - NativeBarrierAction.PASS_VALUE, - NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, - NativeBarrierAction.PASS_STORAGE_ADDRESS, - NativeBarrierAction.PASS_RAW_ADDRESS, - NativeBarrierAction.PASS_ARRAY_BUFFER, - NativeBarrierAction.PASS_WRAPPER_ADDRESS, - } - assert set(CPythonBindingGenerator._PYTHON_BARRIER_DISPATCHER.handlers) == { - PythonBarrierAction.SCALAR_VALUE, - PythonBarrierAction.SCALAR_STORAGE, - PythonBarrierAction.ARRAY_STORAGE, - PythonBarrierAction.STRING_VALUE, - PythonBarrierAction.STRING_STORAGE, - PythonBarrierAction.RAW_ADDRESS, - PythonBarrierAction.WRAPPER_INSTANCE, - } - assert ( - FortranToCBridgeGenerator._RESULT_POLICY_DISPATCHER.handlers.keys() - == CPythonBindingGenerator._RESULT_POLICY_DISPATCHER.handlers.keys() - ) - assert ( - CPythonBindingGenerator._ARGUMENT_RETURN_PROJECTION_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT, True) - ] - == "_project_native_argument_return" - ) - assert ( - CPythonBindingGenerator._PROJECTED_ARGUMENT_OBJECT_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True) - ] - == "_record_projected_argument_object" - ) - assert ( - CPythonBindingGenerator._ARGUMENT_RETURN_PROJECTION_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, True) - ] - == "_project_visible_argument_return" - ) - assert ( - CPythonBindingGenerator._PROJECTED_ARGUMENT_OBJECT_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, True) - ] - == "_record_projected_argument_object" - ) - assert ( - CPythonBindingGenerator._ARRAY_ACCESS_VALIDATION_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT) - ] - == "_writable_array_access_validation" - ) - assert ( - FortranToCBridgeGenerator._FIELD_SETTER_POLICY_DISPATCHER.handlers[SetterAction.WRITE_THROUGH] - == "_build_field_setter" - ) - assert ( - FortranToCBridgeGenerator._FIELD_SETTER_POLICY_DISPATCHER.handlers[SetterAction.REJECT_REPLACEMENT] - == "_skip_field_setter" - ) - assert ( - FortranToCBridgeGenerator._FIELD_GETTER_POLICY_DISPATCHER.handlers[ - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW) - ] - == "_append_borrowed_array_field_getter" - ) - assert ( - FortranToCBridgeGenerator._FIELD_GETTER_POLICY_DISPATCHER.handlers[ - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY) - ] - == "_append_nullable_scalar_field_getter" - ) - assert ( - FortranToCBridgeGenerator._MODULE_VARIABLE_POLICY_DISPATCHER.handlers[ - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY) - ] - == "_scalar_module_variable" - ) - assert ( - FortranToCBridgeGenerator._MODULE_VARIABLE_POLICY_DISPATCHER.handlers[ - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW) - ] - == "_derived_module_variable" - ) - assert ( - CPythonBindingGenerator._ARRAY_RELEASE_POLICY_DISPATCHER.handlers[DestructionPolicy.PYTHON_REFCOUNT] - == "_release_python_owned_array_memory" - ) - assert ( - CPythonBindingGenerator._ARRAY_RELEASE_POLICY_DISPATCHER.handlers[DestructionPolicy.BLOCKED] - == "_blocked_array_release_policy" - ) - - -def test_native_array_handle_binding_builds_runtime_handle_from_named_generated_ops(): - module = parse_pyi_text( - """ -values: Allocatable[Float64[:]] -""", - module_name="native_handle_binding_substrate", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - variable = lowered.variables[0] - op_original = FunctionDef( - "__x2py_values_shape", - (), - (), - FunctionDefResult(NIL), - scope=lowered.scope, - ) - op_wrapper = BindCFunctionDef( - "bind_c___x2py_values_shape", - (), - (), - FunctionDefResult(NIL), - original_function=op_original, - scope=lowered.scope, - ) - handle_variable = variable.clone( - variable.name, - new_class=BindCNativeArrayHandleVariable, - operation_functions={"shape": op_wrapper}, - original_variable=variable, - ) - - binding = CPythonBindingGenerator("", 0) - binding.scope = lowered.scope - binding._native_array_handle_owner_module = Variable(PythonObjectType(), "mod", memory_handling="alias") - body = binding._visit_BindCNativeArrayHandleVariable(handle_variable) - - call_names = {name for node in body for name in _model_call_names(node)} - assert { - "PyDict_New", - "PyDict_SetItem", - "PyImport_ImportModule", - "PyObject_CallObject", - "PyObject_GetAttrString", - } <= call_names - assert binding._python_object_map[handle_variable].name - assert handle_variable.operation_functions == {"shape": op_wrapper} - - -def test_pointer_descriptor_view_operation_wrapper_decodes_generated_cfi_descriptor_pointer(): - scope = Scope(name="descriptor_view_operation", scope_type="module") - policy = _native_array_policy( - descriptor_kind="pointer", - handle_kind="borrowed_module_descriptor", - to_numpy="descriptor_view", - descriptor_interop="pointer_c_descriptor", - operations=("associated", "nullify", "to_numpy"), - ) - original = FunctionDef( - "__x2py_target_to_numpy", - (), - (), - FunctionDefResult(NIL), - scope=scope, - ) - descriptor_arg = Variable(BindCPointer(), "descriptor", is_argument=True, memory_handling="alias") - operation = BindCFunctionDef( - "bind_c___x2py_target_to_numpy", - (FunctionDefArgument(descriptor_arg),), - (), - FunctionDefResult(NIL), - original_function=original, - scope=scope, - ) - source_variable = Variable( - NumpyNDArrayType.get_new(NumpyFloat64Type(), 1, "F"), - "target", - native_array_handle_policy=policy, - ) - handle_variable = source_variable.clone( - source_variable.name, - new_class=BindCNativeArrayHandleVariable, - operation_functions={"to_numpy": operation}, - original_variable=source_variable, - ) - binding = CPythonBindingGenerator("", 0) - binding.scope = scope - - wrapped = binding._native_array_descriptor_view_operation_wrapper(handle_variable, operation) - call_names = set(_model_call_names(wrapped.body)) - code = CPythonCodePrinter("test.c", verbose=0)._visit(wrapped) - - assert "bind_c___x2py_target_to_numpy" in call_names - assert { - "PyDict_New", - "PyDict_SetItem", - "PyLong_FromLongLong", - "PyLong_FromVoidPtr", - } <= call_names - assert "CFI_CDESC_T(1) target_descriptor_storage" in code - assert "CFI_establish(target_descriptor, NULL, CFI_attribute_pointer, CFI_type_double" in code - assert "bind_c___x2py_target_to_numpy(target_descriptor)" in code - assert "((CFI_cdesc_t*)target_descriptor)->base_addr" in code - assert "((CFI_cdesc_t*)target_descriptor)->dim[INT64_C(0)].sm" in code - - -def test_native_array_handle_operation_wrapper_uses_descriptor_reader_only_for_pointer_descriptor_view(): - policy = _native_array_policy( - descriptor_kind="pointer", - handle_kind="borrowed_module_descriptor", - to_numpy="descriptor_view", - descriptor_interop="pointer_c_descriptor", - operations=("associated", "nullify", "to_numpy"), - ) - variable = Variable( - NumpyNDArrayType.get_new(NumpyFloat64Type(), 1, "F"), - "target", - native_array_handle_policy=policy, - ) - binding = CPythonBindingGenerator("", 0) - - assert binding._uses_native_array_descriptor_view_operation_wrapper(variable, "to_numpy") is True - assert binding._uses_native_array_descriptor_view_operation_wrapper(variable, "associated") is False - assert ( - binding._uses_native_array_descriptor_view_operation_wrapper( - variable.clone( - variable.name, - native_array_handle_policy=_native_array_policy( - descriptor_kind="pointer", - handle_kind="borrowed_module_descriptor", - to_numpy="unsupported", - operations=("associated", "nullify", "to_numpy"), - ), - ), - "to_numpy", - ) - is False - ) - - -def test_cfi_descriptor_type_printing_is_local_to_descriptor_reader_path(): - descriptor = Variable(CFIDescriptorType(), "descriptor", memory_handling="alias") - printer = CCodePrinter("test.c", verbose=0) - - assert printer._get_declare_type(descriptor) == "CFI_cdesc_t*" - assert "ISO_Fortran_binding" in printer.get_additional_imports() - - -@pytest.mark.parametrize( - ("descriptor_kind", "annotation"), - [ - ("allocatable", "Allocatable[Float64[:]]"), - ("pointer", "Pointer[Float64[:]]"), - ], -) -def test_native_array_optional_handle_argument_binding_uses_presence_tuple( - descriptor_kind, - annotation, -): - module = parse_pyi_text( - f""" -def maybe(values: {annotation} | None = ...) -> None: ... -""", - module_name=f"{descriptor_kind}_optional_handle_argument_binding", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - argument = lowered.funcs[0].arguments[0] - policy = argument.var.native_array_handle_policy - - assert policy.descriptor_kind == descriptor_kind - assert policy.handle_kind == "optional_absent_handle" - assert policy.optional_absent is True - - bridge = FortranToCBridgeGenerator("", 0) - bridge.scope = lowered.funcs[0].scope - bridged = bridge._convert_argument(argument, lowered.funcs[0]) - - descriptor_arg = bridged["c_arg"].var - descriptor_dummy = lowered.funcs[0].scope.collect_tuple_element( - IndexedElement(descriptor_arg.new_var, convert_to_literal(0)), - ) - presence_var = lowered.funcs[0].scope.collect_tuple_element( - IndexedElement(descriptor_arg.new_var, convert_to_literal(1)), - ) - assert descriptor_arg.class_type is BindCNativeArrayDescriptorType.get_new(has_presence=True) - assert descriptor_dummy.native_array_handle_policy is policy - assert descriptor_dummy.is_optional is True - assert descriptor_dummy.memory_handling == ("alias" if descriptor_kind == "pointer" else "heap") - assert bridged["optional_presence_var"] is presence_var - assert presence_var.is_argument is True - assert bridged["body"] == [] - - binding = CPythonBindingGenerator("", 0) - binding.scope = lowered.funcs[0].scope - collect_arg = Variable(PythonObjectType(), "py_values", memory_handling="alias") - converted = binding._convert_argument( - argument.var, - collect_arg, - bound_argument=False, - is_bind_c_argument=False, - ) - - assert converted["args"][0].class_type is BindCNativeArrayDescriptorType.get_new(has_presence=True) - assert converted["owns_type_check"] is True - assert len(converted["default_init"]) == 2 - assert len(converted["body"]) > 0 - - -def test_native_array_descriptor_argument_binding_forwards_fixed_rank_one_extent(): - module = parse_pyi_text( - """ -def fill(values: Allocatable[Float64[2]]) -> None: ... -""", - module_name="allocatable_handle_fixed_extent_binding", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - argument = lowered.funcs[0].arguments[0] - binding = CPythonBindingGenerator("", 0) - - assert binding._rank_one_fixed_extent(argument.var) == 2 - - -def test_normal_array_bind_c_argument_binding_uses_native_handle_fallback(): - module = parse_pyi_text( - """ -def fill(values: Float64[:]) -> None: ... -""", - module_name="normal_array_handle_fallback_binding", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - argument = lowered.funcs[0].arguments[0] - binding = CPythonBindingGenerator("", 0) - binding.scope = lowered.funcs[0].scope - collect_arg = Variable(PythonObjectType(), "py_values", memory_handling="alias") - - converted = binding._convert_argument( - argument.var, - collect_arg, - bound_argument=False, - is_bind_c_argument=True, - ) - - descriptor_type = converted["args"][0].class_type - assert isinstance(descriptor_type, BindCArrayType) - assert descriptor_type.has_rank is False - assert descriptor_type.has_itemsize is False - assert descriptor_type.has_strides is False - assert converted["owns_type_check"] is True - assert len(converted["body"]) == 1 - assert len(converted["body"][0].blocks) == 2 - assert "array_actual_helper" in str(converted["body"][0]) - - -@pytest.mark.parametrize( - "annotation", - [ - "Float64[...]", - "String[8][:]", - ], -) -def test_normal_array_bind_c_argument_binding_keeps_specialized_array_paths(annotation): - module = parse_pyi_text( - f""" -def fill(values: {annotation}) -> None: ... -""", - module_name="specialized_array_binding", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - argument = lowered.funcs[0].arguments[0] - binding = CPythonBindingGenerator("", 0) - - assert binding._bind_c_array_argument_uses_native_handle_fallback(argument.var) is False diff --git a/tests/codegen/bridges/test_bridge_handle_policy_dispatch.py b/tests/codegen/bridges/test_bridge_handle_policy_dispatch.py deleted file mode 100644 index 5d2652ee9..000000000 --- a/tests/codegen/bridges/test_bridge_handle_policy_dispatch.py +++ /dev/null @@ -1,331 +0,0 @@ -"""Tests split by stable ownership concept from `test_handle_policy_dispatch.py`.""" - -from tests._shared.ownership_policy_support import ( - BindCNativeArrayDescriptorType, - BindCNativeArrayHandleProperty, - BindCNativeArrayHandleVariable, - BindCPointer, - CCodePrinter, - CPythonBindingGenerator, - CPythonCodePrinter, - CodegenAction, - Declare, - FCodePrinter, - FortranToCBridgeGenerator, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - IndexedElement, - PythonObjectType, - Return, - Scope, - Variable, - _model_call_names, - _semantic_ir_to_codegen_ast, - complete_semantic_policies, - convert_to_literal, - parse_pyi_text, - pytest, -) - - -@pytest.mark.parametrize( - ("descriptor_kind", "annotation", "expected_operations"), - [ - ( - "allocatable", - "Allocatable[Float64[:]]", - { - "aligned", - "allocated", - "array_actual", - "deallocate", - "descriptor", - "native_byte_order", - "resize", - "shape", - "to_numpy", - "writeable", - }, - ), - ( - "pointer", - "Pointer[Float64[:]]", - { - "aligned", - "array_actual", - "associated", - "contiguous", - "descriptor", - "native_byte_order", - "nullify", - "shape", - "writeable", - }, - ), - ], -) -def test_native_array_handle_module_variable_bridge_uses_completed_handle_policy_dispatch( - descriptor_kind, - annotation, - expected_operations, -): - module = parse_pyi_text( - f""" -values: {annotation} -""", - module_name=f"{descriptor_kind}_native_handle_bridge_dispatch", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - generator = FortranToCBridgeGenerator("", 0) - generator.scope = lowered.scope - - handle = generator._visit_Variable(lowered.variables[0]) - - assert isinstance(handle, BindCNativeArrayHandleVariable) - assert handle.native_array_handle_policy.handle_kind == "borrowed_module_descriptor" - assert handle.native_array_handle_policy.descriptor_kind == descriptor_kind - assert handle.original_variable is lowered.variables[0] - assert set(handle.operation_functions) == expected_operations - - -def test_native_array_handle_module_operations_print_and_wrap_pointer_handoff_results(): - module = parse_pyi_text( - """ -values: Allocatable[Float64[:]] -""", - module_name="native_handle_module_operations", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - bridged = FortranToCBridgeGenerator("", 0)._visit_Module(lowered) - - fortran_code = FCodePrinter("native_handle_module_operations.f90", verbose=0)._visit(bridged) - assert "function bind_c_private__x2py_values_shape()" in fortran_code - assert "function bind_c_private__x2py_values_array_actual()" in fortran_code - assert "function bind_c_private__x2py_values_descriptor()" in fortran_code - assert "values_descriptor = c_null_ptr" in fortran_code - - cpython_module = CPythonBindingGenerator("", 0)._visit_Module(bridged) - c_code = CPythonCodePrinter("native_handle_module_operations.c", verbose=0)._visit(cpython_module) - assert "_native_array_handle_from_generated_ops" in c_code - assert "PyLong_FromVoidPtr" in c_code - assert "private__x2py_values_array_actual" in c_code - assert "private__x2py_values_descriptor" in c_code - assert "static PyObject* bind_c_private__x2py_values_array_actual(void)" not in c_code - assert "static PyObject* bind_c_private__x2py_values_descriptor(void)" not in c_code - assert "static PyObject* bind_c_private__x2py_values_array_actual_wrapper" in c_code - assert "static PyObject* bind_c_private__x2py_values_descriptor_wrapper" in c_code - - -def test_native_array_descriptor_view_reader_builds_runtime_mapping_from_cfi_descriptor_fields(): - binding = CPythonBindingGenerator("", 0) - binding.scope = Scope(name="descriptor_reader", scope_type="function") - descriptor_pointer = Variable(BindCPointer(), "descriptor", memory_handling="alias") - descriptor_result = binding._new_python_object("descriptor_view") - - body = binding._native_array_descriptor_view_body(descriptor_pointer, descriptor_result, rank=2) - - call_names = {name for node in body for name in _model_call_names(node)} - assert { - "PyDict_New", - "PyDict_SetItem", - "PyList_Append", - "PyList_New", - "PyLong_FromLongLong", - "PyLong_FromVoidPtr", - "PyUnicode_FromString", - } <= call_names - - -def test_native_array_descriptor_view_reader_prints_cfi_descriptor_access_without_global_requirement(): - binding = CPythonBindingGenerator("", 0) - scope = Scope(name="descriptor_reader", scope_type="function") - binding.scope = scope - descriptor_pointer = Variable(BindCPointer(), "descriptor", memory_handling="alias", is_argument=True) - descriptor_result = binding._new_python_object("descriptor_view") - body = binding._native_array_descriptor_view_body(descriptor_pointer, descriptor_result, rank=2) - body.append(Return(descriptor_result)) - function = FunctionDef( - "decode_descriptor", - (FunctionDefArgument(descriptor_pointer),), - body, - FunctionDefResult(descriptor_result), - scope=scope, - ) - - printer = CPythonCodePrinter("test.c", verbose=0) - code = printer._visit(function) - - assert "((CFI_cdesc_t*)descriptor)->base_addr" in code - assert "((CFI_cdesc_t*)descriptor)->elem_len" in code - assert "((CFI_cdesc_t*)descriptor)->rank" in code - assert "((CFI_cdesc_t*)descriptor)->dim[INT64_C(0)].lower_bound" in code - assert "((CFI_cdesc_t*)descriptor)->dim[INT64_C(1)].extent" in code - assert "((CFI_cdesc_t*)descriptor)->dim[INT64_C(1)].sm" in code - assert "PyLong_FromVoidPtr" in code - assert "PyLong_FromLongLong" in code - assert "ISO_Fortran_binding" in printer.get_additional_imports() - - -@pytest.mark.parametrize( - ("descriptor_kind", "annotation"), - [ - ("allocatable", "Allocatable[Float64[:]]"), - ("pointer", "Pointer[Float64[:]]"), - ], -) -def test_native_array_handle_field_generation_uses_completed_handle_policy_dispatch( - descriptor_kind, - annotation, -): - module = parse_pyi_text( - f""" -class box: - values: {annotation} -""", - module_name=f"{descriptor_kind}_handle_field_dispatch", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - field = lowered.classes[0].attributes[0] - policy = field.native_array_handle_policy - - assert policy.descriptor_kind == descriptor_kind - assert policy.handle_kind == "borrowed_field_descriptor" - assert policy.origin == "derived_field" - assert policy.owner_retention == "parent_wrapper" - assert policy.blocker is None - - bridged = FortranToCBridgeGenerator("", 0)._visit_Module(lowered) - wrapped_field = bridged.classes[0].attributes[0] - - assert isinstance(wrapped_field, BindCNativeArrayHandleProperty) - assert wrapped_field.owner_class is lowered.classes[0] - assert wrapped_field.native_array_handle_policy is policy - expected_operations = { - "aligned", - "array_actual", - "descriptor", - "native_byte_order", - "shape", - "writeable", - "allocated" if descriptor_kind == "allocatable" else "associated", - "deallocate" if descriptor_kind == "allocatable" else "nullify", - "resize" if descriptor_kind == "allocatable" else "descriptor", - } - if descriptor_kind == "pointer": - expected_operations.add("contiguous") - assert expected_operations <= set(wrapped_field.operation_functions) - - fortran_code = FCodePrinter("field_handle.f90", verbose=0)._visit(bridged) - assert "self%values" in fortran_code - if descriptor_kind == "allocatable": - assert "bound_values = c_loc(values(lbound(values," in fortran_code - assert "kind=i64)))" in fortran_code - - cpython_module = CPythonBindingGenerator("", 0)._visit_Module(bridged) - c_code = CPythonCodePrinter("field_handle.c", verbose=0)._visit(cpython_module) - assert "_native_array_handle_from_generated_ops" in c_code - assert "values_handle_getter" in c_code - - -def test_native_array_handle_result_generation_uses_completed_handle_policy_dispatch(): - module = parse_pyi_text( - """ -def make_values() -> Allocatable[Float64[:]]: ... -""", - module_name="native_handle_result_dispatch", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - bridged = FortranToCBridgeGenerator("", 0)._visit_Module(lowered) - fortran_code = FCodePrinter("owned_result.f90", verbose=0)._visit(bridged) - assert "if (allocated(make_values" in fortran_code - assert "deallocate(make_values" in fortran_code - - cpython_module = CPythonBindingGenerator("", 0)._visit_Module(bridged) - c_code = CPythonCodePrinter("owned_result.c", verbose=0)._visit(cpython_module) - assert "sizeof(CFI_CDESC_T(1))" in c_code - assert "CFI_attribute_allocatable" in c_code - assert "CFI_allocate(" in c_code - assert "CFI_deallocate(" in c_code - assert 'PyUnicode_FromString("owned")' in c_code - assert "_native_array_handle_from_generated_ops" in c_code - - -@pytest.mark.parametrize( - ("descriptor_kind", "annotation"), - [ - ("allocatable", "Allocatable[Float64[:]]"), - ( - "pointer", - "Annotated[Pointer[Float64[:]], PointerPolicy(nullable=True, transfer='call_local', " - "target_owner='caller', lifetime='call', deallocation='deallocate_resize', " - "shape_source='pointer_bounds', contiguity='contiguous', reassociation='allocate_resize', " - "aliasing='descriptor', mutability='mutable')]", - ), - ], -) -def test_native_array_handle_argument_generation_uses_completed_handle_policy_dispatch( - descriptor_kind, - annotation, -): - module = parse_pyi_text( - f""" -def fill(values: {annotation}) -> Returns["values", {annotation}]: ... -""", - module_name=f"{descriptor_kind}_handle_argument_dispatch", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - argument = lowered.funcs[0].arguments[0] - decision = argument.var.ownership_decision - policy = argument.var.native_array_handle_policy - - assert decision.codegen_action is CodegenAction.IN_PLACE_ARGUMENT - assert decision.mutates_native is True - assert decision.projects_result is True - assert policy.descriptor_kind == descriptor_kind - assert policy.handle_kind == "argument_descriptor" - assert policy.output_projection == "projected_handle" - assert policy.blocker is None - - bridge = FortranToCBridgeGenerator("", 0) - bridge.scope = lowered.funcs[0].scope - bridged = bridge._convert_argument(argument, lowered.funcs[0]) - - descriptor_arg = bridged["c_arg"].var - descriptor_dummy = lowered.funcs[0].scope.collect_tuple_element( - IndexedElement(descriptor_arg.new_var, convert_to_literal(0)), - ) - f_printer = FCodePrinter("test.f90", verbose=0) - f_printer.set_scope(lowered.funcs[0].scope) - f_printer._kind = lambda expr: "f64" - fortran_declaration = f_printer._visit(Declare(descriptor_dummy)) - assert descriptor_arg.class_type is BindCNativeArrayDescriptorType.get_new(has_presence=False) - assert descriptor_dummy.native_array_handle_policy is policy - assert descriptor_dummy.memory_handling == ("alias" if descriptor_kind == "pointer" else "heap") - assert descriptor_dummy.is_argument is True - assert bridged["body"] == [] - assert bridged["optional_presence_var"] is None - assert CCodePrinter("test.c", verbose=0)._get_declare_type(descriptor_dummy) == "void*" - assert (", pointer" if descriptor_kind == "pointer" else ", allocatable") in fortran_declaration - assert f_printer._fortran_argument_access(descriptor_dummy) == "readwrite" - assert "values" in str(bridged["f_arg"]) - - binding = CPythonBindingGenerator("", 0) - binding.scope = lowered.funcs[0].scope - collect_arg = Variable(PythonObjectType(), "py_values", memory_handling="alias") - converted = binding._convert_argument( - argument.var, - collect_arg, - bound_argument=False, - is_bind_c_argument=False, - ) - - assert converted["args"][0].class_type is BindCNativeArrayDescriptorType.get_new(has_presence=False) - assert converted["owns_type_check"] is True - assert len(converted["body"]) > 0 - assert len(converted["default_init"]) == 1 diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index a2c6f0d03..a08d3ff44 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -216,7 +216,6 @@ "x2py/fortran_parser/README.md", "x2py/pyi_parser/README.md", "x2py/semantics/README.md", - "x2py/codegen/README.md", "x2py/compiling/README.md", ] SOURCE_NAVIGATION_HOTSPOTS = [ @@ -239,19 +238,15 @@ "x2py/pipeline/pyi.py", "x2py/semantics/policy_completion.py", "x2py/semantics/readiness.py", - "x2py/semantics/ir2ast.py", - "x2py/codegen/binding_pipeline.py", - "x2py/codegen/bridges/fortran_to_c.py", - "x2py/codegen/bindings/c_to_python.py", - "x2py/codegen/bindings/cpython_api.py", - "x2py/codegen/bindings/numpy_cpython_api.py", - "x2py/codegen/printers/fcode.py", - "x2py/codegen/printers/ccode.py", - "x2py/codegen/printers/cpythoncode.py", - "x2py/codegen/printers/pyi_printer.py", + "x2py/wrapper_codegen/plan.py", + "x2py/wrapper_codegen/planner.py", + "x2py/wrapper_codegen/generator.py", + "x2py/wrapper_codegen/c/binding.py", + "x2py/wrapper_codegen/fortran/bridge.py", + "x2py/wrapper_codegen/printers/pyi_printer.py", + "x2py/wrapper_codegen/printers/source_printers.py", "x2py/compiling/basic.py", "x2py/compiling/compilers.py", - "x2py/compiling/python_wrapper.py", "x2py/compiling/runtime_support.py", "x2py/naming/policy.py", "x2py/stdlib/", @@ -296,9 +291,8 @@ "tests/semantics/conversion/c/", "tests/semantics/readiness/test_c_readiness.py", "tests/semantics/conversion/fortran/", - "tests/lowering/test_semantic_ir.py", - "tests/codegen/printers/", - "tests/codegen/printers/test_modern_example.py", + "tests/wrapper_codegen/printers/", + "tests/wrapper_codegen/printers/test_modern_example.py", "tests/semantics/readiness/", "tests/docs/test_examples.py", "tests/docs/test_structure.py", @@ -382,7 +376,7 @@ "x2py/c_parser/", "x2py/fortran_parser/", "x2py/semantics/", - "x2py/codegen/", + "x2py/wrapper_codegen/", "x2py/compiling/", ] PACKAGE_READMES = [ @@ -390,7 +384,6 @@ "x2py/c_parser/README.md", "x2py/fortran_parser/README.md", "x2py/semantics/README.md", - "x2py/codegen/README.md", "x2py/compiling/README.md", ] ARCHIVED_OLD_DOCS = [ diff --git a/tests/lowering/test_array_interop_policy.py b/tests/lowering/test_array_interop_policy.py deleted file mode 100644 index f57efc9d1..000000000 --- a/tests/lowering/test_array_interop_policy.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Tests split by stable ownership concept from `test_handle_policy_dispatch.py`.""" - -from tests._shared.ownership_policy_support import ( - Scope, - _semantic_ir_to_codegen_ast, - complete_semantic_policies, - parse_pyi_text, -) - - -def test_lowering_attaches_one_array_interop_policy_for_data_buffer_and_descriptor_lanes(): - module = parse_pyi_text( - """ -def consume_array(values: Float64[:]) -> Float64[:]: ... -def consume_handle(values: Allocatable[Float64[:]]) -> None: ... -""", - module_name="array_interop_policy_selectors", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - array_function = lowered.funcs[0] - handle_function = lowered.funcs[1] - - array_argument_policy = array_function.arguments[0].var.array_interop_policy - array_result_policy = array_function.results.var.array_interop_policy - handle_argument = handle_function.arguments[0].var - handle_argument_policy = handle_argument.array_interop_policy - - assert array_argument_policy.abi == "data_buffer" - assert array_argument_policy.descriptor_kind is None - assert array_result_policy.abi == "data_buffer" - assert handle_argument_policy.abi == "descriptor" - assert handle_argument_policy.descriptor_kind == "allocatable" - assert handle_argument_policy.handle_kind == "argument_descriptor" - assert handle_argument.native_array_handle_policy.descriptor_kind == "allocatable" diff --git a/tests/lowering/test_semantic_ir.py b/tests/lowering/test_semantic_ir.py deleted file mode 100644 index 2ee912d18..000000000 --- a/tests/lowering/test_semantic_ir.py +++ /dev/null @@ -1,908 +0,0 @@ -from pathlib import Path - -import pytest - -from x2py import parse_fortran_file -from x2py.contracts import CONTRACT_SYMBOLS -from x2py.codegen.models.core import ClassDef, FunctionOverloadSet -from x2py.codegen.models.datatypes import ( - CharType, - CustomDataType, - NIL, - NumpyFloat64Type, - NumpyInt64Type, - NumpyNDArrayType, -) -from x2py.codegen.scope import Scope -from x2py.semantics.ownership import CodegenAction, NativeBarrierAction, SetterAction, TransferMode -from x2py.semantics.fortran2ir import fortran_module_to_semantic_module -from x2py.semantics.ir2ast import semantic_ir_to_codegen_ast as _semantic_ir_to_codegen_ast -from x2py.semantics.models import RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA, SemanticModule -from x2py.semantics.policy_completion import complete_semantic_policies -from x2py.pipeline.pyi import pyi_text_to_semantic_module as _parse_pyi_text - - -WRAPPER_FORTRAN_DATA = Path(__file__).parents[1] / "data" / "fortran" / "wrapper" -FORTRAN_CLASS_SOURCE = WRAPPER_FORTRAN_DATA / "fclasses_f90.f90" -FORTRAN_OPERATOR_SOURCE = WRAPPER_FORTRAN_DATA / "foperators_f90.f90" -CONTRACT_IMPORT = f"from x2py.contracts import {', '.join(sorted(CONTRACT_SYMBOLS))}\n" - - -def parse_pyi_text(source: str, *args, **kwargs): - if "x2py.contracts" in source: - return _parse_pyi_text(source, *args, **kwargs) - return _parse_pyi_text(f"{CONTRACT_IMPORT}{source}", *args, **kwargs) - - -def semantic_ir_to_codegen_ast(node, *args, **kwargs): - if isinstance(node, SemanticModule): - complete_semantic_policies(node) - return _semantic_ir_to_codegen_ast(node, *args, **kwargs) - - -def test_ir_lowering_requires_completed_ownership_policy(): - module = parse_pyi_text( - """ -def scale(values: Float64[:]) -> None: ... -""", - module_name="raw_policy", - ) - - with pytest.raises(ValueError, match="missing completed ownership policy"): - _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - - -def test_ir_lowering_requires_completed_native_array_handle_policy(): - module = parse_pyi_text( - """ -values: Allocatable[Float64[:]] -""", - module_name="raw_native_handle_policy", - ) - complete_semantic_policies(module) - module.variables[0].metadata.pop(RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA) - - with pytest.raises(ValueError, match="missing completed native-array-handle policy"): - _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - - -def test_native_array_handle_policy_lowers_to_codegen_variables(): - module = parse_pyi_text( - """ -values: Allocatable[Float64[:]] - -def make_values() -> Allocatable[Float64[:]]: ... -""", - module_name="native_handle_lowering", - ) - - lowered = semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - values = lowered.variables[0] - make_values = lowered.funcs[0] - - assert values.native_array_handle_policy.handle_kind == "borrowed_module_descriptor" - assert values.native_array_handle_policy.to_numpy == "descriptor_view" - assert values.native_array_handle_policy.descriptor_interop == "module_allocatable_c_descriptor" - assert make_values.results.var.native_array_handle_policy.handle_kind == "owned_result_descriptor" - assert make_values.results.var.native_array_handle_policy.output_projection == "handle_result" - - -def test_pointer_array_result_lowering_blocks_until_returned_handle_policy_is_complete(): - module = parse_pyi_text( - """ -def make_values() -> Pointer[Float64[:]]: ... -""", - module_name="pointer_handle_result_lowering", - ) - - with pytest.raises(ValueError, match="stable owner storage and target lifetime"): - semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - - -def test_immutable_writable_arguments_lower_with_completed_copy_in_out_policy(): - module = parse_pyi_text( - """ -def normalize( - values: Annotated[Float64[:], Immutable] -) -> Returns["values", Float64[:]]: ... -""", - module_name="immutable_values", - ) - - lowered = semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - values = lowered.funcs[0].arguments[0].var - - assert values.ownership_decision.codegen_action is CodegenAction.COPY_IN_OUT - - -def test_character_array_lowering_preserves_element_length_metadata(): - source = """ -module char_array_mod -contains - subroutine use_labels(labels) - character(len=4), intent(in) :: labels(:) - end subroutine use_labels - subroutine replace_names(names) - character(len=:), allocatable, intent(inout) :: names(:) - if (allocated(names)) deallocate(names) - allocate(character(len=5) :: names(2)) - names(1) = 'red' - names(2) = 'blue' - end subroutine replace_names -end module char_array_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - lowered = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - use_labels = next(func for func in lowered.funcs if func.name == "use_labels") - labels = use_labels.arguments[0].var - assert labels.dtype is CharType() - assert labels.fortran_character_length.python_value == 4 - - replace_names = next(func for func in lowered.funcs if func.name == "replace_names") - names = replace_names.arguments[0].var - assert names.dtype is CharType() - assert names.fortran_character_length == ":" - - -def test_modern_fortran_derived_type_and_type_bound_methods_become_codegen_class(): - parsed = parse_fortran_file( - FORTRAN_CLASS_SOURCE.read_text(), - filename=str(FORTRAN_CLASS_SOURCE), - ) - semantic_module = fortran_module_to_semantic_module(parsed) - semantic_vector = next(cls for cls in semantic_module.classes if cls.name == "vector") - semantic_shift = next(method for method in semantic_vector.methods if method.name == "shift") - assert semantic_shift.passed_object_name == "owner" - assert semantic_shift.passed_object_position == 1 - assert semantic_shift.binding_attributes == ("pass(owner)",) - - scope = Scope(name=semantic_module.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(semantic_module, scope) - - assert [str(cls.name) for cls in codegen_module.classes] == [ - "vector", - "vector_store", - ] - vector, vector_store = codegen_module.classes - assert isinstance(vector, ClassDef) - assert str(vector.name) == "vector" - assert isinstance(vector.class_type, CustomDataType) - assert vector.class_type.name == "vector" - - assert [str(attribute.name) for attribute in vector.attributes] == ["x", "y"] - assert all(attribute.class_type is NumpyFloat64Type() for attribute in vector.attributes) - - assert [str(method.name) for method in vector.methods] == ["scale", "shift_vector", "magnitude"] - scale = vector.methods_as_dict["scale"] - self_arg = scale.arguments[0] - assert self_arg.bound_argument - assert self_arg.var.class_type is vector.class_type - assert self_arg.var.cls_base is vector - - shift = vector.methods_as_dict["shift"] - assert vector.scope.get_python_name(shift.name) == "shift" - assert [str(argument.name) for argument in shift.arguments] == ["owner", "dx", "dy"] - assert shift.arguments[0].bound_argument - assert shift.arguments[0].bound_argument_position == 1 - assert shift.arguments[0].var.cls_base is vector - - magnitude = vector.methods_as_dict["magnitude"] - assert magnitude.arguments[0].bound_argument - assert magnitude.results.var.class_type is NumpyFloat64Type() - - assert isinstance(vector_store, ClassDef) - assert isinstance(vector_store.class_type, CustomDataType) - assert vector_store.class_type.name == "vector_store" - assert [str(attribute.name) for attribute in vector_store.attributes] == [ - "values", - "matrix", - ] - values, matrix = vector_store.attributes - assert isinstance(values.class_type, NumpyNDArrayType) - assert values.class_type.element_type is NumpyFloat64Type() - assert values.memory_handling == "heap" - assert isinstance(matrix.class_type, NumpyNDArrayType) - assert matrix.class_type.element_type is NumpyFloat64Type() - assert matrix.class_type.rank == 2 - assert matrix.class_type.order == "F" - assert matrix.memory_handling == "heap" - - assert [vector_store.scope.get_python_name(method.name) for method in vector_store.methods] == [ - "allocate_values", - "set_values", - "allocate_matrix", - "set_matrix", - "make", - ] - allocate_values = vector_store.methods_as_dict["allocate_values"] - assert allocate_values.arguments[0].bound_argument - assert allocate_values.arguments[0].var.class_type is vector_store.class_type - assert allocate_values.arguments[1].var.class_type is NumpyInt64Type() - - set_values = vector_store.methods_as_dict["set_values"] - assert set_values.arguments[0].bound_argument - assert isinstance(set_values.arguments[1].var.class_type, NumpyNDArrayType) - - set_matrix = vector_store.methods_as_dict["set_matrix"] - assert set_matrix.arguments[0].bound_argument - assert isinstance(set_matrix.arguments[1].var.class_type, NumpyNDArrayType) - assert set_matrix.arguments[1].var.class_type.rank == 2 - assert set_matrix.arguments[1].var.class_type.order == "F" - - make = vector_store.methods_as_dict["make"] - assert str(make.name) == "make_vector_store" - assert not make.arguments[0].bound_argument - assert make.arguments[0].var.class_type is NumpyInt64Type() - assert make.arguments[1].var.class_type is NumpyFloat64Type() - assert make.results.var.class_type is vector_store.class_type - - -def test_generic_interfaces_become_module_and_class_function_overload_sets(): - source = """ -module generic_mod - interface convert - module procedure convert_integer, convert_real - end interface convert - type :: box - contains - procedure :: set_integer - procedure :: set_real - generic :: set => set_integer, set_real - end type box -contains - integer function convert_integer(value) - integer :: value - convert_integer = value - end function convert_integer - real function convert_real(value) - real :: value - convert_real = value - end function convert_real - subroutine set_integer(self, value) - class(box) :: self - integer :: value - end subroutine set_integer - subroutine set_real(self, value) - class(box) :: self - real :: value - end subroutine set_real -end module generic_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - assert len(codegen_module.overload_sets) == 1 - assert isinstance(codegen_module.overload_sets[0], FunctionOverloadSet) - assert codegen_module.overload_sets[0].name == "convert" - assert [str(func.name) for func in codegen_module.overload_sets[0].functions] == [ - "convert_integer_0001", - "convert_real_0001", - ] - assert len(codegen_module.classes[0].overload_sets) == 1 - assert codegen_module.classes[0].overload_sets[0].name == "set" - - -def test_indistinguishable_generic_overloads_raise_generation_error(): - source = """ -module generic_mod - interface convert - module procedure convert_first, convert_second - end interface convert -contains - integer function convert_first(value) - integer :: value - convert_first = value - end function convert_first - integer function convert_second(value) - integer :: value - convert_second = value - end function convert_second -end module generic_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - with pytest.raises(ValueError, match="indistinguishable overload"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_unresolved_generic_target_raises_before_codegen(): - source = """ -module generic_mod - interface convert - module procedure missing - end interface convert -end module generic_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - with pytest.raises(ValueError, match=r"missing specific procedure.*missing"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_allocatable_module_array_without_aliased_lowers_as_descriptor_view(): - source = """ -module alloc_mod - real(8), allocatable :: values(:) -end module alloc_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - values = codegen_module.variables[0] - assert values.ownership_decision.codegen_action is CodegenAction.BORROWED_VIEW - assert values.getter_ownership_decision.codegen_action is CodegenAction.BORROWED_VIEW - assert values.native_array_handle_policy.to_numpy == "descriptor_view" - assert values.native_array_handle_policy.descriptor_interop == "module_allocatable_c_descriptor" - - -def test_plain_derived_module_variable_lowers_completed_live_policy(): - module = parse_pyi_text( - """ -class child: - value: Int32 - -class box: - scalar: Int32 - values: Annotated[Allocatable[Float64[:]], Aliased] - nested: child - -current: box -""", - module_name="snapshot_mod", - ) - - codegen_module = semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - - current = codegen_module.variables[0] - assert current.ownership_decision.transfer is TransferMode.BORROWED_VIEW - assert current.ownership_decision.codegen_action is CodegenAction.BORROWED_VIEW - assert current.setter_ownership_decision.setter_action is SetterAction.REJECT_REPLACEMENT - - -def test_allocatable_result_and_output_lower_for_copy_return_codegen(): - source = """ -module alloc_mod -contains - subroutine fill(values) - real(8), allocatable, intent(out) :: values(:) - end subroutine fill - function make_values() result(values) - real(8), allocatable :: values(:) - end function make_values -end module alloc_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - fill = next(function for function in codegen_module.funcs if str(function.name) == "fill") - values = fill.arguments[0].var - assert values.memory_handling == "heap" - assert isinstance(values.class_type, NumpyNDArrayType) - assert values.class_type.rank == 1 - - make_values = next(function for function in codegen_module.funcs if str(function.name) == "make_values") - result = make_values.results.var - assert result.memory_handling == "heap" - assert isinstance(result.class_type, NumpyNDArrayType) - assert result.class_type.rank == 1 - - -def test_allocatable_inout_array_reaches_codegen_as_replacement_argument(): - inout_source = """ -module alloc_mod -contains - subroutine replace(values) - real(8), allocatable, intent(inout) :: values(:) - end subroutine replace -end module alloc_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(inout_source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - replace = next(function for function in codegen_module.funcs if str(function.name) == "replace") - values = replace.arguments[0].var - assert values.memory_handling == "heap" - assert isinstance(values.class_type, NumpyNDArrayType) - assert values.class_type.rank == 1 - - -def test_allocatable_scalar_derived_outputs_raise_before_codegen(): - source = """ -module alloc_scalar_mod - type :: item - integer :: value - end type item -contains - subroutine replace(value) - type(item), allocatable :: value - end subroutine replace -end module alloc_scalar_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match=r"writable scalar descriptors require explicit intent\(out\)"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_scalar_descriptor_function_signature_lowers_as_scalar_boundary(): - module = parse_pyi_text( - """ -@native_call( - [Allocatable(Arg(0)), Pointer(Arg(1))], - result=Pointer(Return(0)), -) -def combine( - scale: Float64 | None, - current: Float64 | None, -) -> Float64 | None: ... -""", - module_name="descriptor_function", - ) - - lowered = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - - combine = lowered.funcs[0] - scale, current = [argument.var for argument in combine.arguments] - assert scale.ownership_decision.codegen_action is CodegenAction.CALL_LOCAL_INPUT - assert scale.ownership_decision.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS - assert current.ownership_decision.codegen_action is CodegenAction.CALL_LOCAL_INPUT - assert current.ownership_decision.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS - assert combine.results.var.ownership_decision.codegen_action is CodegenAction.SNAPSHOT_COPY - assert combine.results.var.ownership_decision.nullable is True - - -def test_bind_c_scalar_without_iso_c_kind_raises_before_codegen(): - source = """ -module bad_bind_mod -contains - integer function unsafe(n) bind(C) result(res) - integer, value :: n - res = n - end function unsafe -end module bad_bind_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match="bind\\(C\\) scalar argument 'n'"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -@pytest.mark.parametrize( - ("shape", "blocker"), - [ - ("", "writable scalar descriptors require explicit intent"), - ("(:)", "pointer array dummy reassociation needs explicit PointerPolicy metadata"), - ], -) -def test_pointer_output_arguments_raise_before_codegen_without_policy(shape, blocker): - source = f""" -module pointer_mod -contains - subroutine attach(values) - real(8), pointer :: values{shape} - end subroutine attach -end module pointer_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match=blocker): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_pointer_module_variable_lowers_with_default_conservative_handle_policy(): - source = """ -module pointer_module_mod - real(8), pointer :: values(:) -end module pointer_module_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - lowered = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - values = lowered.variables[0] - policy = values.native_array_handle_policy - assert policy.descriptor_kind == "pointer" - assert policy.handle_kind == "borrowed_module_descriptor" - assert policy.getter_behavior == "handle" - assert policy.to_numpy == "unsupported" - assert policy.operations == ("associated", "nullify", "to_numpy") - - -def test_pointer_scalar_module_variable_lowers_as_nullable_snapshot_getter(): - source = """ -module pointer_scalar_module_mod - real(8), pointer :: value -end module pointer_scalar_module_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - value = codegen_module.variables[0] - assert value.ownership_decision.codegen_action is CodegenAction.SNAPSHOT_COPY - assert value.getter_ownership_decision.codegen_action is CodegenAction.SNAPSHOT_COPY - assert value.setter_ownership_decision.setter_action.name == "REJECT_REPLACEMENT" - - -def test_allocatable_scalar_module_variable_lowers_as_nullable_snapshot_getter(): - source = """ -module allocatable_scalar_module_mod - real(8), allocatable :: value -end module allocatable_scalar_module_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - value = codegen_module.variables[0] - assert value.ownership_decision.codegen_action is CodegenAction.SNAPSHOT_COPY - assert value.getter_ownership_decision.codegen_action is CodegenAction.SNAPSHOT_COPY - assert value.setter_ownership_decision.setter_action.name == "REJECT_REPLACEMENT" - - -@pytest.mark.parametrize( - ("source", "message"), - [ - ( - """ -module constructor_generic_mod - type :: item - integer :: value - end type item - interface item - module procedure make_item - end interface item -contains - type(item) function make_item(value) result(instance) - integer, intent(in) :: value - instance%value = value - end function make_item -end module constructor_generic_mod -""", - "generic constructor interfaces are not mapped", - ), - ], -) -def test_unsupported_generic_constructor_raises_before_codegen(source, message): - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match=message): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_bind_c_derived_value_uses_fortran_bridge_and_noninteroperable_type_is_rejected(): - interoperable_source = """ -module bind_c_value_mod - use iso_c_binding - type, bind(C) :: point - real(c_double) :: x - end type point -contains - subroutine consume(value) bind(C) - type(point), value :: value - end subroutine consume -end module bind_c_value_mod -""" - noninteroperable_source = ( - interoperable_source.replace("type, bind(C) :: point", "type :: point") - .replace("module bind_c_value_mod", "module bad_bind_c_value_mod") - .replace("end module bind_c_value_mod", "end module bad_bind_c_value_mod") - ) - - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(interoperable_source)) - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - argument = codegen_module.funcs[0].arguments[0].var - - assert isinstance(argument.class_type, CustomDataType) - assert argument.passes_by_value is True - - bad_module = fortran_module_to_semantic_module(parse_fortran_file(noninteroperable_source)) - with pytest.raises(ValueError, match=r"by-value derived-type argument.*not declared bind\(C\)"): - semantic_ir_to_codegen_ast( - bad_module, - Scope(name=bad_module.name, scope_type="module"), - ) - - -def test_scalar_polymorphic_input_arguments_become_dispatch_overload_sets(): - source = """ -module polymorphic_codegen_mod - type :: base - end type base - type, extends(base) :: child - end type child -contains - subroutine accept(value) - class(base), intent(in) :: value - end subroutine accept -end module polymorphic_codegen_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - assert codegen_module.funcs == () - assert len(codegen_module.overload_sets) == 1 - dispatch = codegen_module.overload_sets[0] - assert isinstance(dispatch, FunctionOverloadSet) - assert str(dispatch.name) == "accept" - assert [func.arguments[0].var.class_type.name for func in dispatch.functions] == ["child", "base"] - - -def test_polymorphic_replacement_arguments_raise_before_codegen_without_policy(): - source = """ -module polymorphic_codegen_mod - type :: base - end type base -contains - subroutine replace(value) - class(base) :: value - end subroutine replace -end module polymorphic_codegen_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match="polymorphic argument 'value'"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_non_default_lower_bound_extent_reaches_codegen_shape_validation(): - source = """ -module lower_bound_mod -contains - subroutine scale_lower(n, values) - integer, intent(in) :: n - real(8), intent(inout) :: values(0:n - 1) - end subroutine scale_lower -end module lower_bound_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - scale_lower = next(function for function in codegen_module.funcs if str(function.name) == "scale_lower") - values = scale_lower.arguments[1].var - assert isinstance(values.class_type, NumpyNDArrayType) - assert values.alloc_shape != (None,) - assert "n" in repr(values.alloc_shape[0]) - - -@pytest.mark.parametrize( - ("source", "match"), - [ - ( - """ -module character_array_mod -contains - subroutine inspect(labels) - character(len=4), intent(in) :: labels(:) - end subroutine inspect -end module character_array_mod -""", - None, - ), - ( - """ -module derived_array_mod - type :: item - integer :: value - end type item -contains - subroutine inspect(items) - type(item), intent(in) :: items(:) - end subroutine inspect -end module derived_array_mod -""", - "array of derived type", - ), - ( - """ -module high_rank_mod -contains - subroutine inspect(values) - real(8), intent(in) :: values(:, :, :, :, :, :, :, :, :, :, :, :, :, :, :, :) - end subroutine inspect -end module high_rank_mod -""", - "supports ranks 1 through 15", - ), - ], -) -def test_unsupported_remaining_array_contracts_raise_before_codegen(source, match): - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - if match is None: - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - else: - with pytest.raises(ValueError, match=match): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - -def test_assumed_rank_numeric_array_arguments_lower_with_dispatch_marker(): - source = """ -module assumed_rank_mod -contains - subroutine inspect(values) - real(8), intent(in) :: values(..) - end subroutine inspect -end module assumed_rank_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - inspect = next(function for function in codegen_module.funcs if str(function.name) == "inspect") - values = inspect.arguments[0].var - assert values.assumed_rank is True - assert values.rank == 1 - assert values.alloc_shape == (None,) - - -def test_multiple_allocatable_copy_returns_lower_before_codegen(): - multiple_source = """ -module alloc_mod -contains - subroutine make_pair(left, right) - real(8), allocatable, intent(out) :: left(:) - real(8), allocatable, intent(out) :: right(:) - end subroutine make_pair -end module alloc_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(multiple_source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - make_pair = next(function for function in codegen_module.funcs if str(function.name) == "make_pair") - assert all(argument.var.memory_handling == "heap" for argument in make_pair.arguments) - - -def test_optional_arguments_preserve_status_and_python_defaults_in_codegen_ast(): - source = """ -module optional_mod -contains - subroutine step(tol, dt, values, status) - real(8), intent(in), optional :: tol - integer, intent(in) :: dt - real(8), intent(inout), optional :: values(:) - integer, intent(out) :: status - end subroutine step -end module optional_mod -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - - step = next(function for function in codegen_module.funcs if str(function.name) == "step") - assert [str(argument.name) for argument in step.arguments] == ["dt", "status", "tol", "values"] - assert [argument.var.is_optional for argument in step.arguments] == [False, False, True, True] - assert [argument.has_default for argument in step.arguments] == [False, False, True, True] - assert step.arguments[2].value is NIL - assert step.arguments[3].value is NIL - - -def test_defined_operators_and_assignment_become_named_codegen_overload_sets(): - semantic_module = fortran_module_to_semantic_module( - parse_fortran_file( - FORTRAN_OPERATOR_SOURCE.read_text(), - filename=str(FORTRAN_OPERATOR_SOURCE), - ) - ) - codegen_module = semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) - vector = next(cls for cls in codegen_module.classes if str(cls.name) == "vector") - overload_sets = {item.name: item for item in vector.overload_sets} - - assert overload_sets["__add__"].native_name == "operator(+)" - assert overload_sets["__sub__"].native_name == "operator(-)" - assert set(overload_sets["__eq__"].native_names) == {"operator(==)", "operator(.eqv.)"} - assert overload_sets["operator_dot"].native_name == "operator(.dot.)" - assert overload_sets["assign"].native_name == "assignment(=)" - assert overload_sets["assign"].functions[0].arguments[0].bound_argument - reflected = next( - function for function in overload_sets["__add__"].functions if "add_real_vector" in str(function.name) - ) - assert not reflected.arguments[0].bound_argument - - -def test_indistinguishable_defined_operator_overloads_raise_generation_error(): - source = """ -module ambiguous_operator - type :: box - integer :: value - end type box - interface operator(+) - module procedure add_first, add_second - end interface operator(+) -contains - type(box) function add_first(left, right) - type(box), intent(in) :: left - integer, intent(in) :: right - end function add_first - type(box) function add_second(left, right) - type(box), intent(in) :: left - integer, intent(in) :: right - end function add_second -end module ambiguous_operator -""" - semantic_module = fortran_module_to_semantic_module(parse_fortran_file(source)) - - with pytest.raises(ValueError, match="indistinguishable overload"): - semantic_ir_to_codegen_ast( - semantic_module, - Scope(name=semantic_module.name, scope_type="module"), - ) diff --git a/tests/naming/test_policy.py b/tests/naming/test_policy.py index f98bf9973..abde19bfb 100644 --- a/tests/naming/test_policy.py +++ b/tests/naming/test_policy.py @@ -2,7 +2,6 @@ import pytest -from x2py.codegen.scope import Scope from x2py.naming import NamingPolicy from x2py.naming import normalize_public_name @@ -58,17 +57,3 @@ def test_generated_symbols_apply_target_language_rules(): ) == "value_0001" ) - - -def test_scope_uses_injected_generated_symbol_language(): - python_scope = Scope(name="owner", scope_type="module") - - assert str(python_scope.get_new_name("__add__", object_type="function")) == "__add__" - - scope = Scope(name="owner", scope_type="module", symbol_language="fortran") - - assert str(scope.get_new_name("module", object_type="function")) == "module_0001" - child = scope.new_child_scope("child", "function") - - assert child.symbol_language == "fortran" - assert str(child.get_new_name("value", object_type="variable")) == "value_0001" diff --git a/tests/pipeline/pyi_builds/test_contract_fixtures.py b/tests/pipeline/pyi_builds/test_contract_fixtures.py index c9d77031a..2bf32fd12 100644 --- a/tests/pipeline/pyi_builds/test_contract_fixtures.py +++ b/tests/pipeline/pyi_builds/test_contract_fixtures.py @@ -14,7 +14,7 @@ pyi_files_for_fixture, ) from x2py.pipeline.pyi import pyi_text_to_semantic_module as parse_pyi_text -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from x2py.pipeline import build as build_pipeline from x2py.pipeline.build import _discover_pyi_imports, _pyi_contract_bundle diff --git a/tests/pipeline/test_wrapper_plan_replay.py b/tests/pipeline/test_wrapper_plan_replay.py deleted file mode 100644 index 94abea3f5..000000000 --- a/tests/pipeline/test_wrapper_plan_replay.py +++ /dev/null @@ -1,50 +0,0 @@ -"""Maintainer replay command coverage for retained wrapper-plan artifacts.""" - -from __future__ import annotations - -import shutil -import subprocess -import sys -from pathlib import Path - -import pytest - -from tests.wrapper.fortran._support import REPO_ROOT - - -@pytest.mark.parametrize("entry", ("source", "pyi")) -def test_maintainer_replay_command_retains_dual_route_fmath_evidence(tmp_path: Path, entry: str): - if shutil.which("gfortran") is None: - pytest.skip("gfortran is required for maintained wrapper-plan replay") - - output_dir = tmp_path / entry - result = subprocess.run( - [ - sys.executable, - "-m", - "tools.replay_wrapper_plan", - "--entry", - entry, - "--unit", - "fmath", - "--output-dir", - str(output_dir), - ], - cwd=REPO_ROOT, - capture_output=True, - text=True, - check=False, - ) - - assert result.returncode == 0, result.stderr - assert "Retained wrapper-plan replay:" in result.stdout - assert (output_dir / "legacy").is_dir() - assert (output_dir / "wrapper-plan").is_dir() - - route_report = (output_dir / "route-report.txt").read_text(encoding="utf-8") - assert f"entry={entry}" in route_report - assert "legacy.selected_route=legacy" in route_report - assert "wrapper-plan.selected_route=wrapper-plan" in route_report - assert "artifact_names_equal=true" in route_report - assert "runtime_assertions=passed" in route_report - assert "conversion_failure_parity=passed" in route_report diff --git a/tests/pipeline/test_wrapper_plan_route_selection.py b/tests/pipeline/test_wrapper_plan_route_selection.py deleted file mode 100644 index d4e654176..000000000 --- a/tests/pipeline/test_wrapper_plan_route_selection.py +++ /dev/null @@ -1,856 +0,0 @@ -"""Phase 1C whole-module wrapper-plan route selection tests.""" - -from __future__ import annotations - -import shutil -from pathlib import Path - -import pytest - -from tests._shared.ownership_policy_support import parse_pyi_text -from tests.wrapper.fortran._support import REPO_ROOT, _compile_native_object, wrapper_source -from x2py.pipeline import build as build_pipeline -from x2py.semantics.models import PYTHON_EXPORTS_METADATA -from x2py.semantics.policy_completion import complete_semantic_policies -from x2py.wrapper_codegen import c as wrapper_c -from x2py.wrapper_codegen import generator as wrapper_generator -from x2py.wrapper_codegen import source_printers as wrapper_source_printers - - -def _completed_module(source: str, *, module_name: str): - module = parse_pyi_text(source, module_name=module_name) - complete_semantic_policies(module) - return module - - -def test_route_selector_records_forced_plan_route_and_covered_lanes(): - module = _completed_module( - """ -@bind("SCALE") -@native_call([Addr(Arg(0))]) -def scale(x: Float64) -> Float64: ... -""", - module_name="fmath", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.owner_path == "fmath" - assert decision.selected_route == "wrapper-plan" - assert decision.uses_wrapper_plan is True - assert decision.covered_lanes == ( - "scalar-inputs", - "scalar-direct-results", - "native-call-runtime", - ) - assert decision.blockers == () - assert decision.rollout_eligible is False - assert decision.rollout_evidence == ( - "tests/wrapper/fortran/scalars/test_verified_baseline.py::" - "test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value", - "tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::" - "test_scalar_copy_in_out_returns_replacement_through_both_routes", - "tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::" - "test_whole_scalar_module_variable_behavior_matches_legacy_route", - "tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::" - "test_complete_general_source_preserves_namespaces_through_both_routes", - "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" - "test_compiled_runtime_policies_release_gil_and_project_native_errors", - "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" - "test_pyi_runtime_policies_release_gil_and_project_native_errors", - "tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_multiple_scalar_results_match_both_routes_without_array_blockers", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_fixed_string_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/scalars/test_verified_baseline.py::" - "test_required_array_buffers_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::" - "test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_ordinary_array_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::" - "test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_array_buffers_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_output_arguments.py::" - "test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_raw_array_addresses_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_copy_f_preserves_logical_axes_through_binding_owned_temporary", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_pointers.py::" - "test_module_native_array_handles_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/module_state/test_allocatable_replacement.py::" - "test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_array_results_follow_data_buffer_and_descriptor_handle_contracts", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_route", - "tests/wrapper/fortran/module_state/test_allocatable_views.py::" - "test_scalar_descriptor_module_variables_return_copied_optional_values", - "tests/wrapper/fortran/module_state/test_allocatable_views.py::" - "test_plain_allocatable_module_array_exposes_current_live_view", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_scalar_derived_objects_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_value_copy_and_optional_derived_inputs_match_source_oracle", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_plain_module_derived_proxy_reads_and_writes_live_members", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_aliased_module_derived_object_uses_direct_live_field_handles", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_derived_module_constant_returns_independent_owned_values", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_fixed_string_fields_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_pointer_field_descriptor_views_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_borrowed_child_retains_owner_and_finalizes_exactly_once", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_eligible_derived_contract_selects_production_plan_without_legacy_lowering", - "tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::" - "test_bound_constructor_replaces_field_initialization_and_reuses_method_plan", - "tests/wrapper/fortran/naming/test_phase9_class_overloads.py::" - "test_exact_method_overloads_match_without_trial_calls", - "tests/wrapper/fortran/naming/test_phase9_class_overloads.py::" - "test_constructor_overloads_share_owned_allocation_and_exact_matching", - "tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::" - "test_immediate_callbacks_cover_all_supported_argument_shapes", - "tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::" - "test_callback_exception_prints_traceback_and_aborts_host_process", - ) - assert decision.selection_reason == "wrapper-plan route forced for internal migration verification" - - rendered = build_pipeline._render_selected_wrapper_plan(module) - assert rendered.artifacts.module_name == "fmath" - assert rendered.extension_init_name == "PyInit_fmath" - - -def test_route_selector_selects_core_scalar_module_for_production_plan_rollout(): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... -""", - module_name="scalar_route_without_parity", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == ( - "scalar-inputs", - "scalar-direct-results", - "native-call-runtime", - ) - assert decision.blockers == () - assert decision.rollout_eligible is True - assert decision.selection_reason == "whole generation unit is covered by completed wrapper-plan lanes" - - -@pytest.mark.parametrize( - ("source", "module_name", "covered_lanes"), - ( - ( - """ -@external -@native_call([Addr(Arg(0)), Addr(Arg(1))]) -def optional_value(base: Int32, value: Int32 = ...) -> Int32: ... -""", - "optional_value", - ( - "scalar-inputs", - "scalar-optional-inputs", - "scalar-direct-results", - "native-call-runtime", - ), - ), - ( - """ -@native_call([Allocatable(Arg(0))]) -def descriptor(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... -""", - "optional_descriptor", - ( - "scalar-inputs", - "scalar-optional-inputs", - "scalar-descriptor-inputs", - "scalar-direct-results", - "native-call-runtime", - ), - ), - ( - 'def bump(value: Annotated[Int32, Immutable]) -> Returns["value", Int32]: ...', - "scalar_writeback", - ("scalar-inputs", "scalar-writebacks", "native-call-runtime"), - ), - ), -) -def test_route_selector_accepts_completed_phase3_scalar_lanes_for_forced_whole_module_parity( - source: str, - module_name: str, - covered_lanes: tuple[str, ...], -): - module = _completed_module(source, module_name=module_name) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == covered_lanes - assert decision.blockers == () - - -@pytest.mark.parametrize( - ("source", "module_name", "covered_lanes"), - ( - ( - "def update(value: Float64[()]) -> None: ...", - "scalar_storage_route", - ("scalar-storage-inputs", "void-calls", "native-call-runtime"), - ), - ( - "def update(value: Addr(Float64)) -> None: ...", - "scalar_raw_address_route", - ("scalar-raw-address-inputs", "void-calls", "native-call-runtime"), - ), - ), -) -def test_route_selector_accepts_isolated_scalar_address_boundaries( - source: str, - module_name: str, - covered_lanes: tuple[str, ...], -): - module = _completed_module(source, module_name=module_name) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == covered_lanes - assert decision.blockers == () - - -def test_route_selector_selects_completed_scalar_address_boundaries_in_production(): - module = _completed_module( - """ -def update_storage(value: Float64[()]) -> None: ... -def update_raw(value: Addr(Float64)) -> None: ... -""", - module_name="scalar_address_boundaries", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.rollout_eligible is True - assert decision.covered_lanes == ( - "scalar-storage-inputs", - "void-calls", - "native-call-runtime", - "scalar-raw-address-inputs", - ) - - -def test_route_selector_selects_multiple_scalar_results_in_production(): - module = _completed_module( - """ -@native_call([Addr(Arg(0)), Return("status", 1)]) -def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... -""", - module_name="multiple_scalar_results", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.rollout_eligible is True - assert decision.covered_lanes == ( - "scalar-inputs", - "scalar-direct-results", - "scalar-hidden-outputs", - "scalar-multiple-results", - "native-call-runtime", - ) - - -def test_route_selector_accepts_completed_scalar_module_variable_lane(): - module = _completed_module( - """ -limit: Final[Int32] = 12 -counter: Int32 = 3 -optional_scale: Allocatable[Float64] - -def summarize() -> Int32: ... -""", - module_name="scalar_state", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == ( - "scalar-direct-results", - "native-call-runtime", - "scalar-module-variables", - ) - assert decision.blockers == () - - -def test_route_selector_records_void_call_and_namespace_lanes(): - module = parse_pyi_text( - """ -def ping() -> None: ... -def value(x: Int32) -> Int32: ... -""", - module_name="namespaced_calls", - ) - module.functions[1].metadata[PYTHON_EXPORTS_METADATA] = [{"namespace": ("child",), "name": "value"}] - complete_semantic_policies(module) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == ( - "void-calls", - "native-call-runtime", - "scalar-inputs", - "scalar-direct-results", - "python-namespaces", - ) - - -def test_route_selector_records_completed_native_status_error_lane(): - module = _completed_module( - """ -@raises(status="status", message="message", success=0) -@native_call([Addr(Arg(0)), Return("status", 0), Return("message", 1)]) -def solve(value: Int32) -> tuple[Int32, String[32]]: ... -""", - module_name="runtime_status", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.rollout_eligible is True - assert decision.covered_lanes == ( - "scalar-inputs", - "void-calls", - "native-call-runtime", - "native-status-errors", - ) - - -def test_route_selector_selects_array_buffer_lane_after_native_handle_actuals_are_supported(): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... -def sum_values(values: Float64[:]) -> Float64: ... -""", - module_name="fmath", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.rollout_eligible is True - assert decision.covered_lanes == ( - "scalar-inputs", - "scalar-direct-results", - "native-call-runtime", - "array-buffer-inputs", - "array-native-handle-actuals", - ) - assert decision.blockers == () - assert decision.selection_reason == "whole generation unit is covered by completed wrapper-plan lanes" - - -def test_route_selector_keeps_unimplemented_scalar_kinds_on_legacy_route(): - module = _completed_module( - "def identity(value: Float128) -> Float128: ...", - module_name="wide_scalar", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "legacy" - assert [blocker.reason for blocker in decision.blockers] == [ - "argument 'value' is not a first-lane primitive scalar", - "result is not a first-lane primitive scalar", - ] - - -def test_route_selector_forces_completed_class_units_through_one_plan_route(): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... - -class sample: - value: Int32 - def reset(self) -> None: ... -""", - module_name="fmath", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_wrapper_plan=True, - ) - - assert decision.selected_route == "wrapper-plan" - assert "class-registration" in decision.covered_lanes - assert "instance-methods" in decision.covered_lanes - assert decision.blockers == () - - -def test_route_selector_uses_completed_bind_c_direct_symbol_plan(): - module = _completed_module("def add_one(value: Int32) -> Int32: ...", module_name="bind_c_value") - module.functions[0].metadata["fortran_bind_c"] = True - module.functions[0].metadata["fortran_bind_c_name"] = "solver_add_one" - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.covered_lanes == ( - "scalar-inputs", - "scalar-direct-results", - "native-call-runtime", - ) - assert decision.blockers == () - - -@pytest.mark.parametrize( - ("source", "module_name", "covered_lanes"), - ( - ( - "def label(value: String[3]) -> String[3]: ...", - "fixed_strings", - ("string-value-inputs", "fixed-string-direct-results", "native-call-runtime"), - ), - ( - "def vector() -> Float64[3]: ...", - "array_result", - ("array-direct-results", "native-call-runtime"), - ), - ( - "@native_call([Return('values', 0)])\ndef hidden() -> Float64[3]: ...", - "array_hidden_output", - ("array-hidden-outputs", "native-call-runtime"), - ), - ), -) -def test_route_selector_selects_completed_string_and_array_lanes_for_production( - source: str, - module_name: str, - covered_lanes: tuple[str, ...], -): - module = _completed_module(source, module_name=module_name) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - assert decision.selected_route == "wrapper-plan" - assert decision.rollout_eligible is True - assert decision.covered_lanes == covered_lanes - assert decision.blockers == () - - -@pytest.mark.parametrize( - ("source", "module_name", "covered_lanes"), - ( - ( - 'def fill(values: Float64[:]) -> Returns["values", Float64[:]]: ...', - "array_writeback", - ( - "array-buffer-inputs", - "array-native-handle-actuals", - "array-writebacks", - "native-call-runtime", - ), - ), - ( - "def maybe(values: Float64[:] = ...) -> None: ...", - "optional_array", - ( - "array-buffer-inputs", - "array-handle-actuals-excluded", - "array-optional-inputs", - "void-calls", - "native-call-runtime", - ), - ), - ), -) -def test_route_selector_uses_phase7_handle_parity_or_keeps_explicit_exclusions_legacy( - source: str, - module_name: str, - covered_lanes: tuple[str, ...], -): - module = _completed_module(source, module_name=module_name) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - ) - - expected_route = "legacy" if "array-handle-actuals-excluded" in covered_lanes else "wrapper-plan" - assert decision.selected_route == expected_route - assert decision.rollout_eligible is (expected_route == "wrapper-plan") - assert decision.covered_lanes == covered_lanes - assert decision.blockers == () - if expected_route == "legacy": - assert decision.selection_reason == "covered lanes exceed the recorded wrapper-plan parity evidence" - else: - assert decision.selection_reason == "whole generation unit is covered by completed wrapper-plan lanes" - - -def test_route_selector_keeps_an_explicitly_forced_legacy_module_entirely_legacy(): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... -""", - module_name="fmath", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_legacy=True, - ) - - assert decision.selected_route == "legacy" - assert decision.selection_reason == "legacy route forced for migration rollback or comparison" - - -@pytest.mark.parametrize( - ("makefile", "strict_wrapper_names"), - ((True, False), (False, True)), -) -def test_route_selector_reserves_makefile_and_strict_name_modes_for_legacy( - makefile: bool, - strict_wrapper_names: bool, -): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... -""", - module_name="fmath", - ) - - decision = build_pipeline._select_wrapper_plan_route( - module, - makefile=makefile, - strict_wrapper_names=strict_wrapper_names, - ) - - assert decision.selected_route == "legacy" - assert decision.selection_reason.endswith("mode remains on the legacy route") - - -def test_route_selector_propagates_plan_construction_failure_without_legacy_retry(monkeypatch): - module = _completed_module( - """ -def scale(x: Float64) -> Float64: ... -""", - module_name="fmath", - ) - - class FailingWrapperPlanner: - def __init__(self, **_kwargs): - pass - - def build(self, _module): - raise RuntimeError("planned failure") - - monkeypatch.setattr(build_pipeline, "WrapperPlanner", FailingWrapperPlanner) - - with pytest.raises(RuntimeError, match="planned failure"): - build_pipeline._render_selected_wrapper_plan( - module, - ) - - -def test_source_plan_build_failure_does_not_retry_legacy_lowering(monkeypatch, tmp_path): - if shutil.which("gfortran") is None: - pytest.skip("gfortran is required for Fortran wrapper runtime tests") - - def fail_plan_build(*args, **kwargs): - raise RuntimeError("planned build failure") - - def fail_legacy_lowering(*args, **kwargs): - raise AssertionError("legacy lowering must not run after plan selection") - - monkeypatch.setattr(build_pipeline, "_build_rendered_wrapper_extension", fail_plan_build) - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - - with pytest.raises(RuntimeError, match="planned build failure"): - build_pipeline.build_fortran_extension( - wrapper_source("fmath.f"), - output_dir=tmp_path, - _force_wrapper_plan_route=True, - ) - - -def test_default_source_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): - class FailingWrapperPlanner: - def __init__(self, **_kwargs): - pass - - def build(self, _module): - raise RuntimeError("planned construction failure") - - def fail_legacy_lowering(*args, **kwargs): - raise AssertionError("legacy lowering must not run for the selected source plan route") - - monkeypatch.setattr(build_pipeline, "WrapperPlanner", FailingWrapperPlanner) - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - - with pytest.raises(RuntimeError, match="planned construction failure"): - build_pipeline.build_fortran_extension( - wrapper_source("fmath.f"), - output_dir=tmp_path, - ) - - -def test_source_and_pyi_forced_plan_routes_build_complete_extensions(tmp_path: Path): - if shutil.which("gfortran") is None: - pytest.skip("gfortran is required for Fortran wrapper runtime tests") - - source = wrapper_source("fmath.f") - source_result = build_pipeline.build_fortran_extension( - source, - output_dir=tmp_path / "source_build", - _force_wrapper_plan_route=True, - ) - native_object = _compile_native_object(source, tmp_path / "native") - contract = REPO_ROOT / "tests" / "wrapper" / "fortran" / "scalars" / "contracts" / "fmath" / "__init__.pyi" - pyi_result = build_pipeline.build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / "pyi_build", - _force_wrapper_plan_route=True, - ) - - assert source_result.compiled is True - assert pyi_result.compiled is True - assert source_result.module_name == "fmath" - assert pyi_result.module_name == "fmath" - assert source_result.shared_library.exists() - assert pyi_result.shared_library.exists() - assert pyi_result.manifest is not None - assert pyi_result.manifest["native_build_plan"] == build_pipeline._manifest_native_plan( - pyi_result.native_build_plan, - base=pyi_result.output_dir, - ) - - -def test_pyi_plan_build_failure_does_not_pre_run_or_retry_legacy_lowering(monkeypatch, tmp_path: Path): - def fail_plan_build(*args, **kwargs): - raise RuntimeError("planned build failure") - - def fail_legacy_lowering(*args, **kwargs): - raise AssertionError("legacy lowering must not run for the selected pyi plan route") - - native_object = tmp_path / "fmath.o" - native_object.write_text("placeholder native object\n", encoding="utf-8") - contract = REPO_ROOT / "tests" / "wrapper" / "fortran" / "scalars" / "contracts" / "fmath" / "__init__.pyi" - monkeypatch.setattr(build_pipeline, "_build_rendered_wrapper_extension", fail_plan_build) - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - - with pytest.raises(RuntimeError, match="planned build failure"): - build_pipeline.build_pyi_extension( - contract, - native_objects=[native_object], - output_dir=tmp_path / "build", - _force_wrapper_plan_route=True, - ) - - -def test_default_pyi_plan_construction_failure_does_not_pre_run_legacy_lowering(monkeypatch, tmp_path: Path): - class FailingWrapperPlanner: - def __init__(self, **_kwargs): - pass - - def build(self, _module): - raise RuntimeError("planned construction failure") - - def fail_legacy_lowering(*args, **kwargs): - raise AssertionError("legacy lowering must not run for the selected pyi plan route") - - native_object = tmp_path / "fmath.o" - native_object.write_text("placeholder native object\n", encoding="utf-8") - contract = REPO_ROOT / "tests" / "wrapper" / "fortran" / "scalars" / "contracts" / "fmath" / "__init__.pyi" - monkeypatch.setattr(build_pipeline, "WrapperPlanner", FailingWrapperPlanner) - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - - with pytest.raises(RuntimeError, match="planned construction failure"): - build_pipeline.build_pyi_extension( - contract, - native_objects=[native_object], - output_dir=tmp_path / "build", - ) - - -def _install_plan_failure_seam(monkeypatch, seam: str, message: str) -> None: - """Install one named plan-route failure before generated artifacts exist.""" - if seam == "validator": - monkeypatch.setattr( - wrapper_generator.WrapperCodeGenerator, - "_validate_plan", - lambda _self, _plan: (_ for _ in ()).throw(RuntimeError(message)), - ) - return - if seam == "generator": - monkeypatch.setattr( - wrapper_c.binding.CBindingGenerator, - "visit", - lambda _self, _plan: (_ for _ in ()).throw(RuntimeError(message)), - ) - return - if seam == "printer": - monkeypatch.setattr( - wrapper_source_printers.CSourcePrinter, - "doprint", - lambda _self, _node: (_ for _ in ()).throw(RuntimeError(message)), - ) - return - raise AssertionError(f"Unknown plan-route failure seam: {seam}") - - -def _build_forced_plan_entry(entry: str, tmp_path: Path) -> None: - """Build one source or contract entry through the forced wrapper-plan route.""" - if entry == "source": - build_pipeline.build_fortran_extension( - wrapper_source("fmath.f"), - output_dir=tmp_path / "source_build", - _force_wrapper_plan_route=True, - ) - return - native_object = tmp_path / "fmath.o" - native_object.write_text("placeholder native object\n", encoding="utf-8") - contract = REPO_ROOT / "tests" / "wrapper" / "fortran" / "scalars" / "contracts" / "fmath" / "__init__.pyi" - build_pipeline.build_pyi_extension( - contract, - native_objects=[native_object], - output_dir=tmp_path / "pyi_build", - _force_wrapper_plan_route=True, - ) - - -@pytest.mark.parametrize("entry", ("source", "pyi")) -@pytest.mark.parametrize( - ("seam", "message"), - ( - ("validator", "planned validator failure"), - ("generator", "planned generator failure"), - ("printer", "planned printer failure"), - ), -) -def test_selected_plan_failure_seams_do_not_pre_run_or_retry_legacy_lowering( - monkeypatch, - tmp_path: Path, - entry: str, - seam: str, - message: str, -): - if entry == "source" and shutil.which("gfortran") is None: - pytest.skip("gfortran is required for source-driven wrapper-plan tests") - - def fail_legacy_lowering(*args, **kwargs): - raise AssertionError(f"legacy lowering must not run for selected {entry} {seam} failure") - - _install_plan_failure_seam(monkeypatch, seam, message) - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - - with pytest.raises(RuntimeError, match=message): - _build_forced_plan_entry(entry, tmp_path) diff --git a/tests/runtime/handles/test_array_actual_abi.py b/tests/runtime/handles/test_array_actual_abi.py index 7658b2956..1cd4c0c71 100644 --- a/tests/runtime/handles/test_array_actual_abi.py +++ b/tests/runtime/handles/test_array_actual_abi.py @@ -277,7 +277,7 @@ def test_array_actual_binding_helper_accepts_ndarray_path_with_shared_validation _native_array_actual_for_binding(values, expected_rank=1) with pytest.raises(TypeError, match="expected dtype"): _native_array_actual_for_binding(values, expected_dtype=np.float32) - with pytest.raises(TypeError, match=r"expected shape .* axis 0"): + with pytest.raises(TypeError, match=r"incompatible shape at axis 0"): _native_array_actual_for_binding(values, expected_shape=(1, 3)) with pytest.raises(TypeError, match=r"expected ordering \(C\)"): _native_array_actual_for_binding(values, expected_layout="C") @@ -415,7 +415,7 @@ def test_array_actual_argument_abi_packer_uses_ndarray_data_pointer_and_shape_fi True, True, True, - ) == (values.ctypes.data, 2, values.dtype.itemsize, 2, 3, 2, 3, 1, 1) + ) == (values.ctypes.data, 2, values.dtype.itemsize, 2, 3, 1, 2, 1, 1) def test_array_actual_argument_abi_packer_uses_allocatable_native_array_actual_without_numpy_conversion(): @@ -449,7 +449,7 @@ def test_array_actual_argument_abi_packer_uses_allocatable_native_array_actual_w True, True, True, - ) == (actual.address, 1, np.dtype(np.float64).itemsize, 2, 2, 1) + ) == (actual.address, 1, np.dtype(np.float64).itemsize, 2, 1, 1) assert calls == ["array_actual"] diff --git a/tests/semantics/conversion/_property_support.py b/tests/semantics/conversion/_property_support.py index ef0c394e1..5d37848f8 100644 --- a/tests/semantics/conversion/_property_support.py +++ b/tests/semantics/conversion/_property_support.py @@ -31,7 +31,7 @@ from x2py.pipeline.pyi import pyi_text_to_semantic_module as parse_pyi_text -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from x2py import parse_fortran_file diff --git a/tests/semantics/conversion/c/_support.py b/tests/semantics/conversion/c/_support.py index 2679d4cb5..ef341ea0e 100644 --- a/tests/semantics/conversion/c/_support.py +++ b/tests/semantics/conversion/c/_support.py @@ -80,7 +80,7 @@ from x2py.semantics.readiness import assess_semantic_wrap_readiness -from x2py.codegen.printers.pyi_printer import emit_module, emit_module_stubs +from x2py.wrapper_codegen.printers import emit_module, emit_module_stubs def _function(module, name): diff --git a/tests/semantics/conversion/fortran/_support.py b/tests/semantics/conversion/fortran/_support.py index 7285d5386..d81c8b308 100644 --- a/tests/semantics/conversion/fortran/_support.py +++ b/tests/semantics/conversion/fortran/_support.py @@ -51,7 +51,7 @@ from x2py.semantics.readiness import assess_semantic_wrap_readiness -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from x2py.semantics.models import ( ProjectionMapping, diff --git a/tests/semantics/conversion/pyi/test_calls_and_projections.py b/tests/semantics/conversion/pyi/test_calls_and_projections.py index 03170d83f..50fb3766b 100644 --- a/tests/semantics/conversion/pyi/test_calls_and_projections.py +++ b/tests/semantics/conversion/pyi/test_calls_and_projections.py @@ -7,7 +7,6 @@ CONTRACT_SYMBOLS, PROJECTED_OUTPUT_METADATA, ProjectionMapping, - Scope, SemanticArgument, SemanticConstraint, SemanticFunction, @@ -23,7 +22,6 @@ parse_pyi_text, pytest, re, - semantic_ir_to_codegen_ast, ) @@ -411,12 +409,6 @@ def fill( assert func.arguments[1].metadata[PROJECTED_OUTPUT_METADATA] is True assert func.projection[1].result_position == 0 - codegen_module = semantic_ir_to_codegen_ast( - from_pyi, - Scope(name=from_pyi.name, scope_type="module"), - ) - assert codegen_module.funcs[0].arguments[1].var.projected_output is True - def test_native_order_outputs_do_not_get_projected_without_native_call(): from_pyi = parse_pyi_text( diff --git a/tests/semantics/conversion/pyi/test_classes_and_overloads.py b/tests/semantics/conversion/pyi/test_classes_and_overloads.py index a4def6e22..996fcf006 100644 --- a/tests/semantics/conversion/pyi/test_classes_and_overloads.py +++ b/tests/semantics/conversion/pyi/test_classes_and_overloads.py @@ -2,17 +2,14 @@ from tests._shared.pyi_conversion_support import ( BIND_TARGET_METADATA, - CPythonBindingGenerator, PROJECTED_OUTPUT_METADATA, SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA, - Scope, USER_PRIVATE_METADATA, asdict, emit_module, parse_pyi_text, pytest, re, - semantic_ir_to_codegen_ast, ) @@ -135,12 +132,6 @@ def __init__( assert ' @overload("init_state")\n @native_call' not in emitted assert parse_pyi_text(emitted, module_name="edited") == linked - with pytest.raises(ValueError, match="Constructor overload dispatch is not mapped"): - semantic_ir_to_codegen_ast( - linked, - Scope(name=linked.name, scope_type="module"), - ) - def test_convert_pyi_to_ir_removed_constructor_suppresses_keyword_initializer(): module = parse_pyi_text( @@ -157,14 +148,6 @@ class state: assert cls.origin.metadata[SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA] is True assert "def __init__" not in emit_module(module) - codegen_module = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - codegen_cls = codegen_module.classes[0] - assert codegen_cls.decorators[SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA] is True - assert CPythonBindingGenerator._suppresses_default_class_initialiser(codegen_cls) is True - def test_convert_pyi_to_ir_self_only_generated_constructor_keeps_default_initializer(): module = parse_pyi_text( @@ -183,13 +166,6 @@ def __init__(self) -> None: ... assert cls.methods == [] assert " def __init__(self) -> None: ..." in emit_module(module) - codegen_module = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - codegen_cls = codegen_module.classes[0] - assert CPythonBindingGenerator._suppresses_default_class_initialiser(codegen_cls) is False - def test_convert_pyi_to_ir_bound_constructor_replaces_generated_keyword_initializer(): module = parse_pyi_text( @@ -231,21 +207,6 @@ def __init__( assert "def __init__(\n self,\n *," not in emitted assert parse_pyi_text(emitted, module_name="edited") == module - codegen_module = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - codegen_cls = codegen_module.classes[0] - assert codegen_cls.decorators[SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA] is True - assert CPythonBindingGenerator._suppresses_default_class_initialiser(codegen_cls) is True - codegen_init = next( - method for method in codegen_cls.methods if codegen_cls.scope.get_python_name(method.name) == "__init__" - ) - assert codegen_cls.scope.get_python_name(codegen_init.name) == "__init__" - assert codegen_init.arguments[0].bound_argument is True - assert codegen_init.arguments[0].bound_argument_position == 0 - assert [str(arg.name) for arg in codegen_init.arguments[1:]] == ["seed", "scale"] - def test_convert_pyi_to_ir_bound_constructor_allows_public_target_method(): module = parse_pyi_text( @@ -486,13 +447,6 @@ def assign_vector_real( assert func.projection[0].result_position == 0 assert from_pyi.classes[0].overload_sets[0].procedures[0].metadata["overload_kind"] == "assignment" - codegen_module = semantic_ir_to_codegen_ast( - from_pyi, - Scope(name=from_pyi.name, scope_type="module"), - ) - assign = next(item for item in codegen_module.classes[0].overload_sets if item.name == "assign") - assert assign.functions[0].arguments[0].var.projected_output is True - def test_type_bound_method_declarations_restore_root_target_metadata(): from_pyi = parse_pyi_text( @@ -534,7 +488,7 @@ def shift_vector( assert functions["shift_vector"].metadata["fortran_passed_object_position"] == 1 -def test_pyi_codegen_keyword_normalized_type_bound_method_uses_native_binding_name(): +def test_pyi_keyword_normalized_type_bound_method_keeps_native_binding_name(): module = parse_pyi_text( """ class visible_t: @@ -544,11 +498,10 @@ def from_(self) -> Int32: ... module_name="fnaming_f90", ) - codegen_module = semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - method = codegen_module.classes[0].methods_as_dict["from_"] + method = module.classes[0].methods[0] - assert method.name == "visible_from" - assert method.type_bound_name == "from" + assert method.name == "from_" + assert method.native_name == "visible_from" def test_method_equality_treats_argument_names_as_placeholders(): diff --git a/tests/semantics/conversion/pyi/test_pyi_conversion_imports_and_packages.py b/tests/semantics/conversion/pyi/test_pyi_conversion_imports_and_packages.py index 6d186705a..94637e9fc 100644 --- a/tests/semantics/conversion/pyi/test_pyi_conversion_imports_and_packages.py +++ b/tests/semantics/conversion/pyi/test_pyi_conversion_imports_and_packages.py @@ -4,7 +4,6 @@ CONTRACT_IMPORT, FORTRAN_PYI_COMPARE_FIXTURES, Path, - Scope, SemanticImport, SemanticImportItem, _semantic_modules_for_source, @@ -19,7 +18,6 @@ pyi_pipeline, pyi_text_to_semantic_module, pytest, - semantic_ir_to_codegen_ast, ) @@ -337,33 +335,6 @@ def test_convert_pyi_to_ir_and_import_parser_edge_cases(): pyi_text_to_semantic_module("from m import\n", module_name="edited") -def test_pyi_codegen_imports_public_generic_not_private_specific_targets(): - module = parse_pyi_text( - """ -@private -def convert_integer( - value: Addr(Int32) -) -> Int32: ... - -@overload("convert_integer") -def convert( - value: Addr(Int32) -) -> Int32: ... -""", - module_name="foverloads_f90", - ) - - codegen_module = semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - imported = { - (str(target.name), str(target.local_alias)) - for native_import in codegen_module.imports - for target in native_import.target - } - - assert ("convert", "convert") in imported - assert all("convert_integer" not in item for names in imported for item in names) - - def test_generated_pyi_loads_and_reemits_for_all_fortran_fixtures(tmp_path: Path): assert FORTRAN_PYI_COMPARE_FIXTURES diff --git a/tests/semantics/policy/test_accessor_and_storage_policy.py b/tests/semantics/policy/test_accessor_and_storage_policy.py index 42c14812e..6a187d759 100644 --- a/tests/semantics/policy/test_accessor_and_storage_policy.py +++ b/tests/semantics/policy/test_accessor_and_storage_policy.py @@ -4,13 +4,8 @@ ADDRESS_ROLE_METADATA, ADDRESS_ROLE_PROJECTION, AssignmentMode, - BindCAccessorModuleVariable, - CPythonBindingGenerator, - CPythonCodePrinter, CodegenAction, DestructionPolicy, - FCodePrinter, - FortranToCBridgeGenerator, MODULE_VARIABLE_INITIALIZER_UNSUPPORTED_BLOCKER, NativeArrayBuildRequirement, NativeBarrierAction, @@ -25,7 +20,6 @@ RESOLVED_OWNERSHIP_POLICY_METADATA, RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA, RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA, - Scope, SemanticClass, SemanticConstraint, SemanticField, @@ -39,14 +33,11 @@ _array_type, _derived_type, _scalar_type, - _semantic_ir_to_codegen_ast, complete_semantic_policies, native_array_descriptor_kind, native_array_handle_build_requirements, parse_pyi_text, pytest, - replace, - semantic_ir_to_codegen_ast, set_ownership_metadata, ) @@ -54,46 +45,6 @@ from x2py.semantics.wrapper_policy import ModuleObjectAccessMechanism -def test_native_array_handle_module_shape_changing_operations_use_scalar_extents(): - module = parse_pyi_text( - """ -values: Allocatable[Float64[:, :]] -target: Pointer[Float64[:]] -""", - module_name="native_handle_module_shape_ops", - ) - complete_semantic_policies(module) - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - bridged = FortranToCBridgeGenerator("", 0)._visit_Module(lowered) - handles = {str(variable.name): variable for variable in bridged.variable_wrappers} - - assert {"resize"} <= set(handles["values"].operation_functions) - assert {"allocate", "deallocate", "resize"}.isdisjoint(handles["target"].operation_functions) - - fortran_code = FCodePrinter("native_handle_module_shape_ops.f90", verbose=0)._visit(bridged) - assert "subroutine bind_c_private__x2py_values_resize(extent_1, extent_2) bind(c)" in fortran_code - assert "integer(i64), value :: extent_1" in fortran_code - assert "allocate(values(0:extent_2 - 1_i64, 0:extent_1 - 1_i64))" in fortran_code - - cpython_module = CPythonBindingGenerator("", 0)._visit_Module(bridged) - c_code = CPythonCodePrinter("native_handle_module_shape_ops.c", verbose=0)._visit(cpython_module) - assert "private__x2py_values_resize" in c_code - - generator = FortranToCBridgeGenerator("", 0) - generator.scope = lowered.scope - pointer_policy = replace( - lowered.variables[1].native_array_handle_policy, - operations=("allocate", "associated", "deallocate", "nullify", "resize", "to_numpy"), - ) - pointer_handle = generator._native_array_module_handle(lowered.variables[1], pointer_policy) - - assert {"allocate", "deallocate", "resize"} <= set(pointer_handle.operation_functions) - assert pointer_handle.native_array_handle_policy is pointer_policy - assert pointer_handle.operation_functions["allocate"].arguments[0].var.name == "extent_1" - assert pointer_handle.operation_functions["resize"].arguments[0].var.name == "extent_1" - assert "private__x2py_target_allocate" not in c_code - - def test_native_array_handle_build_requirements_include_default_pointer_descriptor_accessors(): module = parse_pyi_text( """ @@ -397,15 +348,6 @@ def test_aliased_derived_module_object_is_borrowed_and_rejects_replacement(): assert getter.codegen_action is CodegenAction.BORROWED_VIEW assert setter.setter_action is SetterAction.REJECT_REPLACEMENT - codegen_module = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - codegen_variable = codegen_module.variables[0] - assert codegen_variable.is_target is True - assert codegen_variable.ownership_decision.owner is OwnershipOwner.NATIVE - assert codegen_variable.setter_ownership_decision.setter_action is SetterAction.REJECT_REPLACEMENT - def test_plain_derived_module_object_completes_live_member_proxy_policy(): module = SemanticModule( @@ -471,13 +413,6 @@ def test_derived_module_constant_uses_wrapper_owned_copy_without_setter(): assert getter.transfer is TransferMode.WRAPPER_INSTANCE assert setter.setter_action is SetterAction.OMIT - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - generator = FortranToCBridgeGenerator("", 0) - generator.scope = lowered.scope - constant = generator._visit_Variable(lowered.variables[0]) - assert isinstance(constant, BindCAccessorModuleVariable) - assert constant.setter_function is None - def test_explicit_borrowed_derived_field_setter_rejects_replacement(): child_type = _derived_type("child") diff --git a/tests/semantics/policy/test_native_array_ownership.py b/tests/semantics/policy/test_native_array_ownership.py index 62172af91..ef2bcf750 100644 --- a/tests/semantics/policy/test_native_array_ownership.py +++ b/tests/semantics/policy/test_native_array_ownership.py @@ -1,12 +1,8 @@ """Tests split by stable ownership concept from `test_handle_policy_dispatch.py`.""" from tests._shared.ownership_policy_support import ( - BindCNativeArrayDescriptorType, - CPythonBindingGenerator, - CPythonCodePrinter, CodegenAction, DestructionPolicy, - FortranToCBridgeGenerator, NativeArrayBuildRequirement, NativeArrayHandlePolicyDispatcher, NativeBarrierAction, @@ -15,7 +11,6 @@ PyiPrinter, RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA, RESOLVED_OWNERSHIP_POLICY_METADATA, - Scope, StorageMode, TransferMode, _array_type, @@ -24,12 +19,10 @@ _native_array_policy, _read_only_argument_context, _scalar_type, - _semantic_ir_to_codegen_ast, _string_type, _writable_argument_context, complete_semantic_policies, default_ownership_policy, - native_array_descriptor_argument_type, native_array_handle_build_requirements, parse_pyi_text, pytest, @@ -81,36 +74,6 @@ def test_native_array_handle_dispatcher_rejects_missing_completed_policy_pair(): ) -def test_native_array_descriptor_argument_codegen_selects_bind_c_tuple_abi(): - class Subject: - name = "values" - - required_policy = _native_array_policy(handle_kind="argument_descriptor") - optional_policy = _native_array_policy( - handle_kind="optional_absent_handle", - nullable=True, - optional_absent=True, - ) - - for generator in (FortranToCBridgeGenerator, CPythonBindingGenerator): - required_type = generator._native_array_descriptor_argument_type(required_policy) - optional_type = generator._native_array_descriptor_argument_type(optional_policy) - - assert required_type is native_array_descriptor_argument_type(required_policy) - assert optional_type is native_array_descriptor_argument_type(optional_policy) - assert required_type is BindCNativeArrayDescriptorType.get_new(has_presence=False) - assert optional_type is BindCNativeArrayDescriptorType.get_new(has_presence=True) - assert len(required_type) == 1 - assert len(optional_type) == 2 - - bridge = FortranToCBridgeGenerator("", 0) - binding = CPythonBindingGenerator("", 0) - assert bridge._native_array_descriptor_argument_type(required_policy) is required_type - assert bridge._native_array_descriptor_argument_type(optional_policy) is optional_type - assert binding._native_array_descriptor_argument_type(required_policy) is required_type - assert binding._native_array_descriptor_argument_type(optional_policy) is optional_type - - def test_hidden_allocatable_handle_output_completes_as_owned_result_before_lowering(): module = parse_pyi_text( """ @@ -136,14 +99,6 @@ def make_values() -> Allocatable[Float64[:]]: ... assert policy.descriptor_ownership == "owned" assert policy.output_projection == "projected_handle" - lowered = _semantic_ir_to_codegen_ast(module, Scope(name=module.name, scope_type="module")) - bridged = FortranToCBridgeGenerator("", 0)._visit_Module(lowered) - cpython_module = CPythonBindingGenerator("", 0)._visit_Module(bridged) - c_code = CPythonCodePrinter("hidden_owned_result.c", verbose=0)._visit(cpython_module) - assert "CFI_attribute_allocatable" in c_code - assert "CFI_allocate(" in c_code - assert "private__x2py_owned_values_destroy" in c_code - @pytest.mark.parametrize( ("owner", "transfer", "destruction", "context"), diff --git a/tests/semantics/policy/test_policy_defaults_and_validation.py b/tests/semantics/policy/test_policy_defaults_and_validation.py index a9d697fbd..647175e1f 100644 --- a/tests/semantics/policy/test_policy_defaults_and_validation.py +++ b/tests/semantics/policy/test_policy_defaults_and_validation.py @@ -5,14 +5,10 @@ ADDRESS_ROLE_RAW, ArrayInteropPolicy, ArrayInteropPolicyDispatcher, - AssignmentMode, - CPythonBindingGenerator, CodegenAction, DestructionPolicy, NativeBarrierAction, NativeBarrierDispatcher, - NumpyFloat64Type, - NumpyNDArrayType, ObjectKind, OwnershipContext, OwnershipDecision, @@ -28,7 +24,6 @@ RESOLVED_OWNERSHIP_POLICY_METADATA, RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA, RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA, - Scope, SemanticArgument, SemanticClass, SemanticField, @@ -38,7 +33,6 @@ SetterAction, StorageMode, TransferMode, - Variable, _address_type, _array_type, _derived_type, @@ -49,12 +43,10 @@ _string_storage_type, _string_type, _writable_argument_context, - codegen_action_for_variable, complete_semantic_policies, default_ownership_policy, parse_pyi_text, pytest, - semantic_ir_to_codegen_ast, ) @@ -387,18 +379,6 @@ def descriptor(self, subject, policy, marker): ) == ("seen", "values", "descriptor", "allocatable") -def test_optional_normal_array_bind_c_argument_does_not_use_native_handle_fallback(): - argument = Variable( - NumpyNDArrayType.get_new(NumpyFloat64Type(), 1, "F"), - "values", - is_optional=True, - ) - binding = CPythonBindingGenerator("", 0) - - assert argument.is_optional is True - assert binding._bind_c_array_argument_uses_native_handle_fallback(argument) is False - - def test_immutable_replacement_policy_is_complete_before_ir_lowering(): module = parse_pyi_text( """ @@ -539,23 +519,3 @@ def test_policy_completion_attaches_decisions_before_ir_lowering(): module.functions[0].arguments[0].metadata[RESOLVED_OWNERSHIP_POLICY_METADATA].transfer is TransferMode.IN_PLACE ) assert RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA not in module.functions[0].metadata - - codegen_module = semantic_ir_to_codegen_ast( - module, - Scope(name=module.name, scope_type="module"), - ) - - module_var = codegen_module.variables[0] - field_var = codegen_module.classes[0].attributes[0] - arg_var = codegen_module.funcs[0].arguments[0].var - - assert module_var.ownership_decision.owner is OwnershipOwner.NATIVE - assert module_var.getter_ownership_decision.codegen_action is CodegenAction.BORROWED_VIEW - assert module_var.setter_ownership_decision.assignment_mode is AssignmentMode.VALUE_COPY - assert module_var.setter_ownership_decision.setter_action is SetterAction.REJECT_REPLACEMENT - assert field_var.ownership_decision.owner is OwnershipOwner.WRAPPER - assert field_var.getter_ownership_decision.codegen_action is CodegenAction.BORROWED_VIEW - assert field_var.setter_ownership_decision.assignment_mode is AssignmentMode.VALUE_COPY - assert field_var.setter_ownership_decision.setter_action is SetterAction.REJECT_REPLACEMENT - assert arg_var.ownership_decision.owner is OwnershipOwner.CALLER - assert codegen_action_for_variable(arg_var) is CodegenAction.IN_PLACE_ARGUMENT diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index 5348c511d..c9300581c 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -32,6 +32,8 @@ from x2py.semantics.wrapper_policy import ( ArgumentHandoffMode, BridgeDataAction, + CallbackABIKind, + CallbackTransferAction, FunctionWrapperPolicy, ModuleGetterAction, ModuleVariablePolicy, @@ -219,6 +221,35 @@ def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... assert policy.native_call_slots[1].native_barrier_action is hidden.native_barrier_action +def test_source_hidden_scalar_output_completes_call_local_address_before_planning(): + module = _source_semantic_module("foutputs_f90.f90", module_name="foutputs_f90") + function = next(function for function in module.functions if function.name == "scalar_status") + policy = completed_function_wrapper_policy(function) + + hidden = policy.results[0] + assert hidden.source_kind == "hidden_output" + assert hidden.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS + assert policy.native_call_slots[1].native_barrier_action is hidden.native_barrier_action + + +def test_source_callback_value_and_read_access_are_completed_as_independent_facts(): + module = _source_semantic_module("fcallback_all_f90.f90", module_name="fcallback_all_f90") + function = next(item for item in module.functions if item.name == "apply_value_callback") + policy = completed_function_wrapper_policy(function) + transfer = policy.arguments[0].callback.arguments[0] + + assert transfer.abi is CallbackABIKind.VALUE + assert transfer.access == "read" + assert transfer.adapter_action is CallbackTransferAction.COPY_IN + + array_function = next(item for item in module.functions if item.name == "apply_array_storage_callback") + array_policy = completed_function_wrapper_policy(array_function) + extent = array_policy.arguments[0].callback.arguments[0] + assert extent.abi is CallbackABIKind.REFERENCE + assert extent.access == "read" + assert extent.adapter_action is CallbackTransferAction.COPY_IN + + def test_hidden_scalar_descriptor_result_keeps_descriptor_policy_instead_of_plain_address_storage(): module = parse_pyi_text( """ @@ -733,7 +764,9 @@ def discard_name(name: String[8]) -> None: ... assert argument.codegen_action is CodegenAction.COPY_IN_OUT assert argument.character_length == 8 assert argument.projects_result is True - assert argument.writable is True + # The native call mutates a binding-owned replacement, not the immutable + # Python string supplied at the public boundary. + assert argument.writable is False assert tuple(action.phase for action in replacement.writeback_actions) == tuple(WritebackPhase) assert identity.supported is True diff --git a/tests/tools/test_check_radon_policy.py b/tests/tools/test_check_radon_policy.py index a04e21b48..04ec10fe8 100644 --- a/tests/tools/test_check_radon_policy.py +++ b/tests/tools/test_check_radon_policy.py @@ -11,7 +11,6 @@ changed_block_violates_policy, complexity_blocks_for_file, is_under_source_roots, - legacy_baseline_complexity, main, parse_changed_python_files, resolve_base_ref, @@ -75,21 +74,6 @@ def test_changed_python_files_preserve_pre_rename_paths(): ] -def test_legacy_baseline_is_limited_to_named_imported_hotspots(): - known = ComplexityBlock( - Path("x2py/semantics/ir2ast.py"), - "function", - "semantic_ir_to_codegen_ast", - 1, - 10, - 33, - ) - unknown = ComplexityBlock(Path("x2py/new.py"), "function", "branchy", 1, 10, 33) - - assert legacy_baseline_complexity(known) == 33 - assert legacy_baseline_complexity(unknown) is None - - def test_changed_block_policy_allows_existing_hotspots_unless_worsened(): block = ComplexityBlock(Path("pkg/mod.py"), "function", "legacy", 1, 10, 25) diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index a5741233e..54565310d 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -6,9 +6,9 @@ modules are searchable without relying on old flat filenames. Wrapper generator route status is maintained only in `docs/maintainer/roadmap/wrapper-plan-migration-checklist.md`. Its exhaustive, -test-enforced matrix lists every collected wrapper node as `legacy`, -`dual-route`, `wrapper-plan`, `not-applicable`, or `deferred-real-library` and -records the zero-legacy completion target. +test-enforced matrix lists every collected wrapper node as `wrapper-plan` or +`not-applicable`. Historical migration statuses remain in the roadmap's +recorded progression, not in the live ledger. ## Stage 1 — Searchable Layout, Contract Output, And Fixtures @@ -53,15 +53,15 @@ records the zero-legacy completion target. | Roadmap item | Evidence | | --- | --- | -| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators; isolated scalar-only parity covers primitive kinds, value/address projection, hidden output, copy-in/copy-out, rank-zero storage, raw addresses, and direct-plus-hidden multiple-result tuple assembly without array/string route blockers | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_boundary_plan.py`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | -| Function-call contracts rebuild from generated `.pyi` fixtures with the same optional-argument handling, scalar replacement writeback, hidden output projection, multiple-result ordering, allocatable nullable outputs, native-call projection metadata, native shared-library link inputs, and validation failures as source builds | `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes`, `function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement_through_both_routes`, `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules`, `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection`, `function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | -| Array contracts rebuild from generated `.pyi` fixtures with the same dtype, rank, shape, order, stride, lower-bound, writeability, byte-order, alignment, zero-extent, assumed-rank dispatch, ordinary Python-owned result behavior, and allocatable result-handle behavior as source builds | `arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, `arrays/test_array_results.py::test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument`, `arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides`, `arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views` | -| Character contracts rebuild from generated `.pyi` fixtures with the same fixed-length buffers, assumed-length strings, nullable deferred scalar results, deferred-width native handles, copy-in/copy-out behavior, optional strings, Unicode handling, embedded-NUL validation, and raw fixed-width array addresses as source builds | `strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes`, `strings/test_character_arguments.py::test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes`, `strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes`, `strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_projected_replacement_without_native_call_keeps_writable_argument_storage`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_native_call_projected_output_keeps_visible_storage_writable` | +| Scalar and verified baseline array routines rebuild from generated `.pyi` contracts, expose the same normalized Python names as source builds, compare generated packages against checked fixtures, and use the same runtime assertion bodies; both `fmath.f` and `fmath_f90.f90` also replay the legacy and wrapper-plan generators; isolated scalar-only parity covers primitive kinds, value/address projection, hidden output, copy-in/copy-out, rank-zero storage, raw addresses, and direct-plus-hidden multiple-result tuple assembly without array/string route blockers | `scalars/test_verified_baseline.py::test_fmath_scalar_sources_use_canonical_wrapper_plan`, `scalars/test_verified_baseline.py::test_fortran_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_f90_wrapper_pipeline_builds_importable_extension`, `scalars/test_verified_baseline.py::test_fortran_array_wrapper_pipeline_matches_fmath_results_with_contiguous_arrays`, `scalars/test_verified_baseline.py::test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts`, `scalars/test_fortran_enums.py::test_fortran_enums_preserve_integer_runtime_surface`, `scalars/test_scalar_boundary_plan.py`, `scalars/test_scalar_kinds.py::test_scalar_kind_coverage_uses_compiler_probed_wrapper_types`, `scalars/test_value_and_bind_c.py::test_value_and_existing_bind_c_renamed_symbol_use_correct_abi` | +| Function-call contracts rebuild from generated `.pyi` fixtures with the same optional-argument handling, scalar replacement writeback, hidden output projection, multiple-result ordering, allocatable nullable outputs, native-call projection metadata, native shared-library link inputs, and validation failures as source builds | `function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_fixed_form_optional_arguments_drive_fortran_present_behavior`, `function_calls/test_optional_arguments.py::test_optional_array_descriptors_preserve_presence_and_storage_state`, `function_calls/test_scalar_writeback_plan.py::test_scalar_copy_in_out_returns_replacement`, `function_calls/test_output_arguments.py::test_output_arguments_and_multiple_results_follow_python_projection_rules`, `function_calls/test_native_call_examples.py::test_native_call_examples_cover_scalar_array_string_and_object_projection`, `function_calls/test_native_call_examples.py::test_native_call_examples_build_from_generated_pyi_and_native_shared_library` | +| Array contracts rebuild from generated `.pyi` fixtures with the same dtype, rank, shape, order, stride, lower-bound, writeability, byte-order, alignment, zero-extent, assumed-rank dispatch, ordinary Python-owned result behavior, and allocatable result-handle behavior as source builds | `arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, `arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument`, `arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides`, `arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views` | +| Character contracts rebuild from generated `.pyi` fixtures with the same fixed-length buffers, assumed-length strings, nullable deferred scalar results, deferred-width native handles, copy-in/copy-out behavior, optional strings, Unicode handling, embedded-NUL validation, and raw fixed-width array addresses as source builds | `strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan`, `strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan`, `strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_use_canonical_plan`, `strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_projected_replacement_without_native_call_keeps_writable_argument_storage`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_native_call_projected_output_keeps_visible_storage_writable` | | Derived-type contracts rebuild from generated `.pyi` fixtures with the same fields, methods, type-bound root targets, constructors, finalizers, borrowed child lifetime, scalar object boundaries, inheritance, polymorphic dispatch, complete scalar actual/dummy compatibility, descriptor-backed scalar module proxies, wrapper-owned allocatable/pointer holders, and exact readiness blockers as source builds | `derived_types/test_derived_type_methods.py::test_modern_fortran_derived_type_exposes_class_and_type_bound_methods`, `derived_types/test_constructors_and_finalizers.py::test_fortran_default_constructor_keywords_and_finalization`, `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component`, `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy`, `derived_types/test_derived_type_boundaries.py::test_scalar_derived_types_cross_procedure_boundaries`, `derived_types/test_phase8_derived_plan.py`, `derived_types/test_scalar_derived_actual_dummy_matrix.py`, `derived_types/test_inheritance.py::test_fortran_extension_types_generate_python_inheritance`, `derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy`, `tests/wrapper_codegen/test_phase8_scalar_derived_actual_dummy_matrix.py`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py::test_type_bound_method_declarations_restore_root_target_metadata` | | Callback contracts route through wrapper-plan generation without legacy lowering and rebuild from generated `.pyi` fixtures with the same value, scalar-storage, array, character-storage, and derived callback conversions, call-scoped lifetime, nested same-thread entry, GIL handling, reference cleanup, and fatal exception behavior as source builds | `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `callbacks/test_scalar_callbacks.py::test_immediate_scalar_dummy_procedure_calls_python_callback`, `callbacks/test_scalar_callbacks.py::test_callback_exception_prints_traceback_and_aborts_host_process`, `callbacks/test_array_callbacks.py::test_immediate_dummy_procedure_converts_array_arguments_and_results`, `callbacks/test_derived_callbacks.py::test_immediate_dummy_procedure_converts_derived_arguments_and_results`, `tests/wrapper_codegen/test_phase10_callbacks.py`, `tests/semantics/conversion/pyi/test_types_and_values.py::test_convert_pyi_to_ir_infers_callback_dimension_argument_names` | -| Module-state contracts rebuild from generated `.pyi` fixtures with the same scalar module attributes, parameter behavior, saved native state, plain and `Aliased` live allocatable module views, borrowed field handles, owned allocatable result handles, same-handle allocatable descriptor mutation, explicit-copy independence, fresh extraction after state changes, rank-zero descriptor copying/nullability, and common-block encapsulation as source builds; isolated scalar and native-handle owners also replay legacy and wrapper-plan routes | `module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter`, `module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_matches_legacy_route`, `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view`, `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values`, `module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle`, `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes`, `derived_types/test_pointers.py::test_module_native_array_handles_match_legacy_and_wrapper_plan_routes`, `module_state/test_common_blocks.py::test_common_block_storage_stays_internal_to_wrapped_fortran` | +| Module-state contracts rebuild from generated `.pyi` fixtures with the same scalar module attributes, parameter behavior, saved native state, plain and `Aliased` live allocatable module views, borrowed field handles, owned allocatable result handles, same-handle allocatable descriptor mutation, explicit-copy independence, fresh extraction after state changes, rank-zero descriptor copying/nullability, and common-block encapsulation as source builds; isolated scalar and native-handle owners also replay legacy and wrapper-plan routes | `module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter`, `module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_uses_canonical_plan`, `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view`, `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values`, `module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle`, `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity`, `derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan`, `module_state/test_common_blocks.py::test_common_block_storage_stays_internal_to_wrapped_fortran` | | Runtime behavior contracts rebuild from generated or edited `.pyi` fixtures with the same recursion/reentrancy behavior, `@hold_gil` GIL policy, `@raises(...)` status projection, and generated wrapper policy code as source-backed builds | `runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls`, `runtime_behavior/test_runtime_policies.py::test_pyi_runtime_policies_release_gil_and_project_native_errors`, `runtime_behavior/test_runtime_policies.py::test_compiled_runtime_policies_release_gil_and_project_native_errors`, `runtime_behavior/test_openmp_runtime.py::test_openmp_enabled_procedure_builds_with_explicit_gnu_flags` | -| Naming and generic-interface contracts rebuild from generated `.pyi` fixtures with the same public-name normalization, visibility filtering, keyword/collision policy, public generic dispatch, type-bound binding names, defined operators, comparisons, named operators, and assignment behavior as source builds | `naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy`, `naming/test_generic_interfaces.py::test_fortran_generic_interfaces_dispatch_in_generated_c_extension`, `naming/test_generic_interfaces.py::test_fixed_form_fortran_generic_interface_dispatches_in_generated_c_extension`, `naming/test_defined_operators.py::test_fortran_defined_operators_and_assignment_dispatch_in_generated_c_extension`, `tests/codegen/printers/test_pyi_printer_imports_and_packages.py::test_fortran_generated_contracts_reserve_colliding_public_names_by_namespace`, `tests/semantics/conversion/pyi/test_pyi_conversion_imports_and_packages.py::test_pyi_codegen_imports_public_generic_not_private_specific_targets`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py::test_pyi_codegen_keyword_normalized_type_bound_method_uses_native_binding_name` | +| Naming and generic-interface contracts rebuild from generated `.pyi` fixtures with the same public-name normalization, visibility filtering, keyword/collision policy, public generic dispatch, type-bound binding names, defined operators, comparisons, named operators, and assignment behavior as source builds | `naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy`, `naming/test_generic_interfaces.py::test_fortran_generic_interfaces_dispatch_in_generated_c_extension`, `naming/test_generic_interfaces.py::test_fixed_form_fortran_generic_interface_dispatches_in_generated_c_extension`, `naming/test_defined_operators.py::test_fortran_defined_operators_and_assignment_dispatch_in_generated_c_extension`, `tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py::test_fortran_generated_contracts_reserve_colliding_public_names_by_namespace`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py::test_pyi_keyword_normalized_type_bound_method_keeps_native_binding_name` | | Shared native-array runtime handles validate generated operation tables at construction, require generated `shape`, `array_actual`, and `descriptor` operations, reject `None` from present-handle operations, require typed non-null handoffs for array actuals, normalize descriptor field records or contiguous descriptor data addresses, resolve deferred character dtype from runtime element length, expose common metadata, state dispatch, absent-state `to_numpy()` short-circuiting, present-state extraction rejection for `None`, dtype/rank validation for extracted arrays, completed live-view dispatch for borrowed, contiguous, and strided descriptor views, owner retention, policy-gated pointer operations, explicit unsupported-extraction errors, and no implicit-copy fallback | `tests/runtime/handles/` | ## Stage 7 — Library-Scale And Mixed-Bundle Evidence @@ -79,7 +79,7 @@ records the zero-legacy completion target. | Roadmap item | Evidence | | --- | --- | -| Editable native-order contracts can omit `@native_call` when native dummies remain visible, including scalar/array output storage, raw array addresses with completed shape/orientation, fixed-length string identity calls with no observable Python `str` mutation, function results, and derived-type output slots | `edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call`, `edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_match_legacy_and_wrapper_plan_routes`, `edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi` | +| Editable native-order contracts can omit `@native_call` when native dummies remain visible, including scalar/array output storage, raw array addresses with completed shape/orientation, fixed-length string identity calls with no observable Python `str` mutation, function results, and derived-type output slots | `edit_pyi_contracts/test_native_order_contracts.py::test_editable_contract_can_use_native_order_arguments_without_native_call`, `edit_pyi_contracts/test_native_order_contracts.py::test_raw_array_addresses_use_canonical_plan`, `edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi` | | Post-IR immutable replacement policy copies a read-only Python array into mutable native storage and returns a detached replacement without mutating the original object | `edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_array_policy_copies_in_and_returns_replacement`, `edit_pyi_contracts/modified_contracts/fnative_call_examples_immutable/fnative_call_examples_f90.pyi` | | Immutable writable scalar, string, array, and derived-type arguments use policy-selected native temporaries and return replacements without mutating the original Python-visible object | `edit_pyi_contracts/test_policy_dispatch_contracts.py::test_immutable_scalar_string_array_and_derived_policies_return_replacements`, `edit_pyi_contracts/modified_contracts/fnative_call_examples_immutable_kinds/fnative_call_examples_f90.pyi` | | Contradictory owner/transfer/destruction triples fail before bridge generation with the declaration and rejected triple in the diagnostic | `edit_pyi_contracts/test_policy_dispatch_contracts.py::test_contradictory_ownership_contract_fails_before_bridge_generation`, `edit_pyi_contracts/invalid_contracts/contradictory_ownership/fnative_call_examples_f90.pyi` | @@ -87,23 +87,23 @@ records the zero-legacy completion target. | Edited contracts remove classes, methods, constructors, class members, and individual overload candidates; they can also add a renamed `@bind` declaration and a new overload group over existing native specifics | `edit_pyi_contracts/test_surface_edit_contracts.py`, `edit_pyi_contracts/modified_contracts/foverloads_pruned_surface/`, `edit_pyi_contracts/modified_contracts/foverloads_without_constructor_member/`, `edit_pyi_contracts/modified_contracts/foverloads_added_bindings/` | | Edited `.pyi` contracts can remove a public function and hide declarations with `@private` or `private[...]` while preserving unaffected runtime behavior | `edit_pyi_contracts/test_visibility_contracts.py::test_editable_contract_removes_and_hides_public_declarations`, `edit_pyi_contracts/modified_contracts/module_variables_visibility/fmodule_vars_f90.pyi` | | Native `Addr(Arg(...))` projection remains primitive-scalar-only while array descriptor arguments complete native-array handle policy before lowering, descriptor-argument bridge pass-through no longer blocks readiness, and unsupported pointer result ownership remains an explicit policy blocker | `tests/semantics/policy/test_accessor_and_storage_policy.py::test_policy_completion_rejects_addr_projection_for_array_descriptor_handles`, `tests/semantics/policy/test_native_array_ownership.py::test_native_array_handle_policies_complete_before_ir_lowering`, `tests/semantics/readiness/test_policy_blockers.py::test_pointer_descriptor_inputs_pass_while_reassociation_and_result_policies_block`, `derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy` | -| Native array handle bridge and binding architecture dispatches from completed descriptor-kind and handle-kind policy pairs | `tests/semantics/policy/test_native_array_ownership.py::test_native_array_handle_dispatcher_routes_completed_policy_to_named_method`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_bridge_and_binding_generators_expose_ownership_action_maps`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_module_variable_bridge_uses_completed_handle_policy_dispatch`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_field_generation_uses_completed_handle_policy_dispatch`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_result_generation_uses_completed_handle_policy_dispatch`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_argument_generation_uses_completed_handle_policy_dispatch` | -| Module native-array handles lower to generated borrowed module-handle variables and private operation wrappers for state, shape, pointer/descriptor handoff, allocatable `.to_numpy()`/`deallocate()`/`resize(shape)`, pointer `nullify()`, and policy-gated pointer shape-changing operations; CPython wrapping converts generated pointer-address operations with `PyLong_FromVoidPtr`, and module-variable readiness no longer blocks these supported module-handle paths | `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_module_variable_bridge_uses_completed_handle_policy_dispatch`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_module_operations_print_and_wrap_pointer_handoff_results`, `tests/semantics/policy/test_accessor_and_storage_policy.py::test_native_array_handle_module_shape_changing_operations_use_scalar_extents`, `tests/semantics/readiness/test_policy_blockers.py::test_allocatable_handle_codegen_accepts_owned_results_with_standard_cfi_storage`, `tests/semantics/readiness/test_policy_blockers.py::test_pointer_module_variable_default_handle_policy_is_codegen_ready` | -| Generated native array handle construction has shared substrate before bridge descriptor accessors are enabled: Bind-C variables carry explicit operation-name maps, CPython binding generation builds an ops dictionary from private generated callables and calls the runtime factory, and the runtime factory adapts zero-argument generated operations to the handle protocol, splats shape-changing operations to generated scalar extents, and wraps pointer-address results as native handoffs | `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_native_array_handle_binding_builds_runtime_handle_from_named_generated_ops`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_adapts_private_operations_to_runtime_protocol`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_splats_shape_operations_to_scalar_extents`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_rejects_invalid_descriptor_kind_and_handoff_result` | -| Owned allocatable results use persistent rank-specific `CFI_CDESC_T` storage established with allocatable attribute, allocate payloads through `CFI_allocate`, copy bridge-local result data before releasing it, expose owner-addressed descriptor/state/extraction/resize/deallocate operations, pass that persistent descriptor directly for projected writable mutation, and release payload plus descriptor record through the shared handle finalizer path; single and hidden-output results compile and run in source and generated-`.pyi` modes | `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_result_generation_uses_completed_handle_policy_dispatch`, `tests/semantics/policy/test_native_array_ownership.py::test_hidden_allocatable_handle_output_completes_as_owned_result_before_lowering`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_owned_handle_factory_passes_persistent_owner_to_every_operation`, `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, `module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle` | -| Native array descriptor-handle argument ABI has a dedicated Bind-C tuple model with required descriptor pointer shape and optional presence-token shape, one shared helper selects that shape from completed policy, bridge descriptor handlers pass descriptor dummies through as TS29113 descriptor pointers with optional presence tokens, and CPython binding descriptor handlers own validation while packing required and optional handle arguments through the runtime helper with dtype, rank, and fixed rank-one extent validation metadata | `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::test_bind_c_native_array_descriptor_type_describes_required_descriptor_pointer`, `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::test_bind_c_native_array_descriptor_type_expands_optional_presence_token`, `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::test_bind_c_native_array_descriptor_type_validates_presence_flag_before_cache_lookup`, `tests/wrapper/fortran/arrays/test_bind_c_array_type.py::test_native_array_descriptor_argument_type_uses_completed_optional_absence_policy`, `tests/semantics/policy/test_native_array_ownership.py::test_native_array_descriptor_argument_codegen_selects_bind_c_tuple_abi`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_handle_argument_generation_uses_completed_handle_policy_dispatch`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_native_array_optional_handle_argument_binding_uses_presence_tuple`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_native_array_descriptor_argument_binding_forwards_fixed_rank_one_extent` | -| Generated concrete-rank numeric `T[...]` Bind-C array argument binding keeps the existing NumPy extraction path for ndarray inputs and adds a non-NumPy fallback through the runtime native array-actual helper for allocated allocatable handles and associated pointer handles; optional, assumed-rank, and character array paths stay on their specialized conversions and reject native handles instead of treating absent handles as nullable data buffers | `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_normal_array_bind_c_argument_binding_uses_native_handle_fallback`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_normal_array_bind_c_argument_binding_keeps_specialized_array_paths`, `tests/semantics/policy/test_policy_defaults_and_validation.py::test_optional_normal_array_bind_c_argument_does_not_use_native_handle_fallback`, `tests/wrapper/fortran/arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior` | +| Native array handle bridge and binding architecture dispatches from completed descriptor-kind and handle-kind policy pairs | `tests/semantics/policy/test_native_array_ownership.py::test_native_array_handle_dispatcher_routes_completed_policy_to_named_method`, `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_generated_artifacts_follow_one_typed_action_vocabulary`, `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_plan_edits_fail_central_validation` | +| Module native-array handles lower from typed borrowed-handle plans to private operation wrappers for state, shape, pointer/descriptor handoff, allocatable `.to_numpy()`/`deallocate()`/`resize(shape)`, pointer `nullify()`, and policy-gated pointer shape-changing operations | `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_module_variables_own_borrowed_handle_plans_and_operation_sets`, `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_generated_artifacts_follow_one_typed_action_vocabulary`, `tests/semantics/readiness/test_policy_blockers.py::test_pointer_module_variable_default_handle_policy_is_codegen_ready` | +| Generated native array handle construction uses typed operation sets and the runtime factory adapts generated operations to the handle protocol, including shape changes and pointer-address handoff | `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_module_variables_own_borrowed_handle_plans_and_operation_sets`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_adapts_private_operations_to_runtime_protocol`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_splats_shape_operations_to_scalar_extents`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_handle_factory_rejects_invalid_descriptor_kind_and_handoff_result` | +| Owned allocatable results use planned persistent descriptor ownership, collect bridge-local data before transfer, expose owner-addressed operations, and release through the shared handle finalizer path; source and generated-`.pyi` modes retain compiled behavior coverage | `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_numeric_owned_result_is_collected_before_persistent_descriptor_move`, `tests/semantics/policy/test_native_array_ownership.py::test_hidden_allocatable_handle_output_completes_as_owned_result_before_lowering`, `tests/runtime/handles/test_factories_and_lifecycle.py::test_generated_owned_handle_factory_passes_persistent_owner_to_every_operation`, `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, `module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle` | +| Native array descriptor arguments record required and optional-presence roles in the typed plan; direct bridge and binding lowering consume those roles while runtime helpers validate descriptor kind, dtype, rank, and shape metadata | `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_keeps_datatype_specific_state_under_argument_and_result_plans`, `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_generated_artifacts_follow_one_typed_action_vocabulary`, `tests/runtime/handles/test_descriptor_abi.py` | +| Concrete-rank numeric array arguments retain the NumPy extraction path and use native handle actuals without converting through `to_numpy()`; optional and assumed-rank behavior remains covered at the public wrapper boundary | `tests/wrapper_codegen/test_phase6a_array_buffers.py::test_required_array_buffer_dispatches_through_named_binding_and_bridge_methods`, `tests/runtime/handles/test_array_actual_abi.py`, `tests/wrapper/fortran/arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `tests/wrapper/fortran/function_calls/test_optional_arguments.py::test_optional_arguments_drive_fortran_present_behavior` | | Runtime normal-array argument packing uses the generated Bind-C array tuple shape for ndarray inputs and native handle array-actual handoff: pointer address, optional runtime rank, optional item size, extents, and optional upper bounds plus unit strides; allocated/associated handles pack without calling `to_numpy()`, and unallocated/unassociated handles block before generated handoff | `tests/runtime/handles/test_array_actual_abi.py::test_array_actual_argument_abi_packer_uses_ndarray_data_pointer_and_shape_fields`, `tests/runtime/handles/test_array_actual_abi.py::test_array_actual_argument_abi_packer_uses_allocatable_native_array_actual_without_numpy_conversion`, `tests/runtime/handles/test_array_actual_abi.py::test_array_actual_argument_abi_packer_uses_pointer_native_array_actual_dtype_metadata`, `tests/runtime/handles/test_array_actual_abi.py::test_array_actual_argument_abi_packer_rejects_absent_handles_before_generated_handoff` | | Runtime descriptor-handle argument packing returns validated standard descriptor facts (`base_addr`, `elem_len`, `rank`, and dimension lower bounds, extents, and stride multipliers) for non-projected calls, while projected writable calls require a typed persistent standard-descriptor handoff; generated C establishes call-local CFI storage only for the fact-packed path, optional present handles add a distinct non-null presence token, and optional absent `None` produces null fields without accepting unsupported descriptor kinds | `tests/runtime/handles/test_descriptor_abi.py::test_descriptor_argument_abi_packer_returns_required_descriptor_fields`, `tests/runtime/handles/test_descriptor_abi.py::test_descriptor_argument_abi_packer_positional_helper_matches_generated_call_shape`, `tests/runtime/handles/test_descriptor_abi.py::test_descriptor_argument_abi_packer_maps_optional_presence_and_absence`, `tests/runtime/handles/test_descriptor_abi.py::test_descriptor_argument_abi_packer_rejects_wrong_kind_and_unsupported_descriptor_kind`, `tests/runtime/handles/test_descriptor_abi.py::test_projected_descriptor_handoff_requires_persistent_standard_descriptor_storage` | -| Generated pointer descriptor-view decoding has a shared TS29113 reader primitive and generated-operation integration: the CPython binding generator can decode a `CFI_cdesc_t*` into the runtime mapping shape with `base_addr`, `elem_len`, `rank`, and `dim[i].lower_bound`/`extent`/`sm`, converts pointer fields with `PyLong_FromVoidPtr`, uses that reader for private pointer `descriptor_view` `to_numpy` operation wrappers, and requires `ISO_Fortran_binding.h` only through the descriptor-reader path | `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_descriptor_view_reader_builds_runtime_mapping_from_cfi_descriptor_fields`, `tests/codegen/bridges/test_bridge_handle_policy_dispatch.py::test_native_array_descriptor_view_reader_prints_cfi_descriptor_access_without_global_requirement`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_pointer_descriptor_view_operation_wrapper_decodes_generated_cfi_descriptor_pointer`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_native_array_handle_operation_wrapper_uses_descriptor_reader_only_for_pointer_descriptor_view`, `tests/codegen/bindings/test_binding_handle_policy_dispatch.py::test_cfi_descriptor_type_printing_is_local_to_descriptor_reader_path` | +| Generated pointer descriptor-view decoding is selected by the typed handle plan and direct backend lowering; runtime decoding validates the standard descriptor fields needed to build live strided views | `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_plain_module_descriptor_view_requires_matching_completed_interop`, `tests/wrapper_codegen/test_phase7_native_array_handles.py::test_phase7_generated_artifacts_follow_one_typed_action_vocabulary`, `tests/runtime/handles/test_descriptor_abi.py` | | Runtime pointer descriptor-view extraction accepts decoded descriptor mappings or field-record objects, validates required decoded TS29113 fields and descriptor rank against the handle contract, rejects null decoded descriptors for present associated state, and builds positive- or negative-stride NumPy views in the shared helper | `tests/runtime/handles/test_descriptor_abi.py::test_pointer_c_descriptor_helper_builds_strided_numpy_view_from_decoded_fields`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_c_descriptor_helper_builds_negative_stride_numpy_view_from_decoded_fields`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_c_descriptor_helper_accepts_field_record_objects`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_c_descriptor_helper_validates_decoded_descriptor_fields`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_descriptor_view_policy_uses_decoded_descriptor_fields_for_strided_view`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_descriptor_view_policy_rejects_decoded_descriptor_rank_mismatch`, `tests/runtime/handles/test_descriptor_abi.py::test_pointer_descriptor_view_policy_rejects_null_descriptor_for_present_state` | -| Native array handle extraction is view-only: absent descriptors return `None`; plain and `Aliased` allocatables expose current mutable native storage through different completed mechanisms with identical public behavior; pointer contiguous/descriptor paths never copy; stale views are unsupported; and independent storage requires explicit `.copy()` | `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view`, `derived_types/test_phase8_derived_plan.py::test_aliased_module_derived_object_uses_direct_live_field_handles`, `derived_types/test_phase8_derived_plan.py::test_pointer_field_descriptor_views_match_legacy_and_wrapper_plan_routes`, `tests/runtime/handles/test_handle_protocols.py`, `tests/runtime/handles/test_descriptor_abi.py`, `tests/semantics/policy/test_native_array_ownership.py::test_aliased_does_not_change_allocatable_live_view_semantics` | -| Phase 8 completes canonical derived type identity, origin, owner retention, release, native handoff, field policy, recursive member paths, exact unsupported-shape blockers, subordinate plan facets, lifecycle actions, and cross-backend edit validation before lowering | `tests/wrapper_codegen/test_phase8_derived_types.py`, `tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py::test_procedure_local_derived_type_rename_uses_origin_type_identity`, `tests/codegen/printers/test_pyi_printer_imports_and_packages.py` | +| Native array handle extraction is view-only: absent descriptors return `None`; plain and `Aliased` allocatables expose current mutable native storage through different completed mechanisms with identical public behavior; pointer contiguous/descriptor paths never copy; stale views are unsupported; and independent storage requires explicit `.copy()` | `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view`, `derived_types/test_phase8_derived_plan.py::test_aliased_module_derived_object_uses_direct_live_field_handles`, `derived_types/test_phase8_derived_plan.py::test_pointer_field_descriptor_views_use_canonical_plan`, `tests/runtime/handles/test_handle_protocols.py`, `tests/runtime/handles/test_descriptor_abi.py`, `tests/semantics/policy/test_native_array_ownership.py::test_aliased_does_not_change_allocatable_live_view_semantics` | +| Phase 8 completes canonical derived type identity, origin, owner retention, release, native handoff, field policy, recursive member paths, exact unsupported-shape blockers, subordinate plan facets, lifecycle actions, and cross-backend edit validation before lowering | `tests/wrapper_codegen/test_phase8_derived_types.py`, `tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py::test_procedure_local_derived_type_rename_uses_origin_type_identity`, `tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py` | | Phase 9 completes class registration, constructor kind and lifecycle, method ownership and invocation, exact overload predicates, base ordering, inherited fields, and closed scalar polymorphic variants before lowering; both backends consume the same class/function plans | `tests/wrapper_codegen/test_phase9_class_surfaces.py`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py`, `derived_types/test_phase9_bound_constructors.py`, `naming/test_phase9_class_overloads.py` | | Default and keyword construction, explicit bound construction, exact constructor/method overloads, borrowed-child finalization, extension inheritance, and scalar polymorphic input run through the direct wrapper-plan class path | `derived_types/test_constructors_and_finalizers.py`, `derived_types/test_phase9_bound_constructors.py`, `naming/test_phase9_class_overloads.py`, `derived_types/test_borrowed_finalizers.py`, `derived_types/test_inheritance.py`, `derived_types/test_derived_type_methods.py` | -| Reduced derived procedure boundaries replay passing legacy/source behavior through direct plans for required and optional wrapper inputs, exact-type rejection, in-place and caller-supplied output identity, ordinary, `sequence`, and `bind(C)` exact typed native value copies, direct and hidden owned results, checked allocation, conversion cleanup, and exactly-once owner finalization | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_match_legacy_and_wrapper_plan_routes`, `derived_types/test_phase8_derived_plan.py::test_value_copy_and_optional_derived_inputs_match_source_oracle`, `derived_types/test_scalar_derived_actual_dummy_matrix.py::test_sequence_derived_value_uses_the_same_typed_opaque_call_path`, `derived_types/test_phase8_derived_plan.py::test_borrowed_child_retains_owner_and_finalizes_exactly_once`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_mixed_derived_results_check_allocation_and_own_every_failure_path_before_scalar_conversion` | -| Plain non-target module objects use live typed member proxies while `Aliased` objects use direct addresses; both retain the module, reject module replacement, and expose live scalar, fixed-string, ordinary-array, nested-derived, allocatable-handle, and pointer-handle fields with completed getter/setter and parent-owner behavior. Detached whole-object `Snapshot[T]` is removed | `derived_types/test_phase8_derived_plan.py::test_plain_module_derived_proxy_reads_and_writes_live_members`, `derived_types/test_phase8_derived_plan.py::test_aliased_module_derived_object_uses_direct_live_field_handles`, `derived_types/test_phase8_derived_plan.py::test_fixed_string_fields_match_legacy_and_wrapper_plan_routes`, `derived_types/test_phase8_derived_plan.py::test_pointer_field_descriptor_views_match_legacy_and_wrapper_plan_routes` | -| Eligible dependency-closed opaque-derived units select production wrapper-plan generation without invoking legacy lowering; constructors, methods, inheritance, callbacks, arrays of derived values, pointer/allocatable scalar derived values, immutable visible replacements, and mixed native-result/writeback envelopes retain exact Phase 9/10 or unsupported-policy blockers | `derived_types/test_phase8_derived_plan.py::test_eligible_derived_contract_selects_production_plan_without_legacy_lowering`, `tests/pipeline/test_wrapper_plan_route_selection.py`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_unsupported_derived_shapes_fail_on_exact_completed_policy_blockers` | +| Reduced derived procedure boundaries replay passing legacy/source behavior through direct plans for required and optional wrapper inputs, exact-type rejection, in-place and caller-supplied output identity, ordinary, `sequence`, and `bind(C)` exact typed native value copies, direct and hidden owned results, checked allocation, conversion cleanup, and exactly-once owner finalization | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_use_canonical_plan`, `derived_types/test_phase8_derived_plan.py::test_value_copy_and_optional_derived_inputs_match_source_oracle`, `derived_types/test_scalar_derived_actual_dummy_matrix.py::test_sequence_derived_value_uses_the_same_typed_opaque_call_path`, `derived_types/test_phase8_derived_plan.py::test_borrowed_child_retains_owner_and_finalizes_exactly_once`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_mixed_derived_results_check_allocation_and_own_every_failure_path_before_scalar_conversion` | +| Plain non-target module objects use live typed member proxies while `Aliased` objects use direct addresses; both retain the module, reject module replacement, and expose live scalar, fixed-string, ordinary-array, nested-derived, allocatable-handle, and pointer-handle fields with completed getter/setter and parent-owner behavior. Detached whole-object `Snapshot[T]` is removed | `derived_types/test_phase8_derived_plan.py::test_plain_module_derived_proxy_reads_and_writes_live_members`, `derived_types/test_phase8_derived_plan.py::test_aliased_module_derived_object_uses_direct_live_field_handles`, `derived_types/test_phase8_derived_plan.py::test_fixed_string_fields_use_canonical_plan`, `derived_types/test_phase8_derived_plan.py::test_pointer_field_descriptor_views_use_canonical_plan` | +| Every wrapper build uses completed policy and the canonical wrapper-plan generator; dependency isolation is structural, while unsupported derived shapes retain exact policy blockers | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_use_canonical_plan`, `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `derived_types/test_phase9_bound_constructors.py::test_bound_constructor_replaces_field_initialization_and_reuses_method_plan`, `tests/wrapper_codegen/test_phase0b_contracts.py::test_active_wrapper_build_modules_do_not_import_retired_codegen`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_unsupported_derived_shapes_fail_on_exact_completed_policy_blockers` | ## Build From Source @@ -143,7 +143,6 @@ records the zero-legacy completion target. - `arrays/test_array_contracts.py` - `arrays/test_array_results.py` - `arrays/test_assumed_rank_arrays.py` -- `arrays/test_bind_c_array_type.py` - `arrays/test_array_generated_pyi_contracts.py` - `arrays/test_multidimensional_arrays.py` diff --git a/tests/wrapper/fortran/_support.py b/tests/wrapper/fortran/_support.py index af6774584..93fa3ac36 100644 --- a/tests/wrapper/fortran/_support.py +++ b/tests/wrapper/fortran/_support.py @@ -197,17 +197,13 @@ def _build_source_or_generated_pyi_and_import( return _build_generated_pyi_and_import(source_template, workdir / "generated_pyi_build", expected_contract_package) -def _build_source_legacy_and_import( +def _build_source_and_import( source_template: Path, workdir: Path, expected_generated_sources: set[str], ): - result = build_fortran_extension( - source_template, - output_dir=workdir, - _force_legacy_wrapper_route=True, - ) - + """Build one source entry through the canonical production generator.""" + result = build_fortran_extension(source_template, output_dir=workdir) assert result.shared_library.exists() assert {path.name for path in result.generated_sources} == expected_generated_sources return _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) diff --git a/tests/wrapper/fortran/arrays/test_array_results.py b/tests/wrapper/fortran/arrays/test_array_results.py index cd38d75e2..764825f72 100644 --- a/tests/wrapper/fortran/arrays/test_array_results.py +++ b/tests/wrapper/fortran/arrays/test_array_results.py @@ -96,10 +96,9 @@ def test_array_results_follow_data_buffer_and_descriptor_handle_contracts( np.testing.assert_allclose(result, expected) -def test_ordinary_array_results_match_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): +def test_ordinary_array_results_use_canonical_plan(tmp_path: Path, monkeypatch): """Replay fixed-shape direct results without descriptor-backed neighbors.""" native_object = _compile_native_object(ARRAY_RESULTS_F90_SOURCE, tmp_path / "native") - modules = {} selected = ( "fixed_vector", "automatic_vector", @@ -108,76 +107,63 @@ def test_ordinary_array_results_match_legacy_and_wrapper_plan_routes(tmp_path: P "rank15_result", "zero_vector", ) - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_ordinary_array_results" - shutil.copytree(CONTRACT_FIXTURES / "farray_results_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "".join(f"from .farray_results_f90 import {name}\n" for name in selected), - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fixed_vector") else _sole_native_module(module) - - for module in modules.values(): - np.testing.assert_array_equal(module.fixed_vector(), np.array([1.0, 2.0, 3.0])) - np.testing.assert_array_equal(module.automatic_vector(np.int32(3)), np.array([2.0, 4.0, 6.0])) - matrix = module.automatic_matrix(np.int32(2), np.int32(3)) - np.testing.assert_array_equal(matrix, np.array([[12.0, 13.0, 14.0], [22.0, 23.0, 24.0]])) - assert matrix.flags.f_contiguous - assert module.rank3_cube(np.int32(0), np.int32(2), np.int32(3)).shape == (0, 2, 3) - assert module.rank15_result().shape == (2, *([1] * 14)) - assert module.zero_vector().shape == (0,) + contract_package = tmp_path / "ordinary_array_results" + shutil.copytree(CONTRACT_FIXTURES / "farray_results_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "".join(f"from .farray_results_f90 import {name}\n" for name in selected), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + imported = _import_from_build_dir(result.module_name, result.output_dir) + module = imported if hasattr(imported, "fixed_vector") else _sole_native_module(imported) + + np.testing.assert_array_equal(module.fixed_vector(), np.array([1.0, 2.0, 3.0])) + np.testing.assert_array_equal(module.automatic_vector(np.int32(3)), np.array([2.0, 4.0, 6.0])) + matrix = module.automatic_matrix(np.int32(2), np.int32(3)) + np.testing.assert_array_equal(matrix, np.array([[12.0, 13.0, 14.0], [22.0, 23.0, 24.0]])) + assert matrix.flags.f_contiguous + assert module.rank3_cube(np.int32(0), np.int32(2), np.int32(3)).shape == (0, 2, 3) + assert module.rank15_result().shape == (2, *([1] * 14)) + assert module.zero_vector().shape == (0,) monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError, match="Unable to allocate copy-return output array"): - modules["wrapper_plan"].fixed_vector() + module.fixed_vector() -def test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_owned_allocatable_results_preserve_handle_state(tmp_path: Path): """Replay valid allocated and zero-sized allocatable function results.""" native_object = _compile_native_object(ARRAY_RESULTS_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_allocatable_results" - shutil.copytree(CONTRACT_FIXTURES / "farray_results_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .farray_results_f90 import maybe_alloc_vector, zero_alloc_vector\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "maybe_alloc_vector") else _sole_native_module(module) - - for module in modules.values(): - allocated = module.maybe_alloc_vector(np.int32(3)) - assert isinstance(allocated, AllocatableArray) - assert allocated.allocated is True - np.testing.assert_allclose(allocated.to_numpy(), np.array([5.0, 10.0, 15.0])) - - zero_sized = module.zero_alloc_vector() - assert isinstance(zero_sized, AllocatableArray) - assert zero_sized.allocated is True - assert zero_sized.shape == (0,) - assert zero_sized.to_numpy().shape == (0,) - - allocated.close() - zero_sized.close() + contract_package = tmp_path / "allocatable_results" + shutil.copytree(CONTRACT_FIXTURES / "farray_results_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .farray_results_f90 import maybe_alloc_vector, zero_alloc_vector\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + imported = _import_from_build_dir(result.module_name, result.output_dir) + module = imported if hasattr(imported, "maybe_alloc_vector") else _sole_native_module(imported) + + allocated = module.maybe_alloc_vector(np.int32(3)) + assert isinstance(allocated, AllocatableArray) + assert allocated.allocated is True + np.testing.assert_allclose(allocated.to_numpy(), np.array([5.0, 10.0, 15.0])) + + zero_sized = module.zero_alloc_vector() + assert isinstance(zero_sized, AllocatableArray) + assert zero_sized.allocated is True + assert zero_sized.shape == (0,) + assert zero_sized.to_numpy().shape == (0,) + + allocated.close() + zero_sized.close() diff --git a/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py b/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py index 6309a8a06..ba0e2e752 100644 --- a/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py +++ b/tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py @@ -124,40 +124,32 @@ def test_assumed_rank_bridge_dispatches_each_runtime_rank_argument( assert module.rank_pair_score(left, right) == 100 * left_rank + right_rank + 4 -def test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_assumed_rank_arrays_use_explicit_plan_branches(tmp_path: Path): """Replay runtime ranks one through fifteen through explicit bridge branches.""" native_object = _compile_native_object(ASSUMED_RANK_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_assumed_rank" - shutil.copytree(CONTRACT_FIXTURES / "fassumed_rank_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fassumed_rank_f90 import rank_weighted_sum\nfrom .fassumed_rank_f90 import bump_assumed_rank\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "rank_weighted_sum") else _sole_native_module(module) - - for module in modules.values(): - for rank in (1, 2, 15): - shape = (2, *([1] * (rank - 1))) - values = np.ones(shape, dtype=np.float64, order="F") - assert module.rank_weighted_sum(values) == np.float64(rank + 2) - assert module.bump_assumed_rank(values) is None - np.testing.assert_array_equal(values, np.full(shape, rank + 1.0, order="F")) - - direct = modules["wrapper_plan"] + contract_package = tmp_path / "assumed_rank_contract" + shutil.copytree(CONTRACT_FIXTURES / "fassumed_rank_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fassumed_rank_f90 import rank_weighted_sum\nfrom .fassumed_rank_f90 import bump_assumed_rank\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + imported = _import_from_build_dir(result.module_name, result.output_dir) + module = imported if hasattr(imported, "rank_weighted_sum") else _sole_native_module(imported) + + for rank in (1, 2, 15): + shape = (2, *([1] * (rank - 1))) + values = np.ones(shape, dtype=np.float64, order="F") + assert module.rank_weighted_sum(values) == np.float64(rank + 2) + assert module.bump_assumed_rank(values) is None + np.testing.assert_array_equal(values, np.full(shape, rank + 1.0, order="F")) + with pytest.raises(TypeError): - direct.rank_weighted_sum(np.float64(1.0)) + module.rank_weighted_sum(np.float64(1.0)) with pytest.raises(TypeError): - direct.rank_weighted_sum(np.empty((1,) * 16, dtype=np.float64, order="F")) + module.rank_weighted_sum(np.empty((1,) * 16, dtype=np.float64, order="F")) diff --git a/tests/wrapper/fortran/arrays/test_bind_c_array_type.py b/tests/wrapper/fortran/arrays/test_bind_c_array_type.py deleted file mode 100644 index 8d33273b7..000000000 --- a/tests/wrapper/fortran/arrays/test_bind_c_array_type.py +++ /dev/null @@ -1,270 +0,0 @@ -import pytest - -from x2py.codegen.bind_c import ( - BindCArrayType, - BindCNativeArrayDescriptorType, - BindCPointer, - BindCScalarDescriptorType, - native_array_descriptor_argument_type, -) -from x2py.codegen.models.core import Add, Declare, IndexedElement, Slice, Variable -from x2py.codegen.models.datatypes import ( - Literal, - NumpyBoolType, - NumpyFloat32Type, - NumpyFloat64Type, - NumpyInt32Type, - NumpyInt64Type, - NumpyNDArrayType, - Cast, - StringType, - cast_to, - convert_to_literal, -) -from x2py.codegen.printers.ccode import CCodePrinter -from x2py.codegen.printers.fcode import FCodePrinter -from x2py.codegen.scope import Scope - - -def test_literal_stores_value_and_datatype_without_specialized_subclasses(): - integer = Literal(7, NumpyInt64Type()) - boolean = Literal(False, NumpyBoolType()) - string = Literal("value", StringType()) - - assert integer.python_value == 7 - assert integer.dtype is NumpyInt64Type() - assert integer.shape is None - assert boolean.python_value is False - assert string.python_value == "value" - assert string.shape == (None,) - - -def test_raw_array_uses_array_attributes_and_variable_storage(): - array_type = NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True) - variable = Variable(array_type, "shape", shape=(4,), memory_handling="stack") - - assert array_type.raw is True - assert variable.is_raw_array - assert variable.on_stack - assert CCodePrinter("test.c", verbose=0)._visit(Declare(variable)) == ("int64_t shape[4];\n") - - -def test_cast_to_uses_shared_cast_concept_with_requested_datatype(): - source = Variable(NumpyFloat64Type(), "value") - cast = cast_to(source, NumpyInt32Type()) - - assert type(cast) is Cast - assert cast.dtype is NumpyInt32Type() - - -def test_fortran_declaration_access_prints_intent_and_value_suppresses_it(): - value = Variable(NumpyFloat64Type(), "value", is_argument=True) - printer = FCodePrinter("test.f90", verbose=0) - printer._kind = lambda expr: "f64" - - assert printer._visit(Declare(value, access="readwrite")) == "real(f64), intent(inout) :: value\n" - assert printer._visit(Declare(value, access="write")) == "real(f64), intent(out) :: value\n" - assert printer._visit(Declare(value, access="readwrite", by_value=True)) == "real(f64), value :: value\n" - - -def test_callback_adapter_declaration_preserves_missing_access(): - value = Variable(NumpyFloat64Type(), "value", is_argument=True, fortran_callback_access="unspecified") - read_value = Variable(NumpyFloat64Type(), "read_value", is_argument=True, fortran_callback_access="read") - scalar_value = Variable( - NumpyFloat64Type(), - "scalar_value", - is_argument=True, - fortran_callback_access="read", - passes_by_value=True, - ) - missing_value = Variable( - NumpyFloat64Type(), - "missing_value", - is_argument=True, - fortran_callback_access="unspecified", - passes_by_value=True, - ) - printer = FCodePrinter("test.f90", verbose=0) - printer._kind = lambda expr: "f64" - - assert printer._callback_native_argument_declaration(value) == "real(f64) :: value\n" - assert printer._callback_native_argument_declaration(read_value) == "real(f64), intent(in) :: read_value\n" - assert ( - printer._callback_native_argument_declaration(scalar_value) == "real(f64), intent(in), value :: scalar_value\n" - ) - assert ( - printer._callback_native_argument_declaration(missing_value) - == "real(f64), intent(in), value :: missing_value\n" - ) - - -def test_bind_c_array_type_describes_packed_strided_layout(): - array_type = BindCArrayType.get_new(2, has_strides=True) - - assert array_type is BindCArrayType.get_new(2, has_strides=True) - assert array_type.array_rank == 2 - assert array_type.rank == 1 - assert array_type.container_rank == 1 - assert array_type.has_strides is True - assert len(array_type) == 7 - assert isinstance(array_type[0], BindCPointer) - assert all(isinstance(field, NumpyInt64Type) for field in array_type[1:]) - assert array_type.shape_is_compatible((Literal(7, NumpyInt64Type()),)) - assert not array_type.shape_is_compatible((convert_to_literal(4),)) - - -def test_bind_c_array_type_without_strides_contains_pointer_and_shape(): - array_type = BindCArrayType.get_new(3, has_strides=False) - - assert array_type.array_rank == 3 - assert array_type.has_strides is False - assert len(array_type) == 4 - assert array_type.shape_is_compatible((convert_to_literal(4),)) - - -def test_bind_c_array_type_with_itemsize_places_length_before_shape(): - array_type = BindCArrayType.get_new(2, has_strides=False, has_itemsize=True) - - assert array_type.has_itemsize is True - assert len(array_type) == 4 - assert isinstance(array_type[0], BindCPointer) - assert all(isinstance(field, NumpyInt64Type) for field in array_type[1:]) - assert "_itemsize" in type(array_type).__name__ - assert array_type.shape_is_compatible((convert_to_literal(4),)) - - -@pytest.mark.parametrize( - ("rank", "has_strides", "error"), - [ - (0, True, ValueError), - (1.5, True, TypeError), - (1, 1, TypeError), - ], -) -def test_bind_c_array_type_rejects_invalid_parameters(rank, has_strides, error): - with pytest.raises(error): - BindCArrayType.get_new(rank, has_strides) - - -def test_bind_c_array_type_validates_before_cached_lookup(): - BindCArrayType.get_new(1, True) - - with pytest.raises(TypeError, match="has_strides must be a boolean"): - BindCArrayType.get_new(1, 1) - - -def test_scope_expands_bind_c_array_to_registered_fields(): - scope = Scope(name="f", scope_type="function") - array_type = BindCArrayType.get_new(1, has_strides=True) - packed = Variable(array_type, "packed", shape=(convert_to_literal(4),)) - fields = [Variable(array_type[i], f"field_{i}") for i in range(len(array_type))] - - for i, field in enumerate(fields): - scope.insert_symbolic_alias(IndexedElement(packed, i), field) - - assert scope.collect_all_tuple_elements(packed) == fields - - -def test_bind_c_scalar_descriptor_type_expands_to_value_and_presence_pointers(): - scope = Scope(name="f", scope_type="function") - descriptor_type = BindCScalarDescriptorType() - packed = Variable(descriptor_type, "descriptor", shape=(convert_to_literal(2),)) - value = Variable(BindCPointer(), "value") - present = Variable(BindCPointer(), "present") - - scope.insert_symbolic_alias(IndexedElement(packed, 0), value) - scope.insert_symbolic_alias(IndexedElement(packed, 1), present) - - assert len(descriptor_type) == 2 - assert all(isinstance(field, BindCPointer) for field in descriptor_type) - assert descriptor_type.shape_is_compatible((convert_to_literal(2),)) - assert scope.collect_all_tuple_elements(packed) == [value, present] - - -def test_bind_c_native_array_descriptor_type_describes_required_descriptor_pointer(): - descriptor_type = BindCNativeArrayDescriptorType.get_new() - - assert descriptor_type is BindCNativeArrayDescriptorType.get_new(has_presence=False) - assert descriptor_type.has_presence is False - assert descriptor_type.rank == 1 - assert descriptor_type.container_rank == 1 - assert descriptor_type.order is None - assert descriptor_type.datatype is descriptor_type - assert len(descriptor_type) == 1 - assert isinstance(descriptor_type[0], BindCPointer) - assert descriptor_type.shape_is_compatible((convert_to_literal(1),)) - - -def test_bind_c_native_array_descriptor_type_expands_optional_presence_token(): - scope = Scope(name="f", scope_type="function") - descriptor_type = BindCNativeArrayDescriptorType.get_new(has_presence=True) - packed = Variable(descriptor_type, "native_array_descriptor", shape=(convert_to_literal(2),)) - descriptor = Variable(BindCPointer(), "descriptor") - present = Variable(BindCPointer(), "present") - - scope.insert_symbolic_alias(IndexedElement(packed, 0), descriptor) - scope.insert_symbolic_alias(IndexedElement(packed, 1), present) - - assert descriptor_type is BindCNativeArrayDescriptorType.get_new(has_presence=True) - assert descriptor_type.has_presence is True - assert len(descriptor_type) == 2 - assert all(isinstance(field, BindCPointer) for field in descriptor_type) - assert descriptor_type.shape_is_compatible((convert_to_literal(2),)) - assert scope.collect_all_tuple_elements(packed) == [descriptor, present] - - -def test_bind_c_native_array_descriptor_type_validates_presence_flag_before_cache_lookup(): - BindCNativeArrayDescriptorType.get_new(has_presence=True) - - with pytest.raises(TypeError, match="has_presence must be a boolean"): - BindCNativeArrayDescriptorType.get_new(has_presence=1) - - -def test_native_array_descriptor_argument_type_uses_completed_optional_absence_policy(): - class Policy: - def __init__(self, optional_absent): - self.optional_absent = optional_absent - - assert native_array_descriptor_argument_type(Policy(False)) is BindCNativeArrayDescriptorType.get_new( - has_presence=False - ) - assert native_array_descriptor_argument_type(Policy(True)) is BindCNativeArrayDescriptorType.get_new( - has_presence=True - ) - - -def test_fortran_visiter_visits_array_slice_with_inclusive_stop(): - array_type = NumpyNDArrayType.get_new(NumpyFloat32Type(), 1, None) - array = Variable(array_type, "values", shape=(convert_to_literal(8),)) - stop = Variable(NumpyInt64Type(), "upper") - stride = Variable(NumpyInt64Type(), "stride") - element = IndexedElement( - array, - Slice( - convert_to_literal(1), - Add(stop, convert_to_literal(1)), - stride, - ), - ) - - printer = FCodePrinter("test.f90", verbose=0) - printer.set_scope(Scope(name="f", scope_type="function")) - printer._kind = lambda expr: "i32" - assert printer._visit(element) == ("values(1_i32:upper + 1_i32 - 1_i32:stride)") - - -def test_external_interface_preserves_multidimensional_assumed_size_source_shape(): - array_type = NumpyNDArrayType.get_new(NumpyFloat64Type(), 2, "F") - array = Variable( - array_type, - "A", - memory_handling="alias", - fortran_array_category="assumed_size", - fortran_source_shape=("LDA", "*"), - is_argument=True, - ) - - printer = FCodePrinter("test.f90", verbose=0) - printer._kind = lambda expr: "f64" - - assert printer._external_interface_argument_declaration(array) == "real(f64) :: A(LDA, *)" diff --git a/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py b/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py index 3ac9bfd27..602177d4a 100644 --- a/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py +++ b/tests/wrapper/fortran/arrays/test_multidimensional_arrays.py @@ -258,57 +258,49 @@ def test_rank3_assumed_shape_accepts_fortran_ordered_strided_views(module): module.checksum3_strided(c_ordered_strided_source, contiguous_checksum) -def test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_dense_strided_and_projected_arrays_use_canonical_plan(tmp_path: Path): """Replay dense extents, positive strides, and returned storage identity.""" native_object = _compile_native_object(SOURCE, tmp_path / "native") - modules = {} selected = ("scale2_contiguous", "scale2_strided", "scale2_explicit", "shift3_strided") - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_multid_arrays" - shutil.copytree(CONTRACT_FIXTURES / "multid_arrays", contract_package) - (contract_package / "__init__.pyi").write_text( - "".join(f"from .multid_arrays import {name}\n" for name in selected), - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "scale2_contiguous") else _sole_native_module(module) - - for module in modules.values(): - dense = _matrix() - dense_out = np.zeros_like(dense, order="F") - assert module.scale2_contiguous(dense, dense_out) is dense_out - np.testing.assert_allclose(dense_out, 2.0 * dense) - - explicit_out = np.zeros_like(dense, order="F") - assert module.scale2_explicit(np.int32(4), np.int32(3), dense, explicit_out) is explicit_out - np.testing.assert_allclose(explicit_out, 4.0 * dense) - - strided = _strided_matrix() - strided_out = _strided_matrix_output(strided.shape) - assert module.scale2_strided(strided, strided_out) is strided_out - np.testing.assert_allclose(strided_out, 3.0 * strided) - - empty = _strided_matrix(0, 3) - empty_out = _strided_matrix_output(empty.shape) - assert module.scale2_strided(empty, empty_out) is empty_out - assert empty_out.shape == (0, 3) - - direct = modules["wrapper_plan"] + contract_package = tmp_path / "multid_arrays_contract" + shutil.copytree(CONTRACT_FIXTURES / "multid_arrays", contract_package) + (contract_package / "__init__.pyi").write_text( + "".join(f"from .multid_arrays import {name}\n" for name in selected), + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + imported = _import_from_build_dir(result.module_name, result.output_dir) + module = imported if hasattr(imported, "scale2_contiguous") else _sole_native_module(imported) + + dense = _matrix() + dense_out = np.zeros_like(dense, order="F") + assert module.scale2_contiguous(dense, dense_out) is dense_out + np.testing.assert_allclose(dense_out, 2.0 * dense) + + explicit_out = np.zeros_like(dense, order="F") + assert module.scale2_explicit(np.int32(4), np.int32(3), dense, explicit_out) is explicit_out + np.testing.assert_allclose(explicit_out, 4.0 * dense) + + strided = _strided_matrix() + strided_out = _strided_matrix_output(strided.shape) + assert module.scale2_strided(strided, strided_out) is strided_out + np.testing.assert_allclose(strided_out, 3.0 * strided) + + empty = _strided_matrix(0, 3) + empty_out = _strided_matrix_output(empty.shape) + assert module.scale2_strided(empty, empty_out) is empty_out + assert empty_out.shape == (0, 3) + dense = _matrix() output = np.zeros_like(dense, order="F") with pytest.raises(TypeError): - direct.scale2_strided(_reversed_fortran_matrix(), output) + module.scale2_strided(_reversed_fortran_matrix(), output) with pytest.raises(TypeError): - direct.scale2_strided(_broadcast_fortran_like_matrix(), output) + module.scale2_strided(_broadcast_fortran_like_matrix(), output) with pytest.raises(TypeError): - direct.scale2_contiguous(np.array(dense, order="C"), output) + module.scale2_contiguous(np.array(dense, order="C"), output) diff --git a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py index 1f121eed1..8e517d17f 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py +++ b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py @@ -150,25 +150,16 @@ def _assert_general_source_surface(source: Path, module) -> None: assert module.contract_same_name.module_ping() is None -@pytest.mark.parametrize( - ("route", "route_options"), - [ - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ], -) @pytest.mark.parametrize( "source", [SOURCE_NAMESPACE, TRANSITIVE_NATIVE, MULTI_MODULE, STANDALONE_ONLY, SAME_NAME_MIXED], ids=lambda path: path.stem, ) -def test_complete_general_source_preserves_namespaces_through_both_routes( +def test_complete_general_source_preserves_namespaces_through_canonical_plan( tmp_path: Path, source: Path, - route: str, - route_options: dict[str, bool], ): - result = build_fortran_extension(source, output_dir=tmp_path / route, **route_options) + result = build_fortran_extension(source, output_dir=tmp_path / "build") module = _import_extension(result.module_name, result.output_dir) _assert_general_source_surface(source, module) @@ -275,7 +266,6 @@ def test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "plan_build", - _force_wrapper_plan_route=True, ) plan_module = _import_extension(plan_result.module_name, plan_result.output_dir) assert not hasattr(plan_module, "facade") diff --git a/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py b/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py index 69d7f1de1..a05702c75 100644 --- a/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py +++ b/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py @@ -5,7 +5,6 @@ import numpy as np from tests.wrapper.fortran._support import _build_source_or_generated_pyi_and_import, wrapper_source -from x2py.pipeline import build as build_pipeline CALLBACK_ALL_F90_SOURCE = wrapper_source("fcallback_all_f90.f90") CONTRACT_FIXTURES = Path(__file__).parent / "contracts" @@ -14,12 +13,7 @@ def test_immediate_callbacks_cover_all_supported_argument_shapes( pyi_parity_build_mode: str, tmp_path: Path, - monkeypatch, ): - def fail_legacy_lowering(*_args, **_kwargs): - raise AssertionError("eligible callback units must not enter legacy lowering") - - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) module = _build_source_or_generated_pyi_and_import( CALLBACK_ALL_F90_SOURCE, tmp_path, diff --git a/tests/wrapper/fortran/derived_types/test_derived_type_methods.py b/tests/wrapper/fortran/derived_types/test_derived_type_methods.py index 0f25d792c..5f605ac16 100644 --- a/tests/wrapper/fortran/derived_types/test_derived_type_methods.py +++ b/tests/wrapper/fortran/derived_types/test_derived_type_methods.py @@ -28,4 +28,7 @@ def test_modern_fortran_derived_type_exposes_class_and_type_bound_methods( pyi_parity_build_mode, ) + assert "make(n, fill_value) -> vector_store" in module.vector_store.make.__doc__ + assert "n : int64" in module.vector_store.make.__doc__ + assert "wrapped native instance" not in module.vector_store.make.__doc__ _assert_modern_class_examples(module) diff --git a/tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py b/tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py index 8bf499f6b..034a56aef 100644 --- a/tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py +++ b/tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py @@ -1,4 +1,4 @@ -"""Compiled legacy-oracle parity for dependency-closed Phase 8 object lanes.""" +"""Compiled canonical wrapper-plan coverage for Phase 8 object lanes.""" from __future__ import annotations @@ -9,14 +9,13 @@ import pytest from tests.wrapper.fortran._support import ( - _build_source_legacy_and_import, + _build_source_and_import, _compile_native_object, _import_from_build_dir, _sole_native_module, wrapper_source, ) from x2py import build_pyi_extension -from x2py.pipeline import build as build_pipeline from x2py.runtime.handles import AllocatableArray, PointerArray DERIVED_BOUNDARY_F90_SOURCE = wrapper_source("fderived_boundary_f90.f90") @@ -234,24 +233,16 @@ def reset_final_count() -> None: ... """ -def _build_routes(tmp_path: Path): +def _build_point_boundary(tmp_path: Path): native_object = _compile_native_object(DERIVED_BOUNDARY_F90_SOURCE, tmp_path / "native") - modules = [] - results = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - CONTRACT, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - results.append(result) - return tuple(modules), tuple(results) + result = build_pyi_extension( + CONTRACT, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + return module, result def _exercise_point_boundary(module): @@ -280,20 +271,14 @@ def _exercise_point_boundary(module): point.x = 12.0 -def test_scalar_derived_objects_match_legacy_and_wrapper_plan_routes(tmp_path: Path): - modules, results = _build_routes(tmp_path) - - for module in modules: - _exercise_point_boundary(module) - with pytest.raises(TypeError): - module.point() - - foreign_point = modules[0].make_point(np.float64(3.0), np.float64(4.0)) - with pytest.raises(TypeError, match="Expected exact wrapper type point"): - modules[1].point_sum(foreign_point) +def test_scalar_derived_objects_use_canonical_plan(tmp_path: Path): + module, result = _build_point_boundary(tmp_path) + _exercise_point_boundary(module) + with pytest.raises(TypeError): + module.point() - generated_c = (results[1].output_dir / "fderived_boundary_phase8_opaque_wrapper.c").read_text(encoding="utf-8") - generated_fortran = (results[1].output_dir / "bind_c_fderived_boundary_phase8_opaque_wrapper.f90").read_text( + generated_c = (result.output_dir / "fderived_boundary_phase8_opaque_wrapper.c").read_text(encoding="utf-8") + generated_fortran = (result.output_dir / "bind_c_fderived_boundary_phase8_opaque_wrapper.f90").read_text( encoding="utf-8" ) assert "static PyObject * wrap_point_sum" in generated_c @@ -306,28 +291,6 @@ def test_scalar_derived_objects_match_legacy_and_wrapper_plan_routes(tmp_path: P assert "allocate(result_value, stat=x2py_allocation_status)" in generated_fortran -def test_eligible_derived_contract_selects_production_plan_without_legacy_lowering( - tmp_path: Path, - monkeypatch, -): - native_object = _compile_native_object(DERIVED_BOUNDARY_F90_SOURCE, tmp_path / "native") - - def fail_legacy_lowering(*_args, **_kwargs): - raise AssertionError("eligible Phase 8 contract must not invoke semantic_ir_to_codegen_ast") - - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) - result = build_pyi_extension( - CONTRACT, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / "production", - ) - module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) - - _exercise_point_boundary(module) - assert (result.output_dir / "fderived_boundary_phase8_opaque_wrapper.c").is_file() - - def test_plain_module_derived_proxy_reads_and_writes_live_members(tmp_path: Path): native_object = _compile_native_object(PLAIN_MODULE_SOURCE, tmp_path / "native") result = build_pyi_extension( @@ -335,7 +298,6 @@ def test_plain_module_derived_proxy_reads_and_writes_live_members(tmp_path: Path native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "wrapper_plan", - _force_wrapper_plan_route=True, ) module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) @@ -409,7 +371,6 @@ def test_aliased_module_derived_object_uses_direct_live_field_handles(tmp_path: native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "wrapper_plan", - _force_wrapper_plan_route=True, ) module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) @@ -445,127 +406,104 @@ def test_aliased_module_derived_object_uses_direct_live_field_handles(tmp_path: def test_derived_module_constant_returns_independent_owned_values(tmp_path: Path): native_object = _compile_native_object(DERIVED_CONSTANT_SOURCE, tmp_path / "native") - modules = [] - results = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract = tmp_path / route / "fmodule_vars_f90.pyi" - contract.parent.mkdir() - contract.write_text(DERIVED_CONSTANT_CONTRACT, encoding="utf-8") - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / f"{route}_build", - **route_kwargs, - ) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - results.append(result) - - for module in modules: - first = module.black - second = module.black - assert first is not second - first.r = np.int32(17) - assert first.r == np.int32(17) - assert second.r == np.int32(0) - assert module.black.r == np.int32(0) - assert module.black_sum() == np.int32(0) - - bridge = (results[1].output_dir / "bind_c_fmodule_vars_f90_wrapper.f90").read_text(encoding="utf-8") + contract = tmp_path / "contract" / "fmodule_vars_f90.pyi" + contract.parent.mkdir() + contract.write_text(DERIVED_CONSTANT_CONTRACT, encoding="utf-8") + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + first = module.black + second = module.black + assert first is not second + first.r = np.int32(17) + assert first.r == np.int32(17) + assert second.r == np.int32(0) + assert module.black.r == np.int32(0) + assert module.black_sum() == np.int32(0) + + bridge = (result.output_dir / "bind_c_fmodule_vars_f90_wrapper.f90").read_text(encoding="utf-8") assert "result = c_null_ptr" in bridge assert "allocate(value, stat=x2py_allocation_status)" in bridge assert "value = native_black" in bridge assert "result = c_loc(value)" in bridge -def test_fixed_string_fields_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_fixed_string_fields_use_canonical_plan(tmp_path: Path): source = tmp_path / "native" / "fderived_string_phase8.f90" source.parent.mkdir() source.write_text(STRING_FIELD_SOURCE, encoding="utf-8") native_object = _compile_native_object(source, tmp_path / "native_build") - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract = tmp_path / route / "fderived_string_phase8.pyi" - contract.parent.mkdir() - contract.write_text(STRING_FIELD_CONTRACT, encoding="utf-8") - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / f"{route}_build", - **route_kwargs, - ) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - - for module in modules: - current = module.current - assert current.label == "start " - current.label = "edited " - assert module.current_label() == "edited " - module.reset_label() - assert current.label == "native " - with pytest.raises(TypeError, match="exactly 8 bytes"): - current.label = "short" - - -def test_pointer_field_descriptor_views_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + contract = tmp_path / "contract" / "fderived_string_phase8.pyi" + contract.parent.mkdir() + contract.write_text(STRING_FIELD_CONTRACT, encoding="utf-8") + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + current = module.current + assert current.label == "start " + current.label = "edited " + assert module.current_label() == "edited " + module.reset_label() + assert current.label == "native " + with pytest.raises(TypeError, match="exactly 8 bytes"): + current.label = "short" + + +def test_pointer_field_descriptor_views_use_canonical_plan(tmp_path: Path): source = tmp_path / "native" / "fderived_pointer_phase8.f90" source.parent.mkdir() source.write_text(POINTER_FIELD_SOURCE, encoding="utf-8") native_object = _compile_native_object(source, tmp_path / "native_build") - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract = tmp_path / route / "fderived_pointer_phase8.pyi" - contract.parent.mkdir() - contract.write_text(POINTER_FIELD_CONTRACT, encoding="utf-8") - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / f"{route}_build", - **route_kwargs, - ) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - - for module in modules: - owner = module.current - handle = owner.values - assert isinstance(handle, PointerArray) - assert handle.owner is owner - assert handle.to_numpy() is None - - module.associate_strided() - view = handle.to_numpy() - np.testing.assert_allclose(view, np.array([6.0, 8.0], dtype=np.float64)) - assert view.shape == (2,) - assert view.strides == (16,) - view[1] = np.float64(12.0) - assert module.current_sum() == np.float64(18.0) - - independent = view.copy() - with pytest.raises(AttributeError): - owner.values = handle - handle.nullify() - assert handle.to_numpy() is None - np.testing.assert_allclose(independent, np.array([6.0, 12.0], dtype=np.float64)) + contract = tmp_path / "contract" / "fderived_pointer_phase8.pyi" + contract.parent.mkdir() + contract.write_text(POINTER_FIELD_CONTRACT, encoding="utf-8") + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + owner = module.current + handle = owner.values + assert isinstance(handle, PointerArray) + assert handle.owner is owner + assert handle.to_numpy() is None + + module.associate_strided() + view = handle.to_numpy() + np.testing.assert_allclose(view, np.array([6.0, 8.0], dtype=np.float64)) + assert view.shape == (2,) + assert view.strides == (16,) + view[1] = np.float64(12.0) + assert module.current_sum() == np.float64(18.0) + + independent = view.copy() + with pytest.raises(AttributeError): + owner.values = handle + handle.nullify() + assert handle.to_numpy() is None + np.testing.assert_allclose(independent, np.array([6.0, 12.0], dtype=np.float64)) def test_value_copy_and_optional_derived_inputs_match_source_oracle(tmp_path: Path): source = tmp_path / "source" / "fderived_value_phase8.f90" source.parent.mkdir() source.write_text(VALUE_AND_OPTIONAL_SOURCE, encoding="utf-8") - source_module = _build_source_legacy_and_import( + source_module = _build_source_and_import( source, tmp_path / "source_build", { @@ -583,8 +521,7 @@ def test_value_copy_and_optional_derived_inputs_match_source_oracle(tmp_path: Pa contract, native_objects=[native_object], native_include_dirs=[native_object.parent], - output_dir=tmp_path / "wrapper_plan", - _force_wrapper_plan_route=True, + output_dir=tmp_path / "contract_build", ) direct_module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) @@ -626,34 +563,26 @@ def test_borrowed_child_retains_owner_and_finalizes_exactly_once(tmp_path: Path) source.parent.mkdir() source.write_text(BORROWED_FINALIZER_SOURCE, encoding="utf-8") native_object = _compile_native_object(source, tmp_path / "native") - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract = tmp_path / route / "fderived_finalizer_phase8.pyi" - contract.parent.mkdir() - contract.write_text(BORROWED_FINALIZER_CONTRACT, encoding="utf-8") - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / f"{route}_build", - **route_kwargs, - ) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - - for module in modules: - owner = module.make_parent() - module.reset_final_count() - borrowed = owner.value - if module is modules[1]: - assert borrowed._x2py_owner is owner - del owner - gc.collect() - assert module.get_final_count() == np.int32(0) - - del borrowed - gc.collect() - gc.collect() - assert module.get_final_count() == np.int32(1) + contract = tmp_path / "contract" / "fderived_finalizer_phase8.pyi" + contract.parent.mkdir() + contract.write_text(BORROWED_FINALIZER_CONTRACT, encoding="utf-8") + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + owner = module.make_parent() + module.reset_final_count() + borrowed = owner.value + assert borrowed._x2py_owner is owner + del owner + gc.collect() + assert module.get_final_count() == np.int32(0) + + del borrowed + gc.collect() + gc.collect() + assert module.get_final_count() == np.int32(1) diff --git a/tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py b/tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py index c335cf882..247a86b56 100644 --- a/tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py +++ b/tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py @@ -10,7 +10,6 @@ wrapper_source, ) from x2py import build_pyi_extension -from x2py.pipeline import build as build_pipeline SOURCE = wrapper_source("fclasses_f90.f90") CONTRACT = Path(__file__).parent / "contracts" / "fbound_constructor_phase9" / "__init__.pyi" @@ -18,14 +17,8 @@ def test_bound_constructor_replaces_field_initialization_and_reuses_method_plan( tmp_path: Path, - monkeypatch, ): native_object = _compile_native_object(SOURCE, tmp_path / "native") - - def fail_legacy_lowering(*_args, **_kwargs): - raise AssertionError("eligible Phase 9 contract must not invoke semantic_ir_to_codegen_ast") - - monkeypatch.setattr(build_pipeline, "semantic_ir_to_codegen_ast", fail_legacy_lowering) result = build_pyi_extension( CONTRACT, native_objects=[native_object], @@ -34,6 +27,17 @@ def fail_legacy_lowering(*_args, **_kwargs): ) module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + assert "_x2py_class_" not in module.__doc__ + assert "Constructor\n-----------\nvector(dx, dy) -> vector" in module.vector.__doc__ + assert "Fields\n------\nx : float64\ny : float64" in module.vector.__doc__ + assert "Methods\n-------\nshift(dx, dy) -> None" in module.vector.__doc__ + assert "vector(dx, dy) -> vector" in module.vector.__init__.__doc__ + assert "dx : float64" in module.vector.__init__.__doc__ + assert "shift(dx, dy) -> None" in module.vector.shift.__doc__ + assert "Updates the wrapped native instance in place." in module.vector.shift.__doc__ + assert "owner" not in module.vector.shift.__doc__ + assert "Assignment writes through to native storage." in module.vector.x.__doc__ + value = module.vector(np.float64(2.0), np.float64(3.0)) assert (value.x, value.y) == (np.float64(2.0), np.float64(3.0)) value.shift(np.float64(1.0), np.float64(-1.0)) diff --git a/tests/wrapper/fortran/derived_types/test_pointers.py b/tests/wrapper/fortran/derived_types/test_pointers.py index cad5f6a63..f142d5e90 100644 --- a/tests/wrapper/fortran/derived_types/test_pointers.py +++ b/tests/wrapper/fortran/derived_types/test_pointers.py @@ -232,21 +232,16 @@ def test_pointer_descriptor_views_preserve_slice_shape_strides_and_parent_lifeti np.testing.assert_allclose(field_view, np.array([6.0, 8.0], dtype=np.float64)) -def test_module_native_array_handles_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_module_native_array_handles_use_canonical_plan(tmp_path: Path): """Replay module pointer/allocatable handles without derived-field owners.""" source = tmp_path / "native" / "fpointer_handles_f90.f90" source.parent.mkdir() source.write_text(POINTER_HANDLE_SOURCE, encoding="utf-8") native_object = _compile_native_object(source, tmp_path / "native_build") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract = tmp_path / f"{route}_pointer_handles" / "fpointer_handles_f90.pyi" - contract.parent.mkdir() - contract.write_text( - """from x2py.contracts import Aliased, Allocatable, Annotated, Float64, Pointer, PointerAssociation, PointerPolicy + contract = tmp_path / "pointer_handles" / "fpointer_handles_f90.pyi" + contract.parent.mkdir() + contract.write_text( + """from x2py.contracts import Aliased, Allocatable, Annotated, Float64, Pointer, PointerAssociation, PointerPolicy module_values: Annotated[ Pointer[Float64[:]], @@ -273,45 +268,43 @@ def sum_values(values: Float64[:]) -> Float64: ... def sum_pointer_descriptor(values: Pointer[Float64[:]]) -> Float64: ... def sum_allocatable_descriptor(values: Allocatable[Float64[:]]) -> Float64: ... """, - encoding="utf-8", - ) - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - modules[route] = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) - - for module in modules.values(): - pointer_handle = module.module_values - allocatable_handle = module.module_allocatable - assert isinstance(pointer_handle, PointerArray) - assert isinstance(allocatable_handle, AllocatableArray) - assert module.module_values is pointer_handle - assert module.module_allocatable is allocatable_handle - assert module.sum_pointer_descriptor(pointer_handle) == np.float64(-1.0) - assert module.sum_allocatable_descriptor(allocatable_handle) == np.float64(-1.0) - - module.associate_module_slice() - assert pointer_handle.associated is True - np.testing.assert_allclose(pointer_handle.to_numpy(), np.array([2.0, 4.0])) - assert module.sum_pointer_descriptor(pointer_handle) == np.float64(6.0) - with pytest.raises(ValueError, match="noncontiguous"): - module.sum_values(pointer_handle) - - module.associate_module_contiguous() - assert module.sum_values(pointer_handle) == np.float64(9.0) - pointer_handle.nullify() - assert pointer_handle.associated is False - - module.allocate_module_values() - assert allocatable_handle.allocated is True - np.testing.assert_allclose(allocatable_handle.to_numpy(), np.array([10.0, 20.0, 30.0])) - assert module.sum_allocatable_descriptor(allocatable_handle) == np.float64(60.0) - allocatable_handle.deallocate() - assert allocatable_handle.allocated is False + encoding="utf-8", + ) + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + pointer_handle = module.module_values + allocatable_handle = module.module_allocatable + assert isinstance(pointer_handle, PointerArray) + assert isinstance(allocatable_handle, AllocatableArray) + assert module.module_values is pointer_handle + assert module.module_allocatable is allocatable_handle + assert module.sum_pointer_descriptor(pointer_handle) == np.float64(-1.0) + assert module.sum_allocatable_descriptor(allocatable_handle) == np.float64(-1.0) + + module.associate_module_slice() + assert pointer_handle.associated is True + np.testing.assert_allclose(pointer_handle.to_numpy(), np.array([2.0, 4.0])) + assert module.sum_pointer_descriptor(pointer_handle) == np.float64(6.0) + with pytest.raises(ValueError, match="noncontiguous"): + module.sum_values(pointer_handle) + + module.associate_module_contiguous() + assert module.sum_values(pointer_handle) == np.float64(9.0) + pointer_handle.nullify() + assert pointer_handle.associated is False + + module.allocate_module_values() + assert allocatable_handle.allocated is True + np.testing.assert_allclose(allocatable_handle.to_numpy(), np.array([10.0, 20.0, 30.0])) + assert module.sum_allocatable_descriptor(allocatable_handle) == np.float64(60.0) + allocatable_handle.deallocate() + assert allocatable_handle.allocated is False def test_pointer_array_handles_block_on_unsupported_result_owner_policy( diff --git a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py index 5deb24cd4..fe1d9bf5d 100644 --- a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py +++ b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py @@ -96,26 +96,21 @@ def test_editable_contract_can_use_native_order_arguments_without_native_call(tm assert point.code == np.int32(107) -def test_raw_array_addresses_match_legacy_and_wrapper_plan_routes(tmp_path: Path): - """Replay one required raw array address through both wrapper routes.""" +def test_raw_array_addresses_use_canonical_plan(tmp_path: Path): + """Replay one required raw array address through the canonical plan.""" native_object = _compile_native_object(NATIVE_CALL_EXAMPLES_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_raw_array_address" - contract_package.mkdir() - (contract_package / "__init__.pyi").write_text( - ( - "from .fnative_call_examples_f90 import fill_vector_raw\n" - "from .fnative_call_examples_f90 import shift_matrix_raw_c\n" - "from .fnative_call_examples_f90 import shift_matrix_raw_f\n" - ), - encoding="utf-8", - ) - (contract_package / "fnative_call_examples_f90.pyi").write_text( - """from x2py.contracts import Addr, Annotated, Float64, Int32, ORDER_F, bind + contract_package = tmp_path / "raw_array_address" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + ( + "from .fnative_call_examples_f90 import fill_vector_raw\n" + "from .fnative_call_examples_f90 import shift_matrix_raw_c\n" + "from .fnative_call_examples_f90 import shift_matrix_raw_f\n" + ), + encoding="utf-8", + ) + (contract_package / "fnative_call_examples_f90.pyi").write_text( + """from x2py.contracts import Addr, Annotated, Float64, Int32, ORDER_F, bind @bind("fill_vector") def fill_vector_raw(n: Int32[()], values: Addr(Float64[n])) -> None: ... @@ -136,37 +131,35 @@ def shift_matrix_raw_f( out: Annotated[Addr(Float64[n, m]), ORDER_F] ) -> None: ... """, - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fill_vector_raw") else _sole_native_module(module) - - for module in modules.values(): - vector_size = np.array(4, dtype=np.int32) - raw_vector = np.empty(4, dtype=np.float64) - assert module.fill_vector_raw(vector_size, raw_vector.ctypes.data) is None - np.testing.assert_allclose(raw_vector, np.array([1.5, 3.0, 4.5, 6.0], dtype=np.float64)) - - with pytest.raises(TypeError): - module.fill_vector_raw(vector_size, raw_vector) - with pytest.raises(TypeError): - module.fill_vector_raw(vector_size, "not an address") - - rows = np.array(2, dtype=np.int32) - cols = np.array(3, dtype=np.int32) - for order, function_name in (("C", "shift_matrix_raw_c"), ("F", "shift_matrix_raw_f")): - matrix = np.array([[1.0, 3.0, 5.0], [2.0, 4.0, 6.0]], dtype=np.float64, order=order) - shifted = np.empty((2, 3), dtype=np.float64, order=order) - function = getattr(module, function_name) - assert function(rows, cols, matrix.ctypes.data, shifted.ctypes.data) is None - np.testing.assert_allclose(shifted, matrix + 10.0) + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "fill_vector_raw") else _sole_native_module(package) + + vector_size = np.array(4, dtype=np.int32) + raw_vector = np.empty(4, dtype=np.float64) + assert module.fill_vector_raw(vector_size, raw_vector.ctypes.data) is None + np.testing.assert_allclose(raw_vector, np.array([1.5, 3.0, 4.5, 6.0], dtype=np.float64)) + + with pytest.raises(TypeError): + module.fill_vector_raw(vector_size, raw_vector) + with pytest.raises(TypeError): + module.fill_vector_raw(vector_size, "not an address") + + rows = np.array(2, dtype=np.int32) + cols = np.array(3, dtype=np.int32) + for order, function_name in (("C", "shift_matrix_raw_c"), ("F", "shift_matrix_raw_f")): + matrix = np.array([[1.0, 3.0, 5.0], [2.0, 4.0, 6.0]], dtype=np.float64, order=order) + shifted = np.empty((2, 3), dtype=np.float64, order=order) + function = getattr(module, function_name) + assert function(rows, cols, matrix.ctypes.data, shifted.ctypes.data) is None + np.testing.assert_allclose(shifted, matrix + 10.0) def test_copy_f_preserves_logical_axes_through_binding_owned_temporary(tmp_path: Path): @@ -224,7 +217,6 @@ def shift_matrix_copy_f_projected( native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "pyi_build", - _force_wrapper_plan_route=True, ) module = _import_from_build_dir(result.module_name, result.output_dir) module = module if hasattr(module, "shift_matrix_copy_f") else _sole_native_module(module) @@ -253,22 +245,17 @@ def shift_matrix_copy_f_projected( module.shift_matrix_copy_f(rows, cols, np.asfortranarray(values), out) -def test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_fixed_string_storage_and_raw_address_use_canonical_plan(tmp_path: Path): """Replay both fixed address boundaries through one existing native routine.""" native_object = _compile_native_object(NATIVE_CALL_EXAMPLES_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_fixed_string_addresses" - contract_package.mkdir() - (contract_package / "__init__.pyi").write_text( - "from .fnative_call_examples_f90 import fixed_inout_raw, fixed_inout_storage\n", - encoding="utf-8", - ) - (contract_package / "fnative_call_examples_f90.pyi").write_text( - """from x2py.contracts import Addr, String, bind + contract_package = tmp_path / "fixed_string_addresses" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + "from .fnative_call_examples_f90 import fixed_inout_raw, fixed_inout_storage\n", + encoding="utf-8", + ) + (contract_package / "fnative_call_examples_f90.pyi").write_text( + """from x2py.contracts import Addr, String, bind @bind("fixed_inout") def fixed_inout_raw(label: Addr(String[8])) -> None: ... @@ -276,39 +263,37 @@ def fixed_inout_raw(label: Addr(String[8])) -> None: ... @bind("fixed_inout") def fixed_inout_storage(label: String[8][()]) -> None: ... """, - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fixed_inout_raw") else _sole_native_module(module) - - for module in modules.values(): - raw_label = ctypes.create_string_buffer(8) - raw_label.raw = b"abc " - assert module.fixed_inout_raw(ctypes.addressof(raw_label)) is None - assert raw_label.raw == b"Xbc !" - - storage_label = np.array("abc ", dtype="S8") - assert module.fixed_inout_storage(storage_label) is None - assert storage_label[()] == b"Xbc !" - - with pytest.raises(TypeError): - module.fixed_inout_raw("abc ") - with pytest.raises(TypeError, match="itemsize 8"): - module.fixed_inout_storage(np.array("abc", dtype="S3")) - with pytest.raises(TypeError): - module.fixed_inout_storage(np.array([b"abc "], dtype="S8")) - with pytest.raises(TypeError): - module.fixed_inout_storage(np.array("abc ", dtype="U8")) - with pytest.raises(TypeError): - module.fixed_inout_storage(np.array(b"abc ", dtype=object)) - read_only = np.array("abc ", dtype="S8") - read_only.flags.writeable = False - with pytest.raises(TypeError, match="writeable"): - module.fixed_inout_storage(read_only) + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "fixed_inout_raw") else _sole_native_module(package) + + raw_label = ctypes.create_string_buffer(8) + raw_label.raw = b"abc " + assert module.fixed_inout_raw(ctypes.addressof(raw_label)) is None + assert raw_label.raw == b"Xbc !" + + storage_label = np.array("abc ", dtype="S8") + assert module.fixed_inout_storage(storage_label) is None + assert storage_label[()] == b"Xbc !" + + with pytest.raises(TypeError): + module.fixed_inout_raw("abc ") + with pytest.raises(TypeError, match="itemsize 8"): + module.fixed_inout_storage(np.array("abc", dtype="S3")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array([b"abc "], dtype="S8")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array("abc ", dtype="U8")) + with pytest.raises(TypeError): + module.fixed_inout_storage(np.array(b"abc ", dtype=object)) + read_only = np.array("abc ", dtype="S8") + read_only.flags.writeable = False + with pytest.raises(TypeError, match="writeable"): + module.fixed_inout_storage(read_only) diff --git a/tests/wrapper/fortran/function_calls/test_optional_arguments.py b/tests/wrapper/fortran/function_calls/test_optional_arguments.py index 81c859c31..730358c79 100644 --- a/tests/wrapper/fortran/function_calls/test_optional_arguments.py +++ b/tests/wrapper/fortran/function_calls/test_optional_arguments.py @@ -9,7 +9,6 @@ from x2py import build_pyi_extension from tests.wrapper.fortran._support import ( _compile_native_object, - _build_source_legacy_and_import, _build_source_or_generated_pyi_and_import, _build_source_wrapper_plan_and_import, _import_from_build_dir, @@ -116,51 +115,32 @@ def alloc_state(value: Annotated[Float64, Immutable] | None = ...) -> Int32: ... encoding="utf-8", ) - results = [] - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - entry, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - results.append(result) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - - assert [tuple(path.name for path in result.generated_sources) for result in results] == [ - ( - "bind_c_scalar_optional_descriptors_wrapper.f90", - "scalar_optional_descriptors_wrapper.c", - "scalar_optional_descriptors_wrapper.h", - ), - ( - "bind_c_scalar_optional_descriptors_wrapper.f90", - "scalar_optional_descriptors_wrapper.c", - "scalar_optional_descriptors_wrapper.h", - ), - ] - for module in modules: - assert module.alloc_state() == np.int32(0) - assert module.alloc_state(None) == np.int32(1) - assert module.alloc_state(np.float64(2.5)) == np.int32(2) - with pytest.raises(TypeError): - module.alloc_state("bad") - legacy_header = (tmp_path / "legacy" / "scalar_optional_descriptors_wrapper.h").read_text(encoding="utf-8") - plan_c = (tmp_path / "wrapper_plan" / "scalar_optional_descriptors_wrapper.c").read_text(encoding="utf-8") - assert "int32_t bind_c_alloc_state(void*, void*);" in legacy_header + result = build_pyi_extension( + entry, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + + assert tuple(path.name for path in result.generated_sources) == ( + "bind_c_scalar_optional_descriptors_wrapper.f90", + "scalar_optional_descriptors_wrapper.c", + "scalar_optional_descriptors_wrapper.h", + ) + assert module.alloc_state() == np.int32(0) + assert module.alloc_state(None) == np.int32(1) + assert module.alloc_state(np.float64(2.5)) == np.int32(2) + with pytest.raises(TypeError): + module.alloc_state("bad") + plan_c = (result.output_dir / "scalar_optional_descriptors_wrapper.c").read_text(encoding="utf-8") assert "int32_t bind_c_alloc_state(void * value, void * value_present);" in plan_c - legacy = modules[0] - assert "Omit to make the native optional dummy absent." in legacy.alloc_state.__doc__ - assert "Pass None for a present unallocated or unassociated descriptor." in legacy.alloc_state.__doc__ - assert "Default is None." not in legacy.alloc_state.__doc__ + assert "Omit to make the native optional dummy absent." in module.alloc_state.__doc__ + assert "Pass None for a present unallocated or unassociated descriptor." in module.alloc_state.__doc__ + assert "Default is None." not in module.alloc_state.__doc__ -def test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_optional_array_descriptors_preserve_presence_and_storage_state(tmp_path: Path): """Distinguish omitted/None from present absent-state descriptor handles.""" source = tmp_path / "optional_array_descriptors.f90" source.write_text( @@ -203,36 +183,25 @@ def pointer_state(values: Pointer[Float64[:]] | None = ...) -> Int32: ... encoding="utf-8", ) - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / f"{route}_array_descriptors", - **route_kwargs, - ) - modules[route] = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "array_descriptors", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) values = np.array([1.0, 2.0, 3.0], dtype=np.float64) - for module in modules.values(): - for function_name in ("alloc_state", "pointer_state"): - function = getattr(module, function_name) - assert function() == np.int32(0) - assert function(None) == np.int32(0) + for function_name in ("alloc_state", "pointer_state"): + function = getattr(module, function_name) + assert function() == np.int32(0) + assert function(None) == np.int32(0) - direct = modules["wrapper_plan"] for function_name, pointer in (("alloc_state", False), ("pointer_state", True)): - function = getattr(direct, function_name) + function = getattr(module, function_name) assert function(_optional_descriptor_handle(None, pointer=pointer)) == np.int32(1) assert function(_optional_descriptor_handle(values, pointer=pointer)) == np.int32(6) - with pytest.raises(TypeError, match=r"numpy\.ndarray"): - modules["legacy"].alloc_state(_optional_descriptor_handle(None, pointer=False)) - def test_optional_arguments_drive_fortran_present_behavior( pyi_parity_build_mode: str, @@ -323,16 +292,7 @@ def test_fixed_form_optional_arguments_drive_fortran_present_behavior( assert module.optional_scale(base=np.int32(3), factor=np.int32(6)) == np.int32(9) -def test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states(tmp_path: Path): - legacy = _build_source_legacy_and_import( - OPTIONAL_FIXED_SOURCE, - tmp_path / "legacy", - { - "bind_c_foptional_fixed_wrapper.f90", - "foptional_fixed_wrapper.c", - "foptional_fixed_wrapper.h", - }, - ) +def test_fixed_optional_scalar_plan_matches_all_presence_states(tmp_path: Path): wrapper_plan, result = _build_source_wrapper_plan_and_import( OPTIONAL_FIXED_SOURCE, tmp_path / "wrapper_plan", @@ -343,33 +303,25 @@ def test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states(tm "foptional_fixed_wrapper.c", "foptional_fixed_wrapper.h", } - legacy_header = (tmp_path / "legacy" / "foptional_fixed_wrapper.h").read_text(encoding="utf-8") plan_c = (tmp_path / "wrapper_plan" / "wrapper_plan_build" / "foptional_fixed_wrapper.c").read_text( encoding="utf-8" ) - assert "int32_t bind_c_optional_scale(int32_t, void*);" in legacy_header assert "int32_t bind_c_optional_scale(int32_t base, void * factor);" in plan_c - for module in (legacy, wrapper_plan): - assert module.optional_scale(np.int32(3)) == np.int32(3) - assert module.optional_scale(np.int32(3), None) == np.int32(3) - assert module.optional_scale(np.int32(3), np.int32(4)) == np.int32(7) - assert module.optional_scale(base=np.int32(3), factor=np.int32(6)) == np.int32(9) - with pytest.raises(TypeError): - module.optional_scale(np.int32(3), "bad") + assert wrapper_plan.optional_scale(np.int32(3)) == np.int32(3) + assert wrapper_plan.optional_scale(np.int32(3), None) == np.int32(3) + assert wrapper_plan.optional_scale(np.int32(3), np.int32(4)) == np.int32(7) + assert wrapper_plan.optional_scale(base=np.int32(3), factor=np.int32(6)) == np.int32(9) + with pytest.raises(TypeError): + wrapper_plan.optional_scale(np.int32(3), "bad") -def test_optional_array_buffers_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_optional_array_buffers_preserve_omission_and_identity(tmp_path: Path): """Replay omitted, explicit-None, and present ordinary array storage.""" native_object = _compile_native_object(OPTIONAL_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_optional_arrays" - shutil.copytree(CONTRACT_FIXTURES / "foptional_f90", contract_package) - (contract_package / "foptional_f90.pyi").write_text( - """from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call + contract_package = tmp_path / "optional_arrays" + shutil.copytree(CONTRACT_FIXTURES / "foptional_f90", contract_package) + (contract_package / "foptional_f90.pyi").write_text( + """from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call @native_call([Arg(0), Addr(Arg(1))]) def mutate_optional( @@ -383,34 +335,32 @@ def fill_optional( values: Float64[::] = ... ) -> Returns["values", Float64[::]] | None: ... """, - encoding="utf-8", - ) - (contract_package / "__init__.pyi").write_text( - "from .foptional_f90 import mutate_optional\nfrom .foptional_f90 import fill_optional\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "mutate_optional") else _sole_native_module(module) - - for module in modules.values(): - assert module.mutate_optional() is None - assert module.mutate_optional(None, np.float64(2.0)) is None - values = np.array([1.0, 2.0], dtype=np.float64) - assert module.mutate_optional(values, np.float64(2.5)) is None - np.testing.assert_array_equal(values, np.array([3.5, 4.5])) - - output = np.empty(3, dtype=np.float64) - assert module.fill_optional(np.int32(3), output) is output - np.testing.assert_array_equal(output, np.array([11.0, 12.0, 13.0])) - assert module.fill_optional(np.int32(3)) is None - assert module.fill_optional(np.int32(3), None) is None + encoding="utf-8", + ) + (contract_package / "__init__.pyi").write_text( + "from .foptional_f90 import mutate_optional\nfrom .foptional_f90 import fill_optional\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + imported = _import_from_build_dir(result.module_name, result.output_dir) + module = imported if hasattr(imported, "mutate_optional") else _sole_native_module(imported) + + assert module.mutate_optional() is None + assert module.mutate_optional(None, np.float64(2.0)) is None + values = np.array([1.0, 2.0], dtype=np.float64) + assert module.mutate_optional(values, np.float64(2.5)) is None + np.testing.assert_array_equal(values, np.array([3.5, 4.5])) + + output = np.empty(3, dtype=np.float64) + assert module.fill_optional(np.int32(3), output) is output + np.testing.assert_array_equal(output, np.array([11.0, 12.0, 13.0])) + assert module.fill_optional(np.int32(3)) is None + assert module.fill_optional(np.int32(3), None) is None with pytest.raises(TypeError): - modules["wrapper_plan"].fill_optional(np.int32(3), np.empty(3, dtype=np.float32)) + module.fill_optional(np.int32(3), np.empty(3, dtype=np.float32)) diff --git a/tests/wrapper/fortran/function_calls/test_output_arguments.py b/tests/wrapper/fortran/function_calls/test_output_arguments.py index 115a3a9aa..f4504222e 100644 --- a/tests/wrapper/fortran/function_calls/test_output_arguments.py +++ b/tests/wrapper/fortran/function_calls/test_output_arguments.py @@ -39,6 +39,10 @@ def test_output_arguments_and_multiple_results_follow_python_projection_rules( assert "scalar_status(n) -> int32" in module.scalar_status.__doc__ assert "status : int32" in module.scalar_status.__doc__ assert "fill_vector(n, values) -> ndarray[float64]" in module.fill_vector.__doc__ + assert "Parameters\n----------" in module.fill_vector.__doc__ + assert "Returns\n-------" in module.fill_vector.__doc__ + assert "Raises\n------" in module.fill_vector.__doc__ + assert "Native code may update this value; the updated value is returned." in module.fill_vector.__doc__ assert "Direction:" not in module.fill_vector.__doc__ assert "Initial contents are ignored." not in module.fill_vector.__doc__ assert "Ownership: Caller-owned" in module.fill_vector.__doc__ @@ -113,41 +117,34 @@ def test_output_arguments_and_multiple_results_follow_python_projection_rules( module.fill_matrix(np.int32(2), np.int32(3), np.empty((2, 3), dtype=np.float64, order="C")) -def test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): +def test_hidden_ordinary_array_output_uses_canonical_plan(tmp_path: Path, monkeypatch): """Replay an existing fixed-shape native output as a hidden result.""" native_object = _compile_native_object(OUTPUTS_F90_SOURCE, tmp_path / "native") - modules = {} contract_text = """\ from x2py.contracts import Addr, Arg, Float64, Int32, Return, native_call @native_call([Addr(Arg(0)), Return("values", 0)]) def fill_vector(n: Int32) -> Float64[n]: ... """ - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_hidden_array_output" - shutil.copytree(CONTRACT_FIXTURES / "foutputs_f90", contract_package) - (contract_package / "foutputs_f90.pyi").write_text(contract_text, encoding="utf-8") - (contract_package / "__init__.pyi").write_text( - "from .foutputs_f90 import fill_vector\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fill_vector") else _sole_native_module(module) - - for module in modules.values(): - np.testing.assert_array_equal(module.fill_vector(np.int32(4)), np.array([2.0, 4.0, 6.0, 8.0])) - assert module.fill_vector(np.int32(0)).shape == (0,) + contract_package = tmp_path / "hidden_array_output" + shutil.copytree(CONTRACT_FIXTURES / "foutputs_f90", contract_package) + (contract_package / "foutputs_f90.pyi").write_text(contract_text, encoding="utf-8") + (contract_package / "__init__.pyi").write_text( + "from .foutputs_f90 import fill_vector\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "fill_vector") else _sole_native_module(package) + + np.testing.assert_array_equal(module.fill_vector(np.int32(4)), np.array([2.0, 4.0, 6.0, 8.0])) + assert module.fill_vector(np.int32(0)).shape == (0,) monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError, match="Unable to allocate copy-return output array"): - modules["wrapper_plan"].fill_vector(np.int32(2)) + module.fill_vector(np.int32(2)) diff --git a/tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py b/tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py index a8ead0af1..b4cf19289 100644 --- a/tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py +++ b/tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py @@ -1,4 +1,4 @@ -"""Compiled legacy/direct-plan parity for Phase 3 scalar writeback.""" +"""Compiled canonical-plan coverage for scalar writeback.""" from __future__ import annotations @@ -15,7 +15,7 @@ from x2py import build_pyi_extension -def test_scalar_copy_in_out_returns_replacement_through_both_routes(tmp_path: Path): +def test_scalar_copy_in_out_returns_replacement(tmp_path: Path): source = tmp_path / "scalar_writeback.f90" source.write_text( """ @@ -42,45 +42,26 @@ def bump( ) native_object = _compile_native_object(source, tmp_path / "native") - results = [] - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - results.append(result) - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) - assert [tuple(path.name for path in result.generated_sources) for result in results] == [ - ( - "bind_c_scalar_writeback_wrapper.f90", - "scalar_writeback_wrapper.c", - "scalar_writeback_wrapper.h", - ), - ( - "bind_c_scalar_writeback_wrapper.f90", - "scalar_writeback_wrapper.c", - "scalar_writeback_wrapper.h", - ), - ] - legacy_bridge = (tmp_path / "legacy" / "bind_c_scalar_writeback_wrapper.f90").read_text(encoding="utf-8") - plan_bridge = (tmp_path / "wrapper_plan" / "bind_c_scalar_writeback_wrapper.f90").read_text(encoding="utf-8") - plan_c = (tmp_path / "wrapper_plan" / "scalar_writeback_wrapper.c").read_text(encoding="utf-8") - assert "function bind_c_bump(value_0001) bind(c) result(value_mutable)" in legacy_bridge - assert legacy_bridge.count("integer(i32) :: value_mutable") == 1 + assert tuple(path.name for path in result.generated_sources) == ( + "bind_c_scalar_writeback_wrapper.f90", + "scalar_writeback_wrapper.c", + "scalar_writeback_wrapper.h", + ) + plan_bridge = (result.output_dir / "bind_c_scalar_writeback_wrapper.f90").read_text(encoding="utf-8") + plan_c = (result.output_dir / "scalar_writeback_wrapper.c").read_text(encoding="utf-8") assert 'subroutine bind_c_bump(value) bind(c, name="bind_c_bump")' in plan_bridge assert "void bind_c_bump(int32_t * value);" in plan_c - for module in modules: - original = np.int32(4) - replacement = module.bump(original) - assert original == np.int32(4) - assert replacement == np.int32(5) - with pytest.raises(TypeError): - module.bump("bad") + original = np.int32(4) + replacement = module.bump(original) + assert original == np.int32(4) + assert replacement == np.int32(5) + with pytest.raises(TypeError): + module.bump("bad") diff --git a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py index e7e95e6e1..1b9c5950d 100644 --- a/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py +++ b/tests/wrapper/fortran/layout_rules/test_wrapper_guide_layout.py @@ -53,7 +53,6 @@ "test_array_results.py", "test_assumed_rank_arrays.py", "test_array_generated_pyi_contracts.py", - "test_bind_c_array_type.py", "test_multidimensional_arrays.py", ), "scalars": ( @@ -406,15 +405,7 @@ def test_wrapper_language_suite_and_user_guide_link_current_subject_paths(): assert "fortran/README.md" in (WRAPPER_SUITE_ROOT / "README.md").read_text(encoding="utf-8") guide = (DOCS_ROOT / "user/guide/fortran-wrapper.md").read_text(encoding="utf-8") - runtime_paths = [ - test_path - for test_path in SUBJECT_TEST_PATHS - if not test_path.startswith("layout_rules/") - and test_path - not in { - "arrays/test_bind_c_array_type.py", - } - ] + runtime_paths = [test_path for test_path in SUBJECT_TEST_PATHS if not test_path.startswith("layout_rules/")] missing = [test_path for test_path in runtime_paths if test_path not in guide] assert missing == [] assert "- [x]" not in guide diff --git a/tests/wrapper/fortran/module_state/test_allocatable_replacement.py b/tests/wrapper/fortran/module_state/test_allocatable_replacement.py index 43d49c05f..974224b4a 100644 --- a/tests/wrapper/fortran/module_state/test_allocatable_replacement.py +++ b/tests/wrapper/fortran/module_state/test_allocatable_replacement.py @@ -94,70 +94,57 @@ def test_allocatable_inout_arrays_mutate_and_return_the_same_handle( module.replace_values(np.array([[1.0]], dtype=np.float64), np.int32(1)) -def test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_projected_allocatable_descriptor_preserves_same_handle_identity(tmp_path: Path): """Replay direct persistent-descriptor mutation with same-handle identity.""" replacement_object = _compile_native_object(ALLOCATABLE_INOUT_F90_SOURCE, tmp_path / "native_replacement") factory_object = _compile_native_object(ALLOCATABLE_FACTORY_F90_SOURCE, tmp_path / "native_factory") - modules = {} - factories = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - replacement_contract = tmp_path / f"{route}_replacement_contract" - shutil.copytree(CONTRACT_FIXTURES / "fallocatable_inout_f90", replacement_contract) - replacement_result = build_pyi_extension( - replacement_contract / "__init__.pyi", - native_objects=[replacement_object], - native_include_dirs=[replacement_object.parent], - output_dir=tmp_path / f"{route}_replacement", - **route_kwargs, - ) - replacement_module = _import_from_build_dir( - replacement_result.module_name, - replacement_result.output_dir, - ) - modules[route] = ( - replacement_module - if hasattr(replacement_module, "replace_values") - else _sole_native_module(replacement_module) - ) - - factory_contract = tmp_path / f"{route}_factory_contract" - factory_contract.mkdir() - (factory_contract / "__init__.pyi").write_text( - "from .fallocatable_views_f90 import build_values\n", - encoding="utf-8", - ) - (factory_contract / "fallocatable_views_f90.pyi").write_text( - """from x2py.contracts import Addr, Allocatable, Arg, Float64, Int32, Return, native_call + replacement_contract = tmp_path / "replacement_contract" + shutil.copytree(CONTRACT_FIXTURES / "fallocatable_inout_f90", replacement_contract) + replacement_result = build_pyi_extension( + replacement_contract / "__init__.pyi", + native_objects=[replacement_object], + native_include_dirs=[replacement_object.parent], + output_dir=tmp_path / "replacement", + ) + replacement_module = _import_from_build_dir( + replacement_result.module_name, + replacement_result.output_dir, + ) + module = ( + replacement_module if hasattr(replacement_module, "replace_values") else _sole_native_module(replacement_module) + ) + + factory_contract = tmp_path / "factory_contract" + factory_contract.mkdir() + (factory_contract / "__init__.pyi").write_text( + "from .fallocatable_views_f90 import build_values\n", + encoding="utf-8", + ) + (factory_contract / "fallocatable_views_f90.pyi").write_text( + """from x2py.contracts import Addr, Allocatable, Arg, Float64, Int32, Return, native_call @native_call([Addr(Arg(0)), Return("values", 0)]) def build_values(n: Int32) -> Allocatable[Float64[:]]: ... """, - encoding="utf-8", - ) - factory_result = build_pyi_extension( - factory_contract / "__init__.pyi", - native_objects=[factory_object], - native_include_dirs=[factory_object.parent], - output_dir=tmp_path / f"{route}_factory", - **route_kwargs, - ) - factory_module = _import_from_build_dir(factory_result.module_name, factory_result.output_dir) - factories[route] = ( - factory_module if hasattr(factory_module, "build_values") else _sole_native_module(factory_module) - ) - - for route, module in modules.items(): - values = factories[route].build_values(np.int32(2)) - assert module.replace_values(values, np.int32(3)) is values - np.testing.assert_allclose(values.to_numpy(), np.array([3.0, 6.0, 9.0])) - assert module.replace_values(values, np.int32(0)) is values - assert values.allocated is False - assert module.replace_values(values, np.int32(1)) is values - np.testing.assert_allclose(values.to_numpy(), np.array([1.0, 2.0])) - values.close() + encoding="utf-8", + ) + factory_result = build_pyi_extension( + factory_contract / "__init__.pyi", + native_objects=[factory_object], + native_include_dirs=[factory_object.parent], + output_dir=tmp_path / "factory", + ) + factory_module = _import_from_build_dir(factory_result.module_name, factory_result.output_dir) + factory = factory_module if hasattr(factory_module, "build_values") else _sole_native_module(factory_module) + + values = factory.build_values(np.int32(2)) + assert module.replace_values(values, np.int32(3)) is values + np.testing.assert_allclose(values.to_numpy(), np.array([3.0, 6.0, 9.0])) + assert module.replace_values(values, np.int32(0)) is values + assert values.allocated is False + assert module.replace_values(values, np.int32(1)) is values + np.testing.assert_allclose(values.to_numpy(), np.array([1.0, 2.0])) + values.close() @pytest.mark.skipif(shutil.which("valgrind") is None, reason="Valgrind is required for native ownership checks") diff --git a/tests/wrapper/fortran/module_state/test_allocatable_views.py b/tests/wrapper/fortran/module_state/test_allocatable_views.py index 3c43b54bd..09ce36ac4 100644 --- a/tests/wrapper/fortran/module_state/test_allocatable_views.py +++ b/tests/wrapper/fortran/module_state/test_allocatable_views.py @@ -283,6 +283,10 @@ def test_allocatable_module_fields_and_results_expose_lifetime_safe_handles( ) assert "Functions" in module.__doc__ + assert "Module Attributes" in module.__doc__ + assert "module_values : AllocatableArray[float64]" in module.__doc__ + assert "Persistent allocatable descriptor handle." in module.__doc__ + assert "Replacement assignment is not supported." in module.__doc__ assert "build_values" in module.__doc__ assert "buffer" in module.__doc__ assert "build_values(n) -> AllocatableArray[float64]" in module.build_values.__doc__ diff --git a/tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py b/tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py index 8b3becc3a..a16917749 100644 --- a/tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py +++ b/tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py @@ -10,7 +10,6 @@ import pytest from tests.wrapper.fortran._support import ( - _build_source_legacy_and_import, _build_source_wrapper_plan_and_import, _sole_native_module, ) @@ -87,9 +86,7 @@ def _reload_native_module(build_dir: Path): sys.path.remove(str(build_dir)) -@pytest.mark.parametrize("route", ("legacy", "wrapper_plan")) -def test_whole_scalar_module_variable_behavior_matches_legacy_route( - route: str, +def test_whole_scalar_module_variable_behavior_uses_canonical_plan( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ): @@ -99,13 +96,9 @@ def test_whole_scalar_module_variable_behavior_matches_legacy_route( "fscalar_module_state_f90_wrapper.c", "fscalar_module_state_f90_wrapper.h", } - if route == "legacy": - build_dir = tmp_path / "legacy_build" - module = _build_source_legacy_and_import(source, build_dir, expected_sources) - else: - module, result = _build_source_wrapper_plan_and_import(source, tmp_path / "plan_build") - build_dir = result.output_dir - assert {path.name for path in result.generated_sources} == expected_sources + module, result = _build_source_wrapper_plan_and_import(source, tmp_path / "build") + build_dir = result.output_dir + assert {path.name for path in result.generated_sources} == expected_sources assert module.nmax == np.int32(12) assert module.counter == np.int32(3) diff --git a/tests/wrapper/fortran/naming/test_generic_interfaces.py b/tests/wrapper/fortran/naming/test_generic_interfaces.py index 1e4c314db..fd31a0e6f 100644 --- a/tests/wrapper/fortran/naming/test_generic_interfaces.py +++ b/tests/wrapper/fortran/naming/test_generic_interfaces.py @@ -31,6 +31,17 @@ def test_fortran_generic_interfaces_dispatch_in_generated_c_extension( pyi_parity_build_mode, ) + assert "Module Attributes" not in module.__doc__ + assert "convert(*args, **kwargs)" in module.__doc__ + assert "_x2py_overload_" not in module.__doc__ + assert "convert_integer" not in module.__doc__ + assert "convert(value: int32) -> int32" in module.convert.__doc__ + assert "convert(value: float64) -> float64" in module.convert.__doc__ + assert "convert(value: complex128) -> complex128" in module.convert.__doc__ + assert "convert_integer" not in module.convert.__doc__ + assert "convert_real" not in module.convert.__doc__ + assert "convert_complex" not in module.convert.__doc__ + assert module.convert(np.int32(4)) == np.int32(14) assert module.convert(np.float64(4.0)) == np.float64(4.5) assert module.convert(np.complex128(2.0 + 3.0j)) == np.complex128(3.0 + 2.0j) diff --git a/tests/wrapper/fortran/naming/test_phase9_class_overloads.py b/tests/wrapper/fortran/naming/test_phase9_class_overloads.py index a8a0474d5..ef06e55a2 100644 --- a/tests/wrapper/fortran/naming/test_phase9_class_overloads.py +++ b/tests/wrapper/fortran/naming/test_phase9_class_overloads.py @@ -20,24 +20,17 @@ @pytest.fixture(scope="module") def class_overloads(tmp_path_factory): - """Build the same reduced edited contract through both wrapper routes.""" + """Build the reduced edited contract through the canonical plan.""" root = tmp_path_factory.mktemp("phase9_class_overloads") native_object = _compile_native_object(SOURCE, root / "native") - modules = {} - for route, kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - CONTRACT, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=root / route, - **kwargs, - ) - package = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = _sole_native_module(package) - return modules + result = build_pyi_extension( + CONTRACT, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=root / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + return _sole_native_module(package) @pytest.fixture(scope="module") @@ -50,28 +43,35 @@ def constructor_overloads(tmp_path_factory): native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=root / "wrapper_plan", - _force_wrapper_plan_route=True, ) package = _import_from_build_dir(result.module_name, result.output_dir) return _sole_native_module(package) -@pytest.mark.parametrize("route", ("legacy", "wrapper_plan")) -def test_exact_method_overloads_match_without_trial_calls(class_overloads, route: str): - module = class_overloads[route] +def test_exact_method_overloads_match_without_trial_calls(class_overloads): + module = class_overloads + assert "add(*args, **kwargs)" in module.accumulator.add.__doc__ + assert "add(value: int32) -> None" in module.accumulator.add.__doc__ + assert "add(value: float64) -> None" in module.accumulator.add.__doc__ + assert "Dispatches to a native operation on the wrapped instance." in module.accumulator.add.__doc__ + assert "accumulator_add_" not in module.accumulator.add.__doc__ + value = module.accumulator() value.add(np.int32(2)) value.add(np.float64(0.5)) assert value.total == np.float64(2.5) - with pytest.raises(TypeError) as exc_info: + with pytest.raises(TypeError, match="no matching overload for add"): value.add(np.complex128(1.0 + 0.0j)) - if route == "wrapper_plan": - assert "no matching overload for add" in str(exc_info.value) def test_constructor_overloads_share_owned_allocation_and_exact_matching(constructor_overloads): module = constructor_overloads + assert "accumulator(*args, **kwargs) -> accumulator" in module.accumulator.__init__.__doc__ + assert "accumulator(value: int32) -> accumulator" in module.accumulator.__init__.__doc__ + assert "accumulator(value: float64) -> accumulator" in module.accumulator.__init__.__doc__ + assert "accumulator_add_" not in module.accumulator.__init__.__doc__ + integer = module.accumulator(np.int32(3)) real = module.accumulator(np.float64(1.25)) assert integer.total == np.float64(3.0) diff --git a/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py b/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py index 3c4b8fd8e..64de7e211 100644 --- a/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py +++ b/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py @@ -404,7 +404,6 @@ def test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_li assert "use full_blas_interfaces" not in bridge assert "subroutine daxpy(" in bridge assert "private\n" not in bridge - assert "private :: c_malloc" in bridge assert "public :: bind_c_daxpy" not in bridge _assert_blas_runtime_smoke(module) else: diff --git a/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py b/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py index 9727b5c46..86536ce73 100644 --- a/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py +++ b/tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py @@ -22,25 +22,16 @@ MODIFIED_POLICY_CONTRACT = ( Path(__file__).parent / "modified_contracts" / "fruntime_policy_f90" / "fruntime_policy_f90.pyi" ) -ROUTES = ( - pytest.param("legacy", {"_force_legacy_wrapper_route": True}, id="legacy"), - pytest.param("wrapper-plan", {"_force_wrapper_plan_route": True}, id="wrapper-plan"), -) -def _wrapper_start(source: str, route: str, function_name: str) -> int: - marker = ( - f"static PyObject* bind_c_{function_name}_wrapper" - if route == "legacy" - else f"static PyObject * wrap_{function_name}" - ) - return source.index(marker) +def _wrapper_start(source: str, function_name: str) -> int: + return source.index(f"static PyObject * wrap_{function_name}") -def _assert_runtime_policy_source(source: str, route: str) -> None: - released_start = _wrapper_start(source, route, "pause_for_one_second") - held_start = _wrapper_start(source, route, "pause_with_gil") - solve_start = _wrapper_start(source, route, "solve") +def _assert_runtime_policy_source(source: str) -> None: + released_start = _wrapper_start(source, "pause_for_one_second") + held_start = _wrapper_start(source, "pause_with_gil") + solve_start = _wrapper_start(source, "solve") released_wrapper = source[released_start:held_start] held_wrapper = source[held_start:solve_start] assert "Py_BEGIN_ALLOW_THREADS" in released_wrapper @@ -50,12 +41,9 @@ def _assert_runtime_policy_source(source: str, route: str) -> None: assert "PyErr_SetObject(PyExc_RuntimeError" in source -@pytest.mark.parametrize(("route", "route_kwargs"), ROUTES) def test_compiled_runtime_policies_release_gil_and_project_native_errors( tmp_path: Path, monkeypatch, - route: str, - route_kwargs: dict[str, bool], ): from x2py.pipeline import build from x2py.semantics.models import RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA @@ -77,7 +65,7 @@ def convert_with_runtime_policy(*args, **kwargs): return modules monkeypatch.setattr(build, "fortran_project_to_semantic_modules", convert_with_runtime_policy) - result = build.build_fortran_extension(source, output_dir=tmp_path, **route_kwargs) + result = build.build_fortran_extension(source, output_dir=tmp_path) sys.modules.pop(result.module_name, None) sys.path.insert(0, str(tmp_path)) @@ -112,14 +100,11 @@ def native_pause(): sys.path.remove(str(tmp_path)) wrapper_source = (tmp_path / "fruntime_policy_f90_wrapper.c").read_text(encoding="utf-8") - _assert_runtime_policy_source(wrapper_source, route) + _assert_runtime_policy_source(wrapper_source) -@pytest.mark.parametrize(("route", "route_kwargs"), ROUTES) def test_pyi_runtime_policies_release_gil_and_project_native_errors( tmp_path: Path, - route: str, - route_kwargs: dict[str, bool], ): native_object = _compile_native_object(RUNTIME_POLICY_SOURCE, tmp_path / "native") result = build_pyi_extension( @@ -127,7 +112,6 @@ def test_pyi_runtime_policies_release_gil_and_project_native_errors( native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "pyi_build", - **route_kwargs, ) module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) @@ -158,4 +142,4 @@ def native_pause(): assert failures == [] wrapper_source = (result.output_dir / "fruntime_policy_f90_wrapper.c").read_text(encoding="utf-8") - _assert_runtime_policy_source(wrapper_source, route) + _assert_runtime_policy_source(wrapper_source) diff --git a/tests/wrapper/fortran/scalars/test_fortran_enums.py b/tests/wrapper/fortran/scalars/test_fortran_enums.py index 9a6694ece..7f7735d5b 100644 --- a/tests/wrapper/fortran/scalars/test_fortran_enums.py +++ b/tests/wrapper/fortran/scalars/test_fortran_enums.py @@ -5,7 +5,7 @@ import numpy as np from x2py import parse_fortran_file as parse_fortran_source -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from x2py.semantics.fortran2ir import fortran_module_to_semantic_module from tests.wrapper.fortran._support import _build_source_or_generated_pyi_and_import, wrapper_source diff --git a/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py index c21804671..a0c663f90 100644 --- a/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py +++ b/tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py @@ -15,7 +15,7 @@ from x2py import build_pyi_extension -def _build_contract_routes( +def _build_contract_module( tmp_path: Path, *, module_name: str, @@ -28,30 +28,23 @@ def _build_contract_routes( contract.write_text(contract_text, encoding="utf-8") native_object = _compile_native_object(source, tmp_path / "native") - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {}), - ): - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - generated_c = (result.output_dir / f"{module_name}_wrapper.c").read_text(encoding="utf-8") - if route == "wrapper_plan": - assert "static PyObject * wrap_" in generated_c - modules.append(_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir))) - return tuple(modules) + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + generated_c = (result.output_dir / f"{module_name}_wrapper.c").read_text(encoding="utf-8") + assert "static PyObject * wrap_" in generated_c + return _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) def _build_scalar_boundary_modules(tmp_path: Path): - return _build_contract_routes( - tmp_path, - module_name="scalar_boundary_plan", - source_text=""" + return ( + _build_contract_module( + tmp_path, + module_name="scalar_boundary_plan", + source_text=""" module scalar_boundary_plan use iso_c_binding, only: c_double, c_int32_t contains @@ -108,7 +101,7 @@ def _build_scalar_boundary_modules(tmp_path: Path): end subroutine mapped_status end module scalar_boundary_plan """, - contract_text=""" + contract_text=""" from x2py.contracts import Addr, Annotated, Arg, Float64, Immutable, Int32, Return, Returns, native_call def value_input(value: Int32) -> Int32: ... @@ -135,14 +128,16 @@ def make_raw(value: Addr(Int32)) -> None: ... @native_call([Return("status", 0), Addr(Arg(0))]) def mapped_status(base: Int32) -> Int32: ... """, + ), ) def _build_scalar_kind_modules(tmp_path: Path): - return _build_contract_routes( - tmp_path, - module_name="scalar_kind_plan", - source_text=""" + return ( + _build_contract_module( + tmp_path, + module_name="scalar_kind_plan", + source_text=""" module scalar_kind_plan use iso_c_binding, only: c_bool, c_double, c_double_complex, c_float, & c_float_complex, c_int8_t, c_int16_t, c_int32_t, c_int64_t @@ -202,7 +197,7 @@ def _build_scalar_kind_modules(tmp_path: Path): end function conj_c128 end module scalar_kind_plan """, - contract_text=""" + contract_text=""" from x2py.contracts import Addr, Arg, Bool, Complex128, Complex64, Float32, Float64, Int16, Int32, Int64, Int8, native_call @native_call([Addr(Arg(0))]) @@ -232,14 +227,16 @@ def conj_c64(value: Complex64) -> Complex64: ... @native_call([Addr(Arg(0))]) def conj_c128(value: Complex128) -> Complex128: ... """, + ), ) def _build_multiple_scalar_result_modules(tmp_path: Path): - return _build_contract_routes( - tmp_path, - module_name="multiple_scalar_results_plan", - source_text=""" + return ( + _build_contract_module( + tmp_path, + module_name="multiple_scalar_results_plan", + source_text=""" module multiple_scalar_results_plan use iso_c_binding, only: c_int32_t contains @@ -252,12 +249,13 @@ def _build_multiple_scalar_result_modules(tmp_path: Path): end function with_scalar end module multiple_scalar_results_plan """, - contract_text=""" + contract_text=""" from x2py.contracts import Addr, Arg, Int32, Return, native_call @native_call([Addr(Arg(0)), Return("status", 1)]) def with_scalar(n: Int32) -> tuple[Int32, Int32]: ... """, + ), ) @@ -311,12 +309,11 @@ def maybe_pointer(flag: Int32) -> Float64 | None: ... native_objects=[native_object], native_include_dirs=[native_object.parent], output_dir=tmp_path / "wrapper_plan_scalar_descriptors", - _force_wrapper_plan_route=True, ) return (_sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)),) -def test_scalar_value_storage_raw_address_out_and_inout_match_both_routes(tmp_path: Path): +def test_scalar_value_storage_raw_address_out_and_inout_use_canonical_plan(tmp_path: Path): modules = _build_scalar_boundary_modules(tmp_path) for module in modules: @@ -367,7 +364,7 @@ def test_scalar_value_storage_raw_address_out_and_inout_match_both_routes(tmp_pa module.bump_raw(raw) -def test_multiple_scalar_results_match_both_routes_without_array_blockers(tmp_path: Path): +def test_multiple_scalar_results_use_canonical_plan_without_array_blockers(tmp_path: Path): modules = _build_multiple_scalar_result_modules(tmp_path) for module in modules: @@ -384,7 +381,7 @@ def test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_rout assert module.maybe_pointer(np.int32(0)) is None -def test_scalar_primitive_kinds_match_both_routes_without_array_blockers(tmp_path: Path): +def test_scalar_primitive_kinds_use_canonical_plan_without_array_blockers(tmp_path: Path): modules = _build_scalar_kind_modules(tmp_path) for module in modules: diff --git a/tests/wrapper/fortran/scalars/test_verified_baseline.py b/tests/wrapper/fortran/scalars/test_verified_baseline.py index 1a2081169..0aa15e05b 100644 --- a/tests/wrapper/fortran/scalars/test_verified_baseline.py +++ b/tests/wrapper/fortran/scalars/test_verified_baseline.py @@ -10,7 +10,6 @@ _assert_array_rejects_strided_views, _assert_fmath_array_examples, _assert_fmath_examples, - _build_source_legacy_and_import, _build_source_or_generated_pyi_and_import, _build_source_wrapper_plan_and_import, _compile_native_object, @@ -70,7 +69,7 @@ def test_fortran_wrapper_pipeline_builds_importable_extension( [SCALAR_FIXED_SOURCE, SCALAR_F90_SOURCE], ids=["fixed-form-externals", "free-form-module"], ) -def test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes( +def test_fmath_scalar_sources_use_canonical_wrapper_plan( tmp_path: Path, source: Path, ): @@ -79,40 +78,33 @@ def test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes( f"{source.stem}_wrapper.c", f"{source.stem}_wrapper.h", } - legacy_module = _build_source_legacy_and_import( + wrapper_root, wrapper_result = _build_source_wrapper_plan_and_import( source, - tmp_path / "legacy", - expected_generated_sources, - ) - wrapper_plan_root, wrapper_plan_result = _build_source_wrapper_plan_and_import( - source, - tmp_path / "wrapper_plan", + tmp_path / "build", unwrap_namespace=False, ) if source == SCALAR_F90_SOURCE: - assert not hasattr(wrapper_plan_root, "add_r8") - assert hasattr(wrapper_plan_root, "fmath_f90") - wrapper_plan_module = wrapper_plan_root.fmath_f90 + assert not hasattr(wrapper_root, "add_r8") + assert hasattr(wrapper_root, "fmath_f90") + module = wrapper_root.fmath_f90 else: - assert hasattr(wrapper_plan_root, "add_r8") - wrapper_plan_module = wrapper_plan_root + assert hasattr(wrapper_root, "add_r8") + module = wrapper_root - assert {path.name for path in wrapper_plan_result.generated_sources} == expected_generated_sources - assert any(path.name == f"{source.stem}_wrapper.h" for path in wrapper_plan_result.generated_files) + assert {path.name for path in wrapper_result.generated_sources} == expected_generated_sources + assert any(path.name == f"{source.stem}_wrapper.h" for path in wrapper_result.generated_files) assert any( path.name == "python_runtime.c" and path.parent.name == "x2py_runtime" - for path in wrapper_plan_result.generated_files + for path in wrapper_result.generated_files ) - assert wrapper_plan_result.compiled is True - assert wrapper_plan_result.shared_library.exists() - - _assert_fmath_examples(legacy_module) - _assert_fmath_examples(wrapper_plan_module) + assert wrapper_result.compiled is True + assert wrapper_result.shared_library.exists() - legacy_failure = _scalar_conversion_failure(legacy_module) - wrapper_plan_failure = _scalar_conversion_failure(wrapper_plan_module) - assert wrapper_plan_failure == legacy_failure + _assert_fmath_examples(module) + error_type, message = _scalar_conversion_failure(module) + assert error_type is TypeError + assert "argument" in message def _scalar_conversion_failure(module) -> tuple[type[BaseException], str]: @@ -183,51 +175,43 @@ def test_f90_array_wrapper_distinguishes_contiguous_and_strided_contracts( _assert_fmath_array_examples(module, suffix="_STRIDED", strided=True) -def test_required_array_buffers_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_required_array_buffers_use_canonical_wrapper_plan(tmp_path: Path): """Replay one existing dense rank-one routine through a reduced contract.""" native_object = _compile_native_object(ARRAY_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_required_array" - shutil.copytree(CONTRACT_FIXTURES / "fmath_arrays_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fmath_arrays_f90 import square_r8_contiguous\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "square_r8_contiguous") else _sole_native_module(module) - - for module in modules.values(): - values = np.array([2.0, 3.0, -4.0], dtype=np.float64) - output = np.zeros_like(values) - assert module.square_r8_contiguous(np.int32(values.size), values, output) is None - np.testing.assert_array_equal(output, values**2) - - handle_output = np.zeros_like(values) - assert ( - module.square_r8_contiguous( - np.int32(values.size), - _native_array_actual(values, pointer=False), - _native_array_actual(handle_output, pointer=True), - ) - is None + contract_package = tmp_path / "required_array" + shutil.copytree(CONTRACT_FIXTURES / "fmath_arrays_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fmath_arrays_f90 import square_r8_contiguous\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "square_r8_contiguous") else _sole_native_module(package) + + values = np.array([2.0, 3.0, -4.0], dtype=np.float64) + output = np.zeros_like(values) + assert module.square_r8_contiguous(np.int32(values.size), values, output) is None + np.testing.assert_array_equal(output, values**2) + + handle_output = np.zeros_like(values) + assert ( + module.square_r8_contiguous( + np.int32(values.size), + _native_array_actual(values, pointer=False), + _native_array_actual(handle_output, pointer=True), ) - np.testing.assert_array_equal(handle_output, values**2) + is None + ) + np.testing.assert_array_equal(handle_output, values**2) - empty = np.empty(0, dtype=np.float64) - assert module.square_r8_contiguous(np.int32(0), empty, empty.copy()) is None + empty = np.empty(0, dtype=np.float64) + assert module.square_r8_contiguous(np.int32(0), empty, empty.copy()) is None - module = modules["wrapper_plan"] valid = np.arange(4, dtype=np.float64) output = np.zeros_like(valid) invalid_cases = ( diff --git a/tests/wrapper/fortran/strings/test_character_arguments.py b/tests/wrapper/fortran/strings/test_character_arguments.py index 7fc39fea4..d26ef5616 100644 --- a/tests/wrapper/fortran/strings/test_character_arguments.py +++ b/tests/wrapper/fortran/strings/test_character_arguments.py @@ -1,4 +1,4 @@ -"""Legacy and modern scalar character argument/result tests.""" +"""Fixed-form and modern scalar character argument/result tests.""" from pathlib import Path import shutil @@ -22,6 +22,18 @@ CONTRACT_FIXTURES = Path(__file__).parent / "contracts" +def _build_contract_module(contract: Path, native_object: Path, output_dir: Path, symbol: str): + """Build one edited character contract through the canonical wrapper plan.""" + result = build_pyi_extension( + contract, + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=output_dir, + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + return package if hasattr(package, symbol) else _sole_native_module(package) + + def test_legacy_fortran_character_arguments_and_results(pyi_parity_build_mode: str, tmp_path: Path): module = _build_source_or_generated_pyi_and_import( STRING_LEGACY_SOURCE, @@ -74,172 +86,123 @@ def test_edited_modern_string_contract_wraps_full_axis_spelling_set(tmp_path: Pa assert label[()] == b"Ybcdefg?" -def test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_fixed_width_character_arrays_use_canonical_plan(tmp_path: Path): """Replay one ordinary fixed-width NumPy bytes array without descriptors.""" native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_character_array" - shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90_axes", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fstrings_f90 import fixed_array_extent\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fixed_array_extent") else _sole_native_module(module) - - for module in modules.values(): - labels = np.array([b"first", b"second"], dtype="S8") - assert module.fixed_array_extent(labels) == 16 - assert module.fixed_array_extent(np.empty(0, dtype="S8")) == 0 - - direct = modules["wrapper_plan"] + contract_package = tmp_path / "character_array" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90_axes", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fstrings_f90 import fixed_array_extent\n", + encoding="utf-8", + ) + module = _build_contract_module( + contract_package / "__init__.pyi", native_object, tmp_path / "build", "fixed_array_extent" + ) + + labels = np.array([b"first", b"second"], dtype="S8") + assert module.fixed_array_extent(labels) == 16 + assert module.fixed_array_extent(np.empty(0, dtype="S8")) == 0 + with pytest.raises(TypeError): - direct.fixed_array_extent(np.array([b"short"], dtype="S7")) + module.fixed_array_extent(np.array([b"short"], dtype="S7")) with pytest.raises(TypeError): - direct.fixed_array_extent(np.array([[b"label"]], dtype="S8")) + module.fixed_array_extent(np.array([[b"label"]], dtype="S8")) -def test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_raw_fixed_width_character_arrays_use_canonical_plan(tmp_path: Path): """Replay one fixed-width character array through its raw address contract.""" native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_raw_character_array" - contract_package.mkdir() - (contract_package / "__init__.pyi").write_text( - "from .fstrings_f90 import fixed_array_extent_raw\n", - encoding="utf-8", - ) - (contract_package / "fstrings_f90.pyi").write_text( - """from x2py.contracts import Addr, Int32, String, bind + contract_package = tmp_path / "raw_character_array" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + "from .fstrings_f90 import fixed_array_extent_raw\n", + encoding="utf-8", + ) + (contract_package / "fstrings_f90.pyi").write_text( + """from x2py.contracts import Addr, Int32, String, bind @bind("fixed_array_extent") def fixed_array_extent_raw(labels: Addr(String[8][2])) -> Int32: ... """, - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fixed_array_extent_raw") else _sole_native_module(module) - - for module in modules.values(): - labels = np.array([b"first", b"second"], dtype="S8") - assert module.fixed_array_extent_raw(labels.ctypes.data) == 16 - with pytest.raises(TypeError): - module.fixed_array_extent_raw(labels) - - -def test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes(tmp_path: Path): + encoding="utf-8", + ) + module = _build_contract_module( + contract_package / "__init__.pyi", native_object, tmp_path / "build", "fixed_array_extent_raw" + ) + + labels = np.array([b"first", b"second"], dtype="S8") + assert module.fixed_array_extent_raw(labels.ctypes.data) == 16 + with pytest.raises(TypeError): + module.fixed_array_extent_raw(labels) + + +def test_required_scalar_string_inputs_use_canonical_plan(tmp_path: Path): """Reuse the existing modern string unit through one scalar-input-only entry.""" native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") - modules = [] - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_string_inputs" - shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "\n".join( - ( - "from .fstrings_f90 import char_code_default", - "from .fstrings_f90 import char_code_len1", - "from .fstrings_f90 import char_code_kind1", - "from .fstrings_f90 import char_code_c_char", - "from .fstrings_f90 import string_len_fixed", - "from .fstrings_f90 import string_len_assumed", - "from .fstrings_f90 import string_len_c_char", - "", - ) - ), - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules.append(module if hasattr(module, "char_code_default") else _sole_native_module(module)) - - for module in modules: - assert module.char_code_default("A") == ord("A") - assert module.char_code_len1(np.str_("B")) == ord("B") - assert module.char_code_kind1("C") == ord("C") - assert module.char_code_c_char("D") == ord("D") - assert module.string_len_fixed("short ") == 5 - assert module.string_len_assumed("variable length") == 15 - assert module.string_len_assumed("") == 0 - assert module.string_len_assumed("café") == 5 - assert module.string_len_c_char("c-char ") == 6 - - with pytest.raises(TypeError, match="str"): - module.string_len_assumed(b"bytes") - with pytest.raises(TypeError, match="exactly 8 bytes"): - module.string_len_fixed("short") - with pytest.raises(TypeError, match="embedded NUL"): - module.string_len_assumed("a\0b") - - -def test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes( + contract_package = tmp_path / "string_inputs" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "\n".join( + ( + "from .fstrings_f90 import char_code_default", + "from .fstrings_f90 import char_code_len1", + "from .fstrings_f90 import char_code_kind1", + "from .fstrings_f90 import char_code_c_char", + "from .fstrings_f90 import string_len_fixed", + "from .fstrings_f90 import string_len_assumed", + "from .fstrings_f90 import string_len_c_char", + "", + ) + ), + encoding="utf-8", + ) + module = _build_contract_module( + contract_package / "__init__.pyi", native_object, tmp_path / "build", "char_code_default" + ) + + assert module.char_code_default("A") == ord("A") + assert module.char_code_len1(np.str_("B")) == ord("B") + assert module.char_code_kind1("C") == ord("C") + assert module.char_code_c_char("D") == ord("D") + assert module.string_len_fixed("short ") == 5 + assert module.string_len_assumed("variable length") == 15 + assert module.string_len_assumed("") == 0 + assert module.string_len_assumed("café") == 5 + assert module.string_len_c_char("c-char ") == 6 + + with pytest.raises(TypeError, match="str"): + module.string_len_assumed(b"bytes") + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.string_len_fixed("short") + with pytest.raises(TypeError, match="embedded NUL"): + module.string_len_assumed("a\0b") + + +def test_deferred_allocatable_string_results_use_canonical_plan( tmp_path: Path, monkeypatch, ): """Replay a nullable rank-zero descriptor result as a copied Python string.""" native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_deferred_string_result" - shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fstrings_f90 import string_result_deferred\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "string_result_deferred") else _sole_native_module(module) - - for module in modules.values(): - assert module.string_result_deferred("dynamic") == "dynamic-deferred" - assert module.string_result_deferred("café") == "café-deferred" + contract_package = tmp_path / "deferred_string_result" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fstrings_f90 import string_result_deferred\n", + encoding="utf-8", + ) + module = _build_contract_module( + contract_package / "__init__.pyi", native_object, tmp_path / "build", "string_result_deferred" + ) + + assert module.string_result_deferred("dynamic") == "dynamic-deferred" + assert module.string_result_deferred("café") == "café-deferred" monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError): - modules["wrapper_plan"].string_result_deferred("failure") + module.string_result_deferred("failure") -def test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes(tmp_path: Path): +def test_deferred_character_array_handles_use_canonical_plan(tmp_path: Path): """Keep runtime element width and projected identity on one shared handle path.""" module_name = "deferred_character_handles_plan" source = tmp_path / f"{module_name}.f90" @@ -304,75 +267,52 @@ def replace_names( encoding="utf-8", ) native_object = _compile_native_object(source, tmp_path / "native_character_handles") - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - result = build_pyi_extension( - contract, - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _sole_native_module(_import_from_build_dir(result.module_name, result.output_dir)) - handle = module.make_names() - assert handle.allocated is True - assert handle.dtype == np.dtype("S3") - assert handle.to_numpy().tolist() == [b"red", b"sky"] - - assert module.replace_names(handle) is handle - assert handle.dtype == np.dtype("S5") - assert handle.to_numpy().tolist() == [b"red ", b"blue "] - - direct_handle = module.make_names_function() - assert direct_handle.allocated is True - assert direct_handle.dtype == np.dtype("S4") - assert direct_handle.to_numpy().tolist() == [b"gold", b"blue"] - assert module.maybe_name(np.int32(0)) is None - assert module.maybe_name(np.int32(1)) == "blue" - - -def test_fixed_string_results_match_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): + module = _build_contract_module(contract, native_object, tmp_path / "build", "make_names") + handle = module.make_names() + assert handle.allocated is True + assert handle.dtype == np.dtype("S3") + assert handle.to_numpy().tolist() == [b"red", b"sky"] + + assert module.replace_names(handle) is handle + assert handle.dtype == np.dtype("S5") + assert handle.to_numpy().tolist() == [b"red ", b"blue "] + + direct_handle = module.make_names_function() + assert direct_handle.allocated is True + assert direct_handle.dtype == np.dtype("S4") + assert direct_handle.to_numpy().tolist() == [b"gold", b"blue"] + assert module.maybe_name(np.int32(0)) is None + assert module.maybe_name(np.int32(1)) == "blue" + + +def test_fixed_string_results_use_canonical_plan(tmp_path: Path, monkeypatch): """Replay existing fixed direct results through a result-only contract entry.""" native_object = _compile_native_object(STRING_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_string_results" - shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "\n".join( - ( - "from .fstrings_f90 import char_result_default", - "from .fstrings_f90 import char_result_c_char", - "from .fstrings_f90 import string_result_fixed", - "from .fstrings_f90 import string_result_padded", - "from .fstrings_f90 import string_result_c_char", - "", - ) - ), - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "char_result_default") else _sole_native_module(module) - - for module in modules.values(): - assert module.char_result_default() == "M" - assert module.char_result_c_char() == "C" - assert module.string_result_fixed() == "MODERN!!" - assert module.string_result_padded() == "PAD " - assert module.string_result_c_char() == "C-CHAR!!" + contract_package = tmp_path / "string_results" + shutil.copytree(CONTRACT_FIXTURES / "fstrings_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "\n".join( + ( + "from .fstrings_f90 import char_result_default", + "from .fstrings_f90 import char_result_c_char", + "from .fstrings_f90 import string_result_fixed", + "from .fstrings_f90 import string_result_padded", + "from .fstrings_f90 import string_result_c_char", + "", + ) + ), + encoding="utf-8", + ) + module = _build_contract_module( + contract_package / "__init__.pyi", native_object, tmp_path / "build", "char_result_default" + ) + + assert module.char_result_default() == "M" + assert module.char_result_c_char() == "C" + assert module.string_result_fixed() == "MODERN!!" + assert module.string_result_padded() == "PAD " + assert module.string_result_c_char() == "C-CHAR!!" monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError, match="Unable to allocate copy-return output string"): - modules["wrapper_plan"].string_result_fixed() + module.string_result_fixed() diff --git a/tests/wrapper/fortran/strings/test_character_edge_cases.py b/tests/wrapper/fortran/strings/test_character_edge_cases.py index 9151f305c..21ca85e79 100644 --- a/tests/wrapper/fortran/strings/test_character_edge_cases.py +++ b/tests/wrapper/fortran/strings/test_character_edge_cases.py @@ -56,57 +56,45 @@ def test_fortran_character_edge_cases_follow_copy_in_copy_out_policy( module.unicode_echo("a\0b") -def test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes(tmp_path: Path, monkeypatch): +def test_fixed_hidden_string_output_uses_canonical_plan(tmp_path: Path, monkeypatch): """Replay the existing hidden output through a reduced contract entry.""" native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_hidden_string_result" - shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fcharacter_edges_f90 import make_out\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "make_out") else _sole_native_module(module) - - assert modules["legacy"].make_out() == "go " - assert modules["wrapper_plan"].make_out() == "go " + contract_package = tmp_path / "hidden_string_result" + shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import make_out\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "make_out") else _sole_native_module(package) + + assert module.make_out() == "go " monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError, match="Unable to allocate copy-return output string"): - modules["wrapper_plan"].make_out() + module.make_out() -def test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes( +def test_fixed_string_replacement_and_identity_use_canonical_plan( tmp_path: Path, monkeypatch, ): """Replay projected and discarded mutation against one existing native routine.""" native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_fixed_string_writeback" - contract_package.mkdir() - (contract_package / "__init__.pyi").write_text( - "from .fcharacter_edges_f90 import fixed_discard, fixed_replacement\n", - encoding="utf-8", - ) - (contract_package / "fcharacter_edges_f90.pyi").write_text( - """from x2py.contracts import Returns, String, bind + contract_package = tmp_path / "fixed_string_writeback" + contract_package.mkdir() + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import fixed_discard, fixed_replacement\n", + encoding="utf-8", + ) + (contract_package / "fcharacter_edges_f90.pyi").write_text( + """from x2py.contracts import Returns, String, bind @bind("fixed_inout") def fixed_replacement(name: String[8]) -> Returns["name", String[8]]: ... @@ -114,80 +102,71 @@ def fixed_replacement(name: String[8]) -> Returns["name", String[8]]: ... @bind("fixed_inout") def fixed_discard(name: String[8]) -> None: ... """, - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "fixed_replacement") else _sole_native_module(module) - - for module in modules.values(): - original = "abc " - assert module.fixed_replacement(original) == "Zbc !" - assert original == "abc " - assert module.fixed_discard(original) is None - assert original == "abc " - with pytest.raises(TypeError, match="exactly 8 bytes"): - module.fixed_replacement("abc") - with pytest.raises(TypeError, match="exactly 8 bytes"): - module.fixed_discard("abcdefghi") + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "fixed_replacement") else _sole_native_module(package) + + original = "abc " + assert module.fixed_replacement(original) == "Zbc !" + assert original == "abc " + assert module.fixed_discard(original) is None + assert original == "abc " + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.fixed_replacement("abc") + with pytest.raises(TypeError, match="exactly 8 bytes"): + module.fixed_discard("abcdefghi") monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument name"): - modules["wrapper_plan"].fixed_replacement("abc ") + module.fixed_replacement("abc ") -def test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes( +def test_assumed_and_optional_string_replacements_use_canonical_plan( tmp_path: Path, monkeypatch, ): """Replay runtime-length and absent/concrete presence through a reduced entry.""" native_object = _compile_native_object(CHARACTER_EDGES_F90_SOURCE, tmp_path / "native") - modules = {} - for route, route_kwargs in ( - ("legacy", {"_force_legacy_wrapper_route": True}), - ("wrapper_plan", {"_force_wrapper_plan_route": True}), - ): - contract_package = tmp_path / f"{route}_assumed_optional_string_writeback" - shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) - (contract_package / "__init__.pyi").write_text( - "from .fcharacter_edges_f90 import assumed_inout, optional_inout\n", - encoding="utf-8", - ) - result = build_pyi_extension( - contract_package / "__init__.pyi", - native_objects=[native_object], - native_include_dirs=[native_object.parent], - output_dir=tmp_path / route, - **route_kwargs, - ) - module = _import_from_build_dir(result.module_name, result.output_dir) - modules[route] = module if hasattr(module, "assumed_inout") else _sole_native_module(module) - - for module in modules.values(): - assumed_original = "abc" - optional_original = "abc" - assert module.assumed_inout(assumed_original) == "Qbc" - assert module.assumed_inout("") == "" - assert module.optional_inout() is None - assert module.optional_inout(None) is None - assert module.optional_inout(optional_original) == "Pbc" - assert assumed_original == "abc" - assert optional_original == "abc" - with pytest.raises(TypeError, match="embedded NUL"): - module.assumed_inout("a\0b") - with pytest.raises(TypeError, match="embedded NUL"): - module.optional_inout("a\0b") + contract_package = tmp_path / "assumed_optional_string_writeback" + shutil.copytree(CONTRACT_FIXTURES / "fcharacter_edges_f90", contract_package) + (contract_package / "__init__.pyi").write_text( + "from .fcharacter_edges_f90 import assumed_inout, optional_inout\n", + encoding="utf-8", + ) + result = build_pyi_extension( + contract_package / "__init__.pyi", + native_objects=[native_object], + native_include_dirs=[native_object.parent], + output_dir=tmp_path / "build", + ) + package = _import_from_build_dir(result.module_name, result.output_dir) + module = package if hasattr(package, "assumed_inout") else _sole_native_module(package) + + assumed_original = "abc" + optional_original = "abc" + assert module.assumed_inout(assumed_original) == "Qbc" + assert module.assumed_inout("") == "" + assert module.optional_inout() is None + assert module.optional_inout(None) is None + assert module.optional_inout(optional_original) == "Pbc" + assert assumed_original == "abc" + assert optional_original == "abc" + with pytest.raises(TypeError, match="embedded NUL"): + module.assumed_inout("a\0b") + with pytest.raises(TypeError, match="embedded NUL"): + module.optional_inout("a\0b") monkeypatch.setenv("X2PY_WRAPPER_FAIL_ALLOC", "1") - assert modules["wrapper_plan"].optional_inout() is None - assert modules["wrapper_plan"].optional_inout(None) is None + assert module.optional_inout() is None + assert module.optional_inout(None) is None with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument name"): - modules["wrapper_plan"].assumed_inout("abc") + module.assumed_inout("abc") with pytest.raises(MemoryError, match="Unable to allocate mutable string buffer for argument label"): - modules["wrapper_plan"].optional_inout("abc") + module.optional_inout("abc") diff --git a/tests/codegen/printers/_support.py b/tests/wrapper_codegen/printers/_support.py similarity index 78% rename from tests/codegen/printers/_support.py rename to tests/wrapper_codegen/printers/_support.py index 27243e8f0..60d17a249 100644 --- a/tests/codegen/printers/_support.py +++ b/tests/wrapper_codegen/printers/_support.py @@ -9,26 +9,19 @@ from x2py import parse_fortran_file as parse_fortran_source -from x2py.codegen.binding_pipeline import BindingPipeline - -from x2py.codegen.codegen import Codegen - -from x2py.codegen.scope import Scope - from x2py.semantics.fortran2ir import ( fortran_module_to_semantic_module, ) from x2py.pipeline.pyi import pyi_text_to_semantic_module as _parse_pyi_text -from x2py.semantics.ir2ast import semantic_ir_to_codegen_ast as _semantic_ir_to_codegen_ast - -from x2py.codegen.printers.pyi_printer import ( +from x2py.wrapper_codegen.printers import ( emit_module, emit_module_stubs, opaque_dependency_modules, PyiPrinter, ) +from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner from x2py.semantics.models import ( CALLBACK_DECLARATION_ACCESS_METADATA, @@ -65,12 +58,6 @@ def parse_pyi_text(source: str, *args, **kwargs): return _parse_pyi_text(f"{CONTRACT_IMPORT}{source}", *args, **kwargs) -def semantic_ir_to_codegen_ast(node, *args, **kwargs): - if isinstance(node, SemanticModule): - complete_semantic_policies(node) - return _semantic_ir_to_codegen_ast(node, *args, **kwargs) - - def generate_pyi(source: str) -> str: fmod = parse_fortran_source(source) @@ -79,6 +66,19 @@ def generate_pyi(source: str) -> str: return emit_module(smod) +def generate_wrapper_artifacts(module: SemanticModule): + """Generate wrapper sources through the canonical plan implementation.""" + complete_semantic_policies(module) + return WrapperCodeGenerator().generate(WrapperPlanner().build(module)) + + +def rendered_source(artifacts, suffix: str) -> str: + """Return the sole rendered source with ``suffix``.""" + matches = [source.text for source in artifacts.sources if source.path.suffix == suffix] + assert len(matches) == 1 + return matches[0] + + def normalize(text: str) -> str: return "\n".join(line.rstrip() for line in text.strip().splitlines()) @@ -88,12 +88,9 @@ def normalize(text: str) -> str: "OPERATOR_F90_SOURCE", "RUNTIME_HOLD_GIL_METADATA", "RUNTIME_STATUS_ERROR_METADATA", - "BindingPipeline", - "Codegen", "Path", "ProjectionMapping", "PyiPrinter", - "Scope", "SemanticArgument", "SemanticArrayContract", "SemanticClass", @@ -113,11 +110,12 @@ def normalize(text: str) -> str: "emit_module_stubs", "fortran_module_to_semantic_module", "generate_pyi", + "generate_wrapper_artifacts", "normalize", "opaque_dependency_modules", "parse_fortran_source", "parse_pyi_text", "pytest", - "semantic_ir_to_codegen_ast", + "rendered_source", "x2py", ) diff --git a/tests/codegen/printers/test_calls_and_policy_metadata.py b/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py similarity index 88% rename from tests/codegen/printers/test_calls_and_policy_metadata.py rename to tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py index 0b64c3646..7a9ff837e 100644 --- a/tests/codegen/printers/test_calls_and_policy_metadata.py +++ b/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py @@ -1,15 +1,11 @@ """Tests split by stable ownership concept from `test_imports_and_packages.py`.""" -from tests.codegen.printers._support import ( - BindingPipeline, +from tests.wrapper_codegen.printers._support import ( CALLBACK_DECLARATION_ACCESS_METADATA, - Codegen, - Path, ProjectionMapping, PyiPrinter, RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA, - Scope, SemanticArgument, SemanticArrayContract, SemanticConstraint, @@ -23,11 +19,12 @@ emit_module, fortran_module_to_semantic_module, generate_pyi, + generate_wrapper_artifacts, normalize, parse_fortran_source, parse_pyi_text, pytest, - semantic_ir_to_codegen_ast, + rendered_source, ) @@ -212,7 +209,7 @@ def test_emit_native_call_supports_return_and_work_value_references(): assert "def wrapper() -> Float64: ..." in code -def test_runtime_policy_decorators_round_trip_through_pyi_and_codegen(tmp_path: Path): +def test_runtime_policy_decorators_round_trip_through_pyi(): loaded = parse_pyi_text( """ @raises(status="status", message="message", success=0) @@ -239,55 +236,15 @@ def serialized(x: Float64) -> Float64: ... assert "@hold_gil" in code assert emit_module(parse_pyi_text(code, module_name="runtime_policy")) == code - scope = Scope(name=loaded.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(loaded, scope) - pipeline = BindingPipeline( - Codegen(loaded.name, codegen_module, codegen_module.scope), - loaded.name, - "fortran", - verbose=0, - ) - - pipeline.generate(str(tmp_path)) - generated = pipeline.write(tmp_path) - - c_wrapper = generated[1].read_text() - solve_start = c_wrapper.index("static PyObject* bind_c_solve_wrapper") - serialized_start = c_wrapper.index("static PyObject* bind_c_serialized_wrapper") - solve_wrapper = c_wrapper[solve_start:serialized_start] - serialized_wrapper = c_wrapper[serialized_start : c_wrapper.index("static PyMethodDef", serialized_start)] - assert "Py_BEGIN_ALLOW_THREADS" in solve_wrapper - assert "Py_END_ALLOW_THREADS" in solve_wrapper - assert "Py_BEGIN_ALLOW_THREADS" not in serialized_wrapper - assert "Py_END_ALLOW_THREADS" not in serialized_wrapper - assert "PyErr_SetObject(PyExc_RuntimeError" in c_wrapper - assert "return solve_0001_obj;" in c_wrapper - assert "PyTuple_Pack" not in c_wrapper - assert c_wrapper.count("Py_DECREF(status_obj);") == 2 - assert c_wrapper.count("Py_DECREF(message_obj);") == 2 - assert "solve(x) -> float64" in c_wrapper - assert "RuntimeError" in c_wrapper - - -def test_callback_contract_holds_gil_and_release_gil_is_removed(tmp_path: Path): + +def test_callback_contract_holds_gil_and_release_gil_is_removed(): loaded = parse_pyi_text( """ def apply(callback: Callable[[Float64], Float64], x: Float64) -> Float64: ... """, module_name="callback_policy", ) - scope = Scope(name=loaded.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(loaded, scope) - pipeline = BindingPipeline( - Codegen(loaded.name, codegen_module, codegen_module.scope), - loaded.name, - "fortran", - verbose=0, - ) - pipeline.generate(str(tmp_path)) - generated = pipeline.write(tmp_path) - - c_wrapper = generated[1].read_text() + c_wrapper = rendered_source(generate_wrapper_artifacts(loaded), ".c") assert "Py_BEGIN_ALLOW_THREADS" not in c_wrapper assert "Py_END_ALLOW_THREADS" not in c_wrapper @@ -547,7 +504,7 @@ def test_printer_emits_nullable_scalar_descriptor_boundary_projections(argument_ assert ") -> Float64 | None: ..." in code -def test_defaulted_scalar_descriptors_preserve_omitted_vs_none_in_generated_wrappers(tmp_path): +def test_defaulted_scalar_descriptors_preserve_omitted_vs_none_in_generated_wrappers(): loaded = parse_pyi_text( """ @native_call([Allocatable(Arg(0)), Pointer(Arg(1))]) @@ -555,35 +512,25 @@ def update(scale: Float64 | None = ..., target: Float64 | None = ...) -> None: . """, module_name="optional_scalar_descriptors", ) - scope = Scope(name=loaded.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(loaded, scope) - pipeline = BindingPipeline( - Codegen(loaded.name, codegen_module, codegen_module.scope), - loaded.name, - "fortran", - verbose=0, - ) - - pipeline.generate(str(tmp_path)) - bridge_path, c_wrapper_path, *_ = pipeline.write(tmp_path) - bridge_source = bridge_path.read_text() - c_wrapper = c_wrapper_path.read_text() + artifacts = generate_wrapper_artifacts(loaded) + bridge_source = rendered_source(artifacts, ".f90") + c_wrapper = rendered_source(artifacts, ".c") assert "bound_scale_present" in bridge_source assert "bound_target_present" in bridge_source assert "if (c_associated(bound_scale_present)) then" in bridge_source assert "if (c_associated(bound_target_present)) then" in bridge_source - assert "call update()" in bridge_source - assert "call update(scale = scale_descriptor)" in bridge_source - assert "call update(target = target_descriptor)" in bridge_source - assert "call update(scale = scale_descriptor, target = target_descriptor" in bridge_source + assert "call native_update()" in bridge_source + assert "call native_update(scale=scale_descriptor)" in bridge_source + assert "call native_update(target=target_descriptor)" in bridge_source + assert "call native_update(scale=scale_descriptor, target=target_descriptor" in bridge_source assert "scale_obj = NULL;" in c_wrapper assert "target_obj = NULL;" in c_wrapper assert "if (scale_obj != NULL)" in c_wrapper - assert "scale_present = &scale_value;" in c_wrapper + assert "scale_present = &scale;" in c_wrapper assert "if ((scale_obj != NULL) && (scale_obj != Py_None))" in c_wrapper - assert "scale_nullable = &scale_value;" in c_wrapper + assert "scale_nullable = &scale;" in c_wrapper assert "bind_c_update(scale_nullable, scale_present, target_nullable, target_present);" in c_wrapper assert "Omit to make the native optional dummy absent." in c_wrapper assert "Pass None for a present unallocated or unassociated descriptor." in c_wrapper diff --git a/tests/codegen/printers/test_classes_and_methods.py b/tests/wrapper_codegen/printers/test_classes_and_methods.py similarity index 83% rename from tests/codegen/printers/test_classes_and_methods.py rename to tests/wrapper_codegen/printers/test_classes_and_methods.py index d5c6dfc02..3d3c3b35a 100644 --- a/tests/codegen/printers/test_classes_and_methods.py +++ b/tests/wrapper_codegen/printers/test_classes_and_methods.py @@ -1,13 +1,9 @@ """Tests split by stable ownership concept from `test_imports_and_packages.py`.""" -from tests.codegen.printers._support import ( - BindingPipeline, - Codegen, +from tests.wrapper_codegen.printers._support import ( OPERATOR_F90_SOURCE, - Path, ProjectionMapping, PyiPrinter, - Scope, SemanticArgument, SemanticClass, SemanticFunction, @@ -18,11 +14,12 @@ emit_module_stubs, fortran_module_to_semantic_module, generate_pyi, + generate_wrapper_artifacts, normalize, parse_fortran_source, parse_pyi_text, pytest, - semantic_ir_to_codegen_ast, + rendered_source, ) @@ -271,12 +268,6 @@ def test_emit_and_load_allocatable_module_variable_declaration(): assert loaded.classes[0].fields[0].semantic_type.storage.array.allocatable is True assert "aliased" not in loaded.classes[0].fields[0].semantic_type.metadata - codegen_module = semantic_ir_to_codegen_ast( - loaded, - Scope(name=loaded.name, scope_type="module"), - ) - assert codegen_module.variables[0].is_target is True - def test_emit_and_load_aliased_derived_module_variable_declaration(): source = """ @@ -296,12 +287,6 @@ def test_emit_and_load_aliased_derived_module_variable_declaration(): assert loaded.variables[0].semantic_type.name == "box" assert loaded.variables[0].semantic_type.metadata["aliased"] is True - codegen_module = semantic_ir_to_codegen_ast( - loaded, - Scope(name=loaded.name, scope_type="module"), - ) - assert codegen_module.variables[0].is_target is True - def test_emit_module_stubs_print_plain_derived_module_variable_as_live_object(): source = """ @@ -344,54 +329,32 @@ def test_defined_operator_pyi_round_trip_preserves_native_links_without_fortran_ loaded = parse_pyi_text(code, module_name=semantic_module.name) assert emit_module(loaded) == code - codegen_module = semantic_ir_to_codegen_ast( - loaded, - Scope(name=loaded.name, scope_type="module"), - ) - vector = next(cls for cls in codegen_module.classes if str(cls.name) == "vector") - overload_sets = {item.name: item.native_name for item in vector.overload_sets} - assert overload_sets["__add__"] == "operator(+)" - assert overload_sets["operator_dot"] == "operator(.dot.)" - assert overload_sets["assign"] == "assignment(=)" - assert set(next(item for item in vector.overload_sets if item.name == "__eq__").native_names) == { - "operator(==)", - "operator(.eqv.)", - } - - -def test_defined_operator_pyi_generates_wrapper_sources_without_fortran_source(tmp_path: Path): + + +def test_defined_operator_pyi_generates_wrapper_sources_without_fortran_source(): semantic_module = fortran_module_to_semantic_module( parse_fortran_source(OPERATOR_F90_SOURCE.read_text(), filename=str(OPERATOR_F90_SOURCE)) ) pyi = emit_module(semantic_module) loaded = parse_pyi_text(pyi, module_name=semantic_module.name) - scope = Scope(name=loaded.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(loaded, scope) - pipeline = BindingPipeline( - Codegen(loaded.name, codegen_module, codegen_module.scope), - loaded.name, - "fortran", - verbose=0, - ) - - pipeline.generate(str(tmp_path)) - generated = pipeline.write(tmp_path) + generated = generate_wrapper_artifacts(loaded) - assert [path.name for path in generated] == [ + assert [path.name for path in generated.source_paths] == [ "bind_c_foperators_f90_wrapper.f90", "foperators_f90_wrapper.c", + "foperators_f90_wrapper.h", ] - fortran_wrapper = generated[0].read_text() - c_wrapper = generated[1].read_text() + fortran_wrapper = rendered_source(generated, ".f90") + c_wrapper = rendered_source(generated, ".c") assert "left + right" in fortran_wrapper assert "left = right" in fortran_wrapper assert "left .eqv. right" in fortran_wrapper - assert "left .neqv. right" in fortran_wrapper - assert ".nb_add = (binaryfunc)" in c_wrapper - assert ".tp_richcompare =" in c_wrapper + assert " .neqv. " in fortran_wrapper + assert "def __add__(self, *args, **kwargs):" in c_wrapper + assert "def __ne__(self, *args, **kwargs):" in c_wrapper -def test_bound_constructor_pyi_generates_single_initializer_without_keyword_default(tmp_path: Path): +def test_bound_constructor_pyi_generates_single_initializer_without_keyword_default(): loaded = parse_pyi_text( """ class state: @@ -405,29 +368,20 @@ def __init__(self, seed: Addr(Int32)) -> None: ... """, module_name="edited", ) - scope = Scope(name=loaded.name, scope_type="module") - codegen_module = semantic_ir_to_codegen_ast(loaded, scope) - pipeline = BindingPipeline( - Codegen(loaded.name, codegen_module, codegen_module.scope), - loaded.name, - "fortran", - verbose=0, - ) - - pipeline.generate(str(tmp_path)) - generated = pipeline.write(tmp_path) + generated = generate_wrapper_artifacts(loaded) - assert [path.name for path in generated] == [ + assert [path.name for path in generated.source_paths] == [ "bind_c_edited_wrapper.f90", "edited_wrapper.c", + "edited_wrapper.h", ] - c_wrapper = generated[1].read_text() - assert "init_state" in generated[0].read_text() + c_wrapper = rendered_source(generated, ".c") + assert "init_state" in rendered_source(generated, ".f90") assert "state__default_init_wrapper" not in c_wrapper - assert '(char*)"seed"' in c_wrapper - assert 'PyArg_ParseTupleAndKeywords(args, kwargs, "O", kwlist, &seed_obj)' in c_wrapper - assert "Py_BEGIN_ALLOW_THREADS" not in c_wrapper - assert "Py_END_ALLOW_THREADS" not in c_wrapper + assert 'static char * kwlist[] = {"self", "seed", NULL};' in c_wrapper + assert 'PyArg_ParseTupleAndKeywords(args, kwargs, "OO", kwlist, &bound_self_obj, &seed_obj)' in c_wrapper + assert "Py_BEGIN_ALLOW_THREADS" in c_wrapper + assert "Py_END_ALLOW_THREADS" in c_wrapper def test_emit_module_variables_with_visibility(): diff --git a/tests/codegen/printers/test_modern_example.py b/tests/wrapper_codegen/printers/test_modern_example.py similarity index 96% rename from tests/codegen/printers/test_modern_example.py rename to tests/wrapper_codegen/printers/test_modern_example.py index 510ce65b5..45d41f0a0 100644 --- a/tests/codegen/printers/test_modern_example.py +++ b/tests/wrapper_codegen/printers/test_modern_example.py @@ -2,7 +2,7 @@ from x2py import parse_fortran_file from x2py.semantics.fortran2ir import fortran_module_to_semantic_module -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module def test_modern_fortran_example_pyi_snapshot(): diff --git a/tests/codegen/printers/test_pyi_printer_conversion_smoke.py b/tests/wrapper_codegen/printers/test_pyi_printer_conversion_smoke.py similarity index 92% rename from tests/codegen/printers/test_pyi_printer_conversion_smoke.py rename to tests/wrapper_codegen/printers/test_pyi_printer_conversion_smoke.py index 20ac6ae5b..de27d0502 100644 --- a/tests/codegen/printers/test_pyi_printer_conversion_smoke.py +++ b/tests/wrapper_codegen/printers/test_pyi_printer_conversion_smoke.py @@ -3,7 +3,7 @@ import pytest from x2py.semantics.fortran2ir import fortran_module_to_semantic_module -from x2py.codegen.printers.pyi_printer import emit_module +from x2py.wrapper_codegen.printers import emit_module from tests.semantics._fixture_conversion_utils import FORTRAN_FIXTURES, TESTS_DIR, parse_fixture diff --git a/tests/codegen/printers/test_pyi_printer_imports_and_packages.py b/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py similarity index 99% rename from tests/codegen/printers/test_pyi_printer_imports_and_packages.py rename to tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py index d52d117d1..ce5f27368 100644 --- a/tests/codegen/printers/test_pyi_printer_imports_and_packages.py +++ b/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py @@ -1,6 +1,6 @@ """Tests split by stable ownership concept from `test_imports_and_packages.py`.""" -from tests.codegen.printers._support import ( +from tests.wrapper_codegen.printers._support import ( PyiPrinter, SemanticArgument, SemanticArrayContract, diff --git a/tests/codegen/printers/test_types_and_declarations.py b/tests/wrapper_codegen/printers/test_types_and_declarations.py similarity index 99% rename from tests/codegen/printers/test_types_and_declarations.py rename to tests/wrapper_codegen/printers/test_types_and_declarations.py index 91e202c82..a82b6eca6 100644 --- a/tests/codegen/printers/test_types_and_declarations.py +++ b/tests/wrapper_codegen/printers/test_types_and_declarations.py @@ -1,6 +1,6 @@ """Tests split by stable ownership concept from `test_imports_and_packages.py`.""" -from tests.codegen.printers._support import ( +from tests.wrapper_codegen.printers._support import ( ProjectionMapping, PyiPrinter, SemanticArgument, diff --git a/tests/wrapper_codegen/test_phase0b_contracts.py b/tests/wrapper_codegen/test_phase0b_contracts.py index 8c540bbd8..f863732cb 100644 --- a/tests/wrapper_codegen/test_phase0b_contracts.py +++ b/tests/wrapper_codegen/test_phase0b_contracts.py @@ -18,11 +18,6 @@ SOURCE_ROOT = REPO_ROOT / "x2py" WRAPPER_CODEGEN_ROOT = SOURCE_ROOT / "wrapper_codegen" -LEGACY_CODEGEN_ROOT = SOURCE_ROOT / "codegen" - - -def _python_modules(root: Path) -> list[Path]: - return sorted(path for path in root.rglob("*.py") if "__pycache__" not in path.parts) def _imported_modules(path: Path) -> set[str]: @@ -51,28 +46,16 @@ def _check_source(tmp_path: Path, source: str, *, filename: str = "bad.py") -> s path = _write_module(tmp_path, filename, source) violations = check_wrapper_codegen_paths( [path], - package_root=tmp_path, config=WrapperCodegenCheckConfig(max_complexity=3, max_statements=4, max_nesting=2), ) return {violation.code for violation in violations} -def test_wrapper_codegen_and_legacy_codegen_do_not_import_each_other(): - wrapper_codegen_imports = {path: _imported_modules(path) for path in _python_modules(WRAPPER_CODEGEN_ROOT)} - wrapper_codegen_violations = sorted( - path.relative_to(REPO_ROOT).as_posix() - for path, imports in wrapper_codegen_imports.items() - if _imports_under(imports, "x2py.codegen") - ) - assert wrapper_codegen_violations == [] +def test_canonical_printers_share_one_package(): + printers = WRAPPER_CODEGEN_ROOT / "printers" - legacy_codegen_imports = {path: _imported_modules(path) for path in _python_modules(LEGACY_CODEGEN_ROOT)} - legacy_codegen_violations = sorted( - path.relative_to(REPO_ROOT).as_posix() - for path, imports in legacy_codegen_imports.items() - if _imports_under(imports, "x2py.wrapper_codegen") - ) - assert legacy_codegen_violations == [] + assert (printers / "pyi_printer.py").is_file() + assert (printers / "source_printers.py").is_file() def test_backend_generators_do_not_import_each_other(): @@ -83,15 +66,10 @@ def test_backend_generators_do_not_import_each_other(): assert not _imports_under(bridge_imports, "x2py.wrapper_codegen.c") -def test_only_pipeline_modules_may_import_both_wrapper_routes(): - modules_importing_both = [] - for path in _python_modules(SOURCE_ROOT): - imports = _imported_modules(path) - if _imports_under(imports, "x2py.codegen") and _imports_under(imports, "x2py.wrapper_codegen"): - modules_importing_both.append(path.relative_to(SOURCE_ROOT).as_posix()) +def test_wrapper_build_pipeline_imports_canonical_generator(): + imports = _imported_modules(SOURCE_ROOT / "pipeline" / "build.py") - outside_pipeline = sorted(path for path in modules_importing_both if not path.startswith("pipeline/")) - assert outside_pipeline == [] + assert _imports_under(imports, "x2py.wrapper_codegen") def test_wrapper_codegen_package_static_contracts_pass(): @@ -110,12 +88,6 @@ def test_wrapper_codegen_checker_command_runs_the_package_checker(): assert result.returncode == 0, result.stdout -def test_checker_rejects_legacy_codegen_imports(tmp_path: Path): - codes = _check_source(tmp_path, "import x2py.codegen\n") - - assert "legacy-codegen-import" in codes - - def test_checker_rejects_module_level_production_functions(tmp_path: Path): codes = _check_source(tmp_path, "def build_plan():\n return None\n") @@ -177,7 +149,7 @@ def _convert_item(self, value): """, ) - violations = check_wrapper_codegen_paths([path], package_root=tmp_path) + violations = check_wrapper_codegen_paths([path]) assert "complexity" in {violation.code for violation in violations} diff --git a/tests/wrapper_codegen/test_phase0e_backend_foundation.py b/tests/wrapper_codegen/test_phase0e_backend_foundation.py index 4c5decb66..f4e508d6d 100644 --- a/tests/wrapper_codegen/test_phase0e_backend_foundation.py +++ b/tests/wrapper_codegen/test_phase0e_backend_foundation.py @@ -132,7 +132,7 @@ def test_fortran_source_printer_wraps_long_parenthesized_call_arguments(): def test_source_printers_do_not_import_wrapper_plan_models(): - path = REPO_ROOT / "x2py" / "wrapper_codegen" / "source_printers.py" + path = REPO_ROOT / "x2py" / "wrapper_codegen" / "printers" / "source_printers.py" imports = { node.module for node in ast.walk(ast.parse(Path(path).read_text(encoding="utf-8"))) diff --git a/tests/wrapper_codegen/test_phase10_callbacks.py b/tests/wrapper_codegen/test_phase10_callbacks.py index a9ce14b62..74c320104 100644 --- a/tests/wrapper_codegen/test_phase10_callbacks.py +++ b/tests/wrapper_codegen/test_phase10_callbacks.py @@ -75,6 +75,8 @@ def test_callback_policy_completes_every_legacy_observed_transfer_before_plannin ) array = policies["apply_array_storage_callback"].arguments[0].callback + assert array.arguments[0].abi is CallbackABIKind.REFERENCE + assert array.arguments[0].adapter_action is CallbackTransferAction.COPY_IN assert array.arguments[1].abi is CallbackABIKind.DATA_AND_SHAPE assert array.arguments[1].array.shape == ("count",) @@ -164,9 +166,12 @@ def test_callback_artifacts_use_linear_context_adapter_and_trampoline_paths(): assert "Py_END_ALLOW_THREADS" not in c_source assert "abstract interface" in bridge + assert "integer(c_int32_t), intent(in), value :: arg_0" in bridge + assert "integer(c_int32_t), intent(in) :: count" in bridge assert "procedure(x2py_callback_trampoline" in bridge assert "size(arg_1_callback_storage, dim=1, kind=c_int64_t)" in bridge assert "int(len(arg_0_callback_storage), kind=c_int64_t)" in bridge + assert "Int32_to_PyLong((int32_t *)count_data)" in c_source assert bridge.count("call native_apply_array_storage_callback(") == 1 assert bridge.count("call callback(") == 3 assert max(map(len, bridge.splitlines())) <= 132 diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index 7a34a9bea..caafcfb9c 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -138,7 +138,12 @@ def test_supported_writeback_actions_select_scalar_result_behavior(codegen_actio c_source = _rendered_source(WrapperCodeGenerator().generate(edited), ".c") assert "bind_c_bump(&value);" in c_source - assert "PyObject * result_obj = Int32_to_PyLong(&value);" in c_source + if codegen_action is CodegenAction.COPY_IN_OUT: + assert "PyObject * result_obj = NULL;" in c_source + assert "result_obj = Int32_to_PyLong(&value);" in c_source + else: + assert "PyObject * result_obj = value_obj;" in c_source + assert "Py_INCREF(result_obj);" in c_source def test_direct_plan_edits_change_binding_and_bridge_generation_then_freeze_plan(): diff --git a/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py b/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py index a8b8c9aab..db3998a38 100644 --- a/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py +++ b/tests/wrapper_codegen/test_phase2d_native_runtime_envelope.py @@ -61,6 +61,9 @@ def test_planner_records_editable_native_runtime_and_status_error_facts(): assert solve.binding.status_error.exception_kind is PythonExceptionKind.RUNTIME_ERROR assert solve.binding.status_error.status_role == solve.native_call_slots[1].symbolic_role assert solve.binding.status_error.message_role == solve.native_call_slots[2].symbolic_role + assert "Raises\n------" in solve.binding.docstring + assert solve.binding.docstring.count("RuntimeError\n") == 1 + assert "If native status differs from the success value 0." in solve.binding.docstring assert solve.native_call_slots[1].semantic_type_name == "Int32" assert solve.native_call_slots[1].datatype_family is DatatypeFamily.INTEGER assert solve.native_call_slots[1].bridge_data_action is BridgeDataAction.DIRECT_TRANSFER diff --git a/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py b/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py index 6c9a20d21..b2db26731 100644 --- a/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py +++ b/tests/wrapper_codegen/test_phase2e_scalar_boundaries.py @@ -121,7 +121,8 @@ def test_scalar_copy_in_out_reuses_one_binding_local_without_bridge_copy(): assert c_source.count("int32_t value;") == 1 assert "value = PyInt32_to_Int32(value_obj);" in c_source assert "bind_c_bump(&value);" in c_source - assert "PyObject * result_obj = Int32_to_PyLong(&value);" in c_source + assert "PyObject * result_obj = NULL;" in c_source + assert "result_obj = Int32_to_PyLong(&value);" in c_source assert "integer(c_int32_t) :: value" in bridge_source assert "call native_bump(value)" in bridge_source assert "value =" not in bridge_source diff --git a/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py b/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py index 1d0944336..0f7e4aa8f 100644 --- a/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py +++ b/tests/wrapper_codegen/test_phase3_scalar_presence_and_writeback.py @@ -137,7 +137,8 @@ def test_scalar_writeback_is_an_explicit_binding_lifecycle_result(): assert "void bind_c_bump(int32_t * value);" in c_source assert "bind_c_bump(&value);" in c_source - assert "PyObject * result_obj = Int32_to_PyLong(&value);" in c_source + assert "PyObject * result_obj = NULL;" in c_source + assert "result_obj = Int32_to_PyLong(&value);" in c_source assert "subroutine bind_c_bump(value)" in fortran_source assert "call native_bump(value)" in fortran_source diff --git a/tests/wrapper_codegen/test_phase5b_fixed_string_results.py b/tests/wrapper_codegen/test_phase5b_fixed_string_results.py index 5247b59ff..7d5a5d0f7 100644 --- a/tests/wrapper_codegen/test_phase5b_fixed_string_results.py +++ b/tests/wrapper_codegen/test_phase5b_fixed_string_results.py @@ -166,7 +166,7 @@ def test_fixed_string_result_plan_edits_fail_before_backend_lowering(edit: str, WrapperCodeGenerator().generate(plan) -def test_fixed_string_result_policy_blocks_mixed_result_aggregation_until_cleanup_is_planned(): +def test_fixed_string_result_policy_uses_ordered_mixed_result_cleanup(): module = parse_pyi_text( """ @native_call([Return("status", 1)]) @@ -177,9 +177,19 @@ def mixed() -> tuple[String[8], Int32]: ... complete_semantic_policies(module) policy = module.functions[0].metadata[RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] - assert policy.supported is False - assert "fixed string result lane requires exactly one Python-visible result" in policy.blockers + assert policy.supported is True + assert policy.blockers == () assert policy.results[0].ownership.kind is ObjectKind.STRING + plan = WrapperPlanner().build(module) + function = plan.namespaces[0].functions[0] + assert tuple(result.result_position for result in function.results) == (0, 1) + + c_source = next( + source.text for source in WrapperCodeGenerator().generate(plan).sources if source.path.suffix == ".c" + ) + assert "free(result);" in c_source + assert "PyTuple_New(2)" in c_source + assert "Py_DECREF(result_0_obj);" in c_source def test_fixed_string_result_policy_blocks_status_error_until_failure_release_is_planned(): diff --git a/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py b/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py index 6f4ed82b0..431708f93 100644 --- a/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py +++ b/tests/wrapper_codegen/test_phase5c_fixed_string_writeback.py @@ -72,7 +72,7 @@ def test_fixed_replacement_projects_completed_argument_and_lifecycle_facts(): assert argument.storage_mode is StorageMode.STACK assert argument.boundary_storage_mode is StorageMode.STACK assert argument.nullable is False - assert argument.mutates_native is True + assert argument.mutates_native is False assert argument.projects_result is True assert argument.result_position == 0 assert argument.binding.codegen_action is CodegenAction.COPY_IN_OUT diff --git a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py index a1559dacc..26d1a30d6 100644 --- a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py +++ b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py @@ -71,6 +71,20 @@ def projected( return WrapperPlanner().build(module) +def _late_extent_external_plan(): + module = parse_pyi_text( + """ +from x2py.contracts import Float64, Int32, external + +@external +def late_extent(values: Float64[n], n: Int32) -> None: ... +""", + module_name="late_extent_external", + ) + complete_semantic_policies(module) + return WrapperPlanner().build(module) + + def test_dense_array_plan_records_extent_dependencies_flat_storage_and_order(): functions = {function.binding.python_name: function for function in _dense_plan().namespaces[0].functions} dense_f = functions["dense_f"].arguments[-1].array @@ -134,6 +148,15 @@ def test_dense_array_lowering_uses_planned_shape_checks_and_bridge_orientation() assert "real(c_double) :: values(3, *)" in bridge_source +def test_external_interface_declares_late_extent_before_dependent_array(): + artifacts = WrapperCodeGenerator().generate(_late_extent_external_plan()) + bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + signature = "subroutine late_extent(values, n)" + interface = bridge_source.split(signature, maxsplit=1)[1].split("end subroutine late_extent", maxsplit=1)[0] + assert interface.index("integer(c_int32_t) :: n") < interface.index("real(c_double), dimension(n) :: values") + + def test_unavailable_dense_extent_role_fails_before_backend_lowering(): plan = _dense_plan() array = plan.namespaces[0].functions[0].arguments[-1].array diff --git a/tests/wrapper_codegen/test_phase6c_strided_arrays.py b/tests/wrapper_codegen/test_phase6c_strided_arrays.py index 120d1cd00..b66876045 100644 --- a/tests/wrapper_codegen/test_phase6c_strided_arrays.py +++ b/tests/wrapper_codegen/test_phase6c_strided_arrays.py @@ -45,7 +45,8 @@ def test_strided_array_lowering_validates_and_passes_one_explicit_bridge_slice() c_source = next(source.text for source in artifacts.sources if source.path.suffix == ".c") bridge_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") - assert "expected ordering (F)" in c_source + assert '_native_array_actual_argument_for_binding_positional"' in c_source + assert 'PyUnicode_FromString("F")' in c_source assert "values_upper_bound_0" in c_source assert "values_stride_1" in c_source assert "real(c_double), pointer, dimension(:, :) :: values_base" in bridge_source diff --git a/tests/wrapper_codegen/test_phase8_derived_types.py b/tests/wrapper_codegen/test_phase8_derived_types.py index ad959b0d5..3c9205948 100644 --- a/tests/wrapper_codegen/test_phase8_derived_types.py +++ b/tests/wrapper_codegen/test_phase8_derived_types.py @@ -260,17 +260,6 @@ def consume(value: point) -> None: ... ), ( """ -from x2py.contracts import Float64, Returns - -class point: - x: Float64 - -def update(value: point) -> tuple[Float64, Returns["value", point]]: ... -""", - "cannot combine native results with visible argument writeback for 'value'", - ), - ( - """ class node: next: node """, @@ -293,6 +282,34 @@ def test_unsupported_derived_shapes_fail_on_exact_completed_policy_blockers( WrapperPlanner().build(module) +def test_native_result_and_derived_writeback_share_ordered_output_aggregation(): + module = parse_pyi_text( + """ +from x2py.contracts import Float64, Returns + +class point: + x: Float64 + +def update(value: point) -> tuple[Float64, Returns["value", point]]: ... +""", + module_name="phase8_mixed_writeback", + ) + complete_semantic_policies(module) + + report = WrapperPlanSupportAnalyzer().analyze(module) + assert report.supported is True + plan = WrapperPlanner().build(module) + function = plan.namespaces[0].functions[0] + assert function.results[0].result_position == 0 + assert function.writeback_actions[2].result_position == 1 + + c_source = next( + source.text for source in WrapperCodeGenerator().generate(plan).sources if source.path.suffix == ".c" + ) + assert "PyTuple_New(2)" in c_source + assert "Py_DECREF(result_0_obj);" in c_source + + def test_mixed_derived_results_check_allocation_and_own_every_failure_path_before_scalar_conversion(): module = parse_pyi_text( """ diff --git a/tests/wrapper_codegen/test_phase9_class_surfaces.py b/tests/wrapper_codegen/test_phase9_class_surfaces.py index 1a92fe3b9..ff2bb7db1 100644 --- a/tests/wrapper_codegen/test_phase9_class_surfaces.py +++ b/tests/wrapper_codegen/test_phase9_class_surfaces.py @@ -6,7 +6,7 @@ from x2py.pipeline.pyi import pyi_file_to_semantic_module from x2py.semantics.policy_completion import complete_semantic_policies -from x2py.semantics.wrapper_policy import ClassInvocationKind, ClassOverloadMatchKind +from x2py.semantics.wrapper_policy import ClassInvocationKind, OverloadMatchKind from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner ROOT = Path(__file__).parents[1] / "wrapper" / "fortran" @@ -62,8 +62,8 @@ def test_class_overloads_project_exact_typed_matches_and_type_bound_calls(): assert constructor is not None for overload in (method, constructor): assert tuple(matches[0].kind for matches in overload.candidate_matches) == ( - ClassOverloadMatchKind.NUMPY_SCALAR, - ClassOverloadMatchKind.NUMPY_SCALAR, + OverloadMatchKind.NUMPY_SCALAR, + OverloadMatchKind.NUMPY_SCALAR, ) assert tuple(matches[0].semantic_type_name for matches in overload.candidate_matches) == ( "Int32", @@ -79,10 +79,42 @@ def test_bound_constructor_links_one_existing_method_function_plan(): plan = _plan(BOUND_CONSTRUCTOR) surface = _surface(plan, "vector") target = surface.constructor.target + namespace = plan.namespaces[0] assert target is not None assert target is surface.methods[0].function assert target.binding.python_name.startswith("_x2py_class_") + assert target.binding.public is False + assert "_x2py_class_" not in namespace.docstring + + +def test_class_docstrings_describe_only_the_public_surface(): + plan = _plan(BOUND_CONSTRUCTOR) + surface = _surface(plan, "vector") + + assert "Constructor\n-----------\nvector(dx, dy) -> vector" in surface.docstring + assert "Fields\n------\nx : float64\ny : float64" in surface.docstring + assert "Methods\n-------\nshift(dx, dy) -> None" in surface.docstring + assert "vector(dx, dy) -> vector" in surface.constructor.docstring + assert "dx : float64" in surface.constructor.docstring + assert "shift(dx, dy) -> None" in surface.methods[0].docstring + assert "Updates the wrapped native instance in place." in surface.methods[0].docstring + assert "owner" not in surface.methods[0].docstring + assert "_x2py_class_" not in surface.docstring + + +def test_overload_docstrings_distinguish_candidates_by_public_types(): + plan = _plan(OVERLOADS) + surface = _surface(plan, "accumulator") + method = next(overload for overload in surface.overloads if overload.python_name == "add") + + assert "add(value: int32) -> None" in method.docstring + assert "add(value: float64) -> None" in method.docstring + assert "Dispatches to a native operation on the wrapped instance." in method.docstring + assert "accumulator(value: int32) -> accumulator" in surface.constructor.docstring + assert "accumulator(value: float64) -> accumulator" in surface.constructor.docstring + assert "_x2py_class_" not in method.docstring + assert "accumulator_add_" not in method.docstring def test_invalid_class_graph_and_overload_edits_fail_before_emission(): @@ -94,7 +126,7 @@ def test_invalid_class_graph_and_overload_edits_fail_before_emission(): overloads = _plan(OVERLOADS) overload = next(item for item in _surface(overloads, "accumulator").overloads if item.python_name == "add") overload.candidate_matches = (overload.candidate_matches[0], overload.candidate_matches[0]) - with pytest.raises(ValueError, match="ambiguous-class-overload"): + with pytest.raises(ValueError, match="ambiguous-overload"): WrapperCodeGenerator().generate(overloads) diff --git a/tools/check_radon_policy.py b/tools/check_radon_policy.py index c219140d7..934ed1922 100644 --- a/tools/check_radon_policy.py +++ b/tools/check_radon_policy.py @@ -19,25 +19,6 @@ DEFAULT_HOTSPOT_MIN_COMPLEXITY = 11 ZERO_SHA = "0" * 40 HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@") -LEGACY_COMPLEXITY_BASELINE = { - ("x2py/codegen/bindings/c_to_python.py", "function", "CPythonBindingGenerator._visit_FunctionDef"): 35, - ("x2py/codegen/bridges/fortran_to_c.py", "function", "FortranToCBridgeGenerator._visit_Module"): 23, - ("x2py/codegen/models/core.py", "function", "Module.__init__"): 30, - ("x2py/codegen/models/core.py", "function", "FunctionCall.__init__"): 24, - ("x2py/codegen/models/core.py", "function", "FunctionDef.__init__"): 27, - ("x2py/codegen/printers/ccode.py", "function", "CCodePrinter.is_c_pointer"): 25, - ("x2py/codegen/printers/ccode.py", "function", "CCodePrinter._print_ModuleHeader"): 25, - ("x2py/codegen/printers/ccode.py", "function", "CCodePrinter._print_FunctionDef"): 26, - ("x2py/codegen/printers/ccode.py", "function", "CCodePrinter._print_FunctionCall"): 22, - ("x2py/codegen/printers/cpythoncode.py", "function", "CPythonCodePrinter._print_PyClassDef"): 37, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter._print_Module"): 38, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter._print_Declare"): 47, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter.function_signature"): 21, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter._print_FunctionDef"): 27, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter._print_FunctionCall"): 29, - ("x2py/codegen/printers/fcode.py", "function", "FCodePrinter._wrap_fortran"): 23, - ("x2py/semantics/ir2ast.py", "function", "semantic_ir_to_codegen_ast"): 33, -} @dataclass(frozen=True) @@ -283,7 +264,7 @@ def changed_complexity_blocks( if not block_changed(block, changed_lines): continue changed_blocks_checked += 1 - base_complexity = base_complexities.get(block_key(block), legacy_baseline_complexity(block)) + base_complexity = base_complexities.get(block_key(block)) if changed_block_violates_policy(block, base_complexity, max_changed_complexity): changed_violations.append(block) return changed_blocks_checked, changed_violations @@ -299,10 +280,6 @@ def changed_block_violates_policy( return base_complexity is None or block.complexity > base_complexity -def legacy_baseline_complexity(block: ComplexityBlock) -> int | None: - return LEGACY_COMPLEXITY_BASELINE.get((block.path.as_posix(), block.kind, block.name)) - - def base_complexity_by_key(base_ref: str, changed_file: str | None) -> dict[tuple[str, str], int]: if changed_file is None: return {} diff --git a/tools/replay_wrapper_plan.py b/tools/replay_wrapper_plan.py deleted file mode 100644 index 9e209d6e1..000000000 --- a/tools/replay_wrapper_plan.py +++ /dev/null @@ -1,298 +0,0 @@ -#!/usr/bin/env python3 -"""Retain and verify one maintained dual-route wrapper-plan replay.""" - -from __future__ import annotations - -import argparse -from dataclasses import dataclass -import importlib -from pathlib import Path -import sys - -from tests.wrapper.fortran._support import _assert_fmath_examples, _compile_native_object, _sole_native_module -from tests.wrapper.fortran.scalars.test_verified_baseline import _scalar_conversion_failure -from x2py.fortran_parser.parser import parse_fortran_project -from x2py.pipeline import build as build_pipeline -from x2py.pipeline.preprocessing import PreprocessingConfig -from x2py.semantics.fortran2ir import fortran_project_to_semantic_modules -from x2py.semantics.policy_completion import complete_semantic_policies - -REPO_ROOT = Path(__file__).resolve().parents[1] - - -@dataclass(frozen=True) -class ReplayUnit: - """One existing scalar generation unit retained for maintainer replay.""" - - source: Path - contract: Path - - -_REPLAY_UNITS = { - "fmath": ReplayUnit( - source=REPO_ROOT / "tests" / "data" / "fortran" / "wrapper" / "fmath.f", - contract=(REPO_ROOT / "tests" / "wrapper" / "fortran" / "scalars" / "contracts" / "fmath" / "__init__.pyi"), - ) -} - - -def replay_wrapper_plan(*, entry: str, unit_name: str, output_dir: Path) -> Path: - """Replay one maintained source or contract unit through both wrapper routes.""" - unit = _replay_unit(unit_name) - _prepare_output_directory(output_dir) - module = _semantic_module(entry, unit) - legacy_decision = _route_decision(module, force_legacy=True) - wrapper_plan_decision = _route_decision(module, force_wrapper_plan=True) - legacy_result, wrapper_plan_result = _build_routes(entry, unit, output_dir) - _assert_artifact_parity(legacy_result, wrapper_plan_result) - legacy_module = _import_extension(legacy_result.module_name, legacy_result.output_dir) - wrapper_plan_module = _import_extension(wrapper_plan_result.module_name, wrapper_plan_result.output_dir) - _assert_fmath_examples(legacy_module) - _assert_fmath_examples(wrapper_plan_module) - _assert_failure_parity(legacy_module, wrapper_plan_module) - report_path = output_dir / "route-report.txt" - report_path.write_text( - _route_report( - entry=entry, - unit_name=unit_name, - output_dir=output_dir, - legacy_decision=legacy_decision, - wrapper_plan_decision=wrapper_plan_decision, - legacy_result=legacy_result, - wrapper_plan_result=wrapper_plan_result, - ), - encoding="utf-8", - ) - return report_path - - -def _replay_unit(unit_name: str) -> ReplayUnit: - """Return one explicitly maintained existing replay unit.""" - try: - return _REPLAY_UNITS[unit_name] - except KeyError as exc: - choices = ", ".join(sorted(_REPLAY_UNITS)) - raise ValueError(f"Unsupported replay unit {unit_name!r}; choose one of: {choices}") from exc - - -def _prepare_output_directory(output_dir: Path) -> None: - """Create an empty retained-artifact directory for one deterministic replay.""" - if output_dir.exists() and any(output_dir.iterdir()): - raise ValueError(f"Replay output directory must be empty: {output_dir}") - output_dir.mkdir(parents=True, exist_ok=True) - - -def _semantic_module(entry: str, unit: ReplayUnit): - """Build the policy-completed module that the selected replay entry uses.""" - if entry == "source": - return _source_semantic_module(unit.source) - if entry == "pyi": - return _pyi_semantic_module(unit.contract) - raise ValueError(f"Unsupported replay entry {entry!r}") - - -def _source_semantic_module(source: Path): - """Return the merged source-driven semantic module for plan diagnostics.""" - parsed = parse_fortran_project( - {str(source): build_pipeline._fortran_source_for_pipeline(source, PreprocessingConfig())} - ) - modules = fortran_project_to_semantic_modules(parsed) - build_pipeline._apply_source_python_exports(modules) - module = build_pipeline._merge_wrapper_modules(modules, name=source.stem) - complete_semantic_policies(module) - return module - - -def _pyi_semantic_module(contract: Path): - """Return the merged semantic-contract module for plan diagnostics.""" - bundle = build_pipeline._pyi_contract_bundle(contract) - module = build_pipeline._merge_wrapper_modules( - list(bundle.modules), - name=build_pipeline._bundle_output_name(bundle), - ) - complete_semantic_policies(module) - return module - - -def _route_decision(module, *, force_legacy: bool = False, force_wrapper_plan: bool = False): - """Select one explicit route for the retained diagnostic report.""" - return build_pipeline._select_wrapper_plan_route( - module, - makefile=False, - strict_wrapper_names=False, - force_legacy=force_legacy, - force_wrapper_plan=force_wrapper_plan, - ) - - -def _build_routes(entry: str, unit: ReplayUnit, output_dir: Path): - """Build the retained legacy and wrapper-plan artifact sets for one entry.""" - if entry == "source": - return _build_source_routes(unit, output_dir) - return _build_pyi_routes(unit, output_dir) - - -def _build_source_routes(unit: ReplayUnit, output_dir: Path): - """Build the existing source fixture through both explicit internal routes.""" - legacy = build_pipeline.build_fortran_extension( - unit.source, - output_dir=output_dir / "legacy", - _force_legacy_wrapper_route=True, - ) - wrapper_plan = build_pipeline.build_fortran_extension( - unit.source, - output_dir=output_dir / "wrapper-plan", - _force_wrapper_plan_route=True, - ) - return legacy, wrapper_plan - - -def _build_pyi_routes(unit: ReplayUnit, output_dir: Path): - """Build the existing semantic contract through both explicit internal routes.""" - native_object = _compile_native_object(unit.source, output_dir / "native") - common = { - "native_objects": (native_object,), - "native_include_dirs": (native_object.parent,), - } - legacy = build_pipeline.build_pyi_extension( - unit.contract, - output_dir=output_dir / "legacy", - _force_legacy_wrapper_route=True, - **common, - ) - wrapper_plan = build_pipeline.build_pyi_extension( - unit.contract, - output_dir=output_dir / "wrapper-plan", - _force_wrapper_plan_route=True, - **common, - ) - return legacy, wrapper_plan - - -def _assert_artifact_parity(legacy_result, wrapper_plan_result) -> None: - """Require both routes to retain the same generated artifact names.""" - legacy_names = tuple(path.name for path in legacy_result.generated_sources) - wrapper_plan_names = tuple(path.name for path in wrapper_plan_result.generated_sources) - if legacy_names != wrapper_plan_names: - raise AssertionError(f"Generated artifact names differ: {legacy_names!r} != {wrapper_plan_names!r}") - if not legacy_names: - raise AssertionError("Maintainer replay retained no generated wrapper artifacts") - - -def _import_extension(module_name: str, output_dir: Path): - """Import one retained extension without reusing the other route's module.""" - sys.modules.pop(module_name, None) - sys.path.insert(0, str(output_dir)) - try: - return _sole_native_module(importlib.import_module(module_name)) - finally: - sys.path.remove(str(output_dir)) - - -def _assert_failure_parity(legacy_module, wrapper_plan_module) -> None: - """Reuse the existing conversion failure and recovery assertion for both routes.""" - legacy_failure = _scalar_conversion_failure(legacy_module) - wrapper_plan_failure = _scalar_conversion_failure(wrapper_plan_module) - if legacy_failure != wrapper_plan_failure: - raise AssertionError(f"Scalar conversion failures differ: {legacy_failure!r} != {wrapper_plan_failure!r}") - - -def _route_report( - *, - entry: str, - unit_name: str, - output_dir: Path, - legacy_decision, - wrapper_plan_decision, - legacy_result, - wrapper_plan_result, -) -> str: - """Render deterministic route, artifact, and build-requirement evidence.""" - lines = [ - f"entry={entry}", - f"unit={unit_name}", - *_decision_lines("legacy", legacy_decision), - *_decision_lines("wrapper-plan", wrapper_plan_decision), - *_build_lines("legacy", legacy_result, output_dir), - *_build_lines("wrapper-plan", wrapper_plan_result, output_dir), - "artifact_names_equal=true", - "runtime_assertions=passed", - "conversion_failure_parity=passed", - ] - return "\n".join(lines) + "\n" - - -def _decision_lines(label: str, decision) -> tuple[str, ...]: - """Return stable recorded fields from one structured pipeline decision.""" - blockers = ",".join(f"{item.owner_path}:{item.reason}" for item in decision.blockers) or "none" - return ( - f"{label}.owner={decision.owner_path}", - f"{label}.selected_route={decision.selected_route}", - f"{label}.covered_lanes={','.join(decision.covered_lanes) or 'none'}", - f"{label}.rollout_eligible={str(decision.rollout_eligible).lower()}", - f"{label}.rollout_evidence={','.join(decision.rollout_evidence) or 'none'}", - f"{label}.selection_reason={decision.selection_reason}", - f"{label}.blockers={blockers}", - ) - - -def _build_lines(label: str, result, output_dir: Path) -> tuple[str, ...]: - """Return stable artifact and native-build requirements for one route.""" - plan = result.native_build_plan - return ( - f"{label}.module_name={result.module_name}", - f"{label}.compiled={str(result.compiled).lower()}", - f"{label}.generated_sources={_paths(result.generated_sources, output_dir)}", - f"{label}.generated_files={_paths(result.generated_files, output_dir)}", - f"{label}.native_compilation_units={_paths((unit.source for unit in plan.compilation_units), output_dir)}", - f"{label}.native_produced_objects={_paths(plan.produced_objects, output_dir)}", - f"{label}.native_prebuilt_artifacts={_paths((item.path for item in plan.prebuilt_artifacts), output_dir)}", - f"{label}.native_link_items={_link_items(plan.link_items, output_dir)}", - ) - - -def _paths(paths, output_dir: Path) -> str: - """Render replay-relative paths without output-directory-specific absolute text.""" - values = tuple(_relative_path(Path(path), output_dir) for path in paths) - return ",".join(values) or "none" - - -def _link_items(items, output_dir: Path) -> str: - """Render the ordered native link plan with stable path spellings.""" - values = [] - for item in items: - value = _relative_path(item.value, output_dir) if isinstance(item.value, Path) else str(item.value) - values.append(f"{item.kind}:{value}") - return ",".join(values) or "none" - - -def _relative_path(path: Path, output_dir: Path) -> str: - """Return an output-relative path or a stable external artifact name.""" - try: - return path.relative_to(output_dir).as_posix() - except ValueError: - return path.name - - -def _arguments() -> argparse.Namespace: - """Parse the private maintainer replay command line.""" - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--entry", choices=("source", "pyi"), required=True) - parser.add_argument("--unit", choices=tuple(sorted(_REPLAY_UNITS)), required=True) - parser.add_argument("--output-dir", type=Path, required=True) - return parser.parse_args() - - -def main() -> None: - """Run one clean retained-artifact wrapper-plan replay.""" - args = _arguments() - report_path = replay_wrapper_plan( - entry=args.entry, - unit_name=args.unit, - output_dir=args.output_dir, - ) - print(f"Retained wrapper-plan replay: {report_path}") - - -if __name__ == "__main__": - main() diff --git a/x2py/README.md b/x2py/README.md index cd0f57cf4..aab5b9ac8 100644 --- a/x2py/README.md +++ b/x2py/README.md @@ -15,15 +15,16 @@ jumping directly into generated-code internals. | `runtime/` | Python runtime objects used by generated extensions. | | `types/` | Semantic-to-Python ecosystem type mappings. | | `c_parser/` and `fortran_parser/` | Native source frontends and parser models. | -| `semantics/` | Language-neutral semantic IR, policy completion, readiness, `.pyi` conversion, and codegen lowering. | -| `codegen/` | Codegen AST, Fortran bridge generation, CPython binding generation, and printers. | +| `semantics/` | Language-neutral semantic IR, policy completion, readiness, and `.pyi` conversion. | +| `wrapper_codegen/` | Canonical wrapper plans, direct native bridge/binding generation, and source printers. | | `compiling/` | Native compiler objects, wrapper compilation, runtime support installation, and linking. | | `utilities/` | Small domain-neutral helpers, including class visitor dispatch. | -The package root contains only `__init__.py`, `__main__.py`, and `cli.py`. -Supported library symbols are flattened through `x2py.__init__`; internal code -imports the canonical owning module. `x2py.contracts` remains a deliberate -public submodule because its import path is part of semantic `.pyi` syntax. +The package root contains the public entrypoint modules plus the shared +`stage_values.py` record support. Supported library symbols are flattened +through `x2py.__init__`; internal modules import their canonical owner. +`x2py.contracts` remains a deliberate public submodule because its import path +is part of semantic `.pyi` syntax. ## Source Navigation Docs diff --git a/x2py/__init__.py b/x2py/__init__.py index 45adb38f0..516cf6405 100644 --- a/x2py/__init__.py +++ b/x2py/__init__.py @@ -39,7 +39,7 @@ ) from x2py.semantics.pyi2ir import convert_pyi_to_ir from x2py.pipeline.pyi import pyi_file_to_semantic_module, pyi_paths_to_semantic_modules, pyi_text_to_semantic_module -from x2py.codegen.printers.pyi_printer import emit_module_stubs, opaque_dependency_modules +from x2py.wrapper_codegen.printers import emit_module_stubs, opaque_dependency_modules from x2py.semantics.readiness import assess_pyi_wrap_readiness, assess_semantic_wrap_readiness from x2py.runtime.handles import AllocatableArray, NativeArrayHandleBase, PointerArray diff --git a/x2py/cli.py b/x2py/cli.py index 5639f01f7..d150f63ba 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -569,7 +569,7 @@ def _convert_fortran_semantic_sources( def _semantic_payload_for_converted_files(converted_files) -> dict[str, dict]: - from x2py.codegen.printers.pyi_printer import emit_module_stubs + from x2py.wrapper_codegen.printers import emit_module_stubs out: dict[str, dict] = {} available_modules = [module for _p, modules in converted_files for module in modules] @@ -596,7 +596,7 @@ def _is_fortran_semantic_file(modules) -> bool: def _fortran_contract_payload(path: Path, modules, available_modules) -> dict[str, object]: - from x2py.codegen.printers.pyi_printer import emit_module_stubs + from x2py.wrapper_codegen.printers import emit_module_stubs native_modules = [module for module in modules if module.origin.source_kind == "module"] external_modules = [module for module in modules if module.origin.source_kind != "module"] diff --git a/x2py/codegen/README.md b/x2py/codegen/README.md deleted file mode 100644 index 5316b50de..000000000 --- a/x2py/codegen/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Codegen Package - -This package owns generated wrapper representations and printers. It receives -codegen AST from semantic lowering and emits Fortran bridge and C/CPython -binding source for the implemented Fortran wrapper path. - -## Package Map - -| Path | Owns | -| --- | --- | -| `models/` | Codegen AST nodes and datatype models. | -| `bridges/fortran_to_c.py` | Fortran-to-C ABI bridge generation. | -| `bindings/c_to_python.py` | CPython extension binding generation. | -| `bindings/cpython_api.py` and `bindings/numpy_cpython_api.py` | Helper AST/API models for Python and NumPy C APIs. | -| `printers/fcode.py` | Fortran source printing. | -| `printers/cpythoncode.py` and `printers/ccode.py` | C/CPython source printing. | -| `printers/pyi_printer.py` | Semantic `.pyi` contract printing. | -| `binding_pipeline.py` | Ordered bridge and binding generation pipeline. | -| `scope.py` | Codegen scope and name lookup helpers. | - -## Rules Of Thumb - -- Keep runtime wrapper policy above codegen when possible: semantic lowering and - ownership policy decide what should happen; generators emit it. -- Use explicit dispatch tables for secondary policy dimensions such as datatype - or ownership action. -- Do not add placeholder backends without documented runtime contracts and - tests. -- A wrapper feature is supported only when generated sources compile, import, - and pass runtime behavior and failure-path tests. - -## Tests And Docs - -- User wrapper contract: `docs/user/guide/fortran-wrapper.md` -- Source navigation: `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md` -- Pipeline map: `docs/maintainer/internal-architecture/pipeline-map.md` -- Runtime tests: `tests/wrapper/fortran/` -- `.pyi` printer tests: `tests/codegen/printers/` diff --git a/x2py/codegen/__init__.py b/x2py/codegen/__init__.py deleted file mode 100644 index 7719e9ef4..000000000 --- a/x2py/codegen/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Code generation package.""" - -from x2py.codegen.codegen import Codegen - -__all__ = ("Codegen",) diff --git a/x2py/codegen/bind_c.py b/x2py/codegen/bind_c.py deleted file mode 100644 index 2b554d502..000000000 --- a/x2py/codegen/bind_c.py +++ /dev/null @@ -1,1208 +0,0 @@ -""" -Module describing all elements of the AST needed to represent elements which appear in a Fortran-C binding -file. -""" - -from functools import cache - -from .models.core import ( - ClassDef, - Deallocate, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - Module, - Function, -) -from .models.datatypes import ( - FixedSizeType, - init_model_object, - NumpyInt64Type, - register_model_class, - StringType, - Type, - TupleType, - convert_to_literal, -) -from .models.core import Variable - -__all__ = ( - "C_NULL_CHAR", - "BindCAccessorModuleVariable", - "BindCArrayType", - "BindCArrayVariable", - "BindCClassDef", - "BindCClassProperty", - "BindCFunctionDef", - "BindCModule", - "BindCModuleConstant", - "BindCModuleVariable", - "BindCNativeArrayDescriptorType", - "BindCNativeArrayHandleProperty", - "BindCNativeArrayHandleVariable", - "BindCPointer", - "BindCResultTupleType", - "BindCScalarDescriptorType", - "BindCSizeOf", - "BindCVariable", - "CLocFunc", - "C_F_Pointer", - "DeallocatePointer", - "FortranTransfer", - "c_malloc", - "native_array_descriptor_argument_type", -) - -# ======================================================================================= -# Datatypes -# ======================================================================================= - - -class BindCPointer(FixedSizeType): - """ - Datatype representing a C pointer in Fortran. - - Datatype representing a C pointer in Fortran. This data type is defined - in the iso_c_binding module. - """ - - __slots__ = () - _name = "bindcpointer" - - -class BindCArrayType(Type, TupleType): - """ - Datatype for a tuple containing all the information necessary to describe an array. - - Datatype for a tuple containing a pointer to array data and integers describing their - shape and strides. - """ - - __slots__ = ("_array_rank", "_element_types", "_has_itemsize", "_has_rank", "_has_strides") - _name = "BindCArrayType" - - @classmethod - def get_new(cls, rank, has_strides, has_rank=False, has_itemsize=False): - """ - Get the parametrised BindCArrayType subclass. - - Get the parametrised BindCArrayType subclass. - - Parameters - ---------- - rank : int - The rank of the array being described. - has_strides : bool - Indicates whether strides are used to describe the array. - has_rank : bool - Indicates whether the descriptor carries a runtime rank field. - has_itemsize : bool - Indicates whether the descriptor carries a fixed-width character - element byte length. - """ - if not isinstance(rank, int): - raise TypeError("rank must be an integer") - if rank < 1: - raise ValueError("rank must be positive") - if not isinstance(has_strides, bool): - raise TypeError("has_strides must be a boolean") - if not isinstance(has_rank, bool): - raise TypeError("has_rank must be a boolean") - if not isinstance(has_itemsize, bool): - raise TypeError("has_itemsize must be a boolean") - return cls._get_new(rank, has_strides, has_rank, has_itemsize) - - @classmethod - @cache - def _get_new(cls, rank, has_strides, has_rank, has_itemsize): - rank_types = (NumpyInt64Type(),) if has_rank else () - itemsize_types = (NumpyInt64Type(),) if has_itemsize else () - shape_types = (NumpyInt64Type(),) * rank - ubound_types = (NumpyInt64Type(),) * rank * has_strides - stride_types = (NumpyInt64Type(),) * rank * has_strides - element_types = (BindCPointer(), *rank_types, *itemsize_types, *shape_types, *ubound_types, *stride_types) - - def __init__(self): - self._array_rank = rank - self._has_strides = has_strides - self._has_rank = has_rank - self._has_itemsize = has_itemsize - self._element_types = element_types - Type.__init__(self) - - name = f"BindCArray{rank}DType" - if has_rank: - name += "_ranked" - if has_itemsize: - name += "_itemsize" - if has_strides: - name += "_strided" - return type(name, (BindCArrayType,), {"__init__": __init__})() - - @property - def array_rank(self): - """Rank of the array described by this packed argument.""" - return self._array_rank - - @property - def has_strides(self): - """Whether upper bounds and strides are present in the packed argument.""" - return self._has_strides - - @property - def has_rank(self): - """Whether a runtime rank field is present in the packed argument.""" - return self._has_rank - - @property - def has_itemsize(self): - """Whether a fixed-width character itemsize field is present.""" - return self._has_itemsize - - @property - def element_types(self): - """Types of the pointer, shape, upper-bound, and stride fields.""" - return self._element_types - - @property - def container_rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def order(self): - """Memory order is not applicable to the packed descriptor.""" - return None - - @property - def datatype(self): - """The descriptor is heterogeneous, so its datatype is the descriptor itself.""" - return self - - def shape_is_compatible(self, shape): - """Return whether ``shape`` has one entry with the descriptor field count.""" - return isinstance(shape, tuple) and len(shape) == 1 and shape[0] == len(self) - - def __getitem__(self, index): - return self._element_types[index] - - def __len__(self): - return len(self._element_types) - - def __iter__(self): - return iter(self._element_types) - - -class BindCScalarDescriptorType(Type, TupleType): - """Two-pointer C descriptor for nullable optional scalar dummies.""" - - __slots__ = () - _name = "BindCScalarDescriptorType" - _element_types = (BindCPointer(), BindCPointer()) - - @property - def element_types(self): - """Types of the value pointer and supplied/present token fields.""" - return self._element_types - - @property - def container_rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def order(self): - """Memory order is not applicable to the packed descriptor.""" - return None - - @property - def datatype(self): - """The descriptor is heterogeneous, so its datatype is the descriptor itself.""" - return self - - def shape_is_compatible(self, shape): - """Return whether ``shape`` has one entry with the descriptor field count.""" - return isinstance(shape, tuple) and len(shape) == 1 and shape[0] == len(self) - - def __getitem__(self, index): - return self._element_types[index] - - def __len__(self): - return len(self._element_types) - - def __iter__(self): - return iter(self._element_types) - - -class BindCNativeArrayDescriptorType(Type, TupleType): - """C descriptor-pointer tuple for native array handle dummy arguments.""" - - __slots__ = ("_element_types", "_has_presence") - _name = "BindCNativeArrayDescriptorType" - - @classmethod - def get_new(cls, *, has_presence=False): - """Return the descriptor tuple shape for required or optional handles.""" - if not isinstance(has_presence, bool): - raise TypeError("has_presence must be a boolean") - return cls._get_new(has_presence) - - @classmethod - @cache - def _get_new(cls, has_presence): - element_types = (BindCPointer(), BindCPointer()) if has_presence else (BindCPointer(),) - - def __init__(self): - self._has_presence = has_presence - self._element_types = element_types - Type.__init__(self) - - name = "BindCNativeArrayDescriptorType" - if has_presence: - name += "_present" - return type(name, (BindCNativeArrayDescriptorType,), {"__init__": __init__})() - - @property - def has_presence(self): - """Whether the tuple includes an optional-dummy presence token.""" - return self._has_presence - - @property - def element_types(self): - """Types of the descriptor pointer and optional presence fields.""" - return self._element_types - - @property - def container_rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def rank(self): - """Rank of the packed descriptor itself.""" - return 1 - - @property - def order(self): - """Memory order is not applicable to the packed descriptor.""" - return None - - @property - def datatype(self): - """The descriptor is heterogeneous, so its datatype is the descriptor itself.""" - return self - - def shape_is_compatible(self, shape): - """Return whether ``shape`` has one entry with the descriptor field count.""" - return isinstance(shape, tuple) and len(shape) == 1 and shape[0] == len(self) - - def __getitem__(self, index): - return self._element_types[index] - - def __len__(self): - return len(self._element_types) - - def __iter__(self): - return iter(self._element_types) - - -def native_array_descriptor_argument_type(policy): - """Return the Bind-C tuple shape selected for a native array descriptor argument.""" - return BindCNativeArrayDescriptorType.get_new(has_presence=bool(policy.optional_absent)) - - -class BindCResultTupleType(Type, TupleType): - """Datatype for a heterogeneous set of C-compatible function outputs.""" - - __slots__ = ("_element_types",) - _name = "BindCResultTupleType" - - @classmethod - def get_new(cls, element_types): - element_types = tuple(element_types) - if len(element_types) < 2: - raise ValueError("Bind-C result tuples require at least two elements") - return cls._get_new(element_types) - - @classmethod - @cache - def _get_new(cls, element_types): - def __init__(self): - self._element_types = element_types - Type.__init__(self) - - name = f"BindCResultTuple{len(element_types)}Type" - return type(name, (BindCResultTupleType,), {"__init__": __init__})() - - @property - def element_types(self): - """Types of the packed C-compatible result fields.""" - return self._element_types - - @property - def container_rank(self): - """Rank of the packed result descriptor itself.""" - return 1 - - @property - def rank(self): - """Rank of the packed result descriptor itself.""" - return 1 - - @property - def order(self): - """Memory order is not applicable to the packed result descriptor.""" - return None - - @property - def datatype(self): - """The descriptor is heterogeneous, so its datatype is the descriptor itself.""" - return self - - def shape_is_compatible(self, shape): - """Return whether ``shape`` has one entry with the descriptor field count.""" - return isinstance(shape, tuple) and len(shape) == 1 and shape[0] == len(self) - - def __getitem__(self, index): - return self._element_types[index] - - def __len__(self): - return len(self._element_types) - - def __iter__(self): - return iter(self._element_types) - - -# ======================================================================================= -# Wrapper classes -# ======================================================================================= - - -class BindCFunctionDef(FunctionDef): - """ - Represents the definition of a C-compatible function. - - Contains the C-compatible version of the function which is - used for the wrapper. - As compared to a normal FunctionDef, this version contains - arguments for the shape of arrays. It should be generated by - calling `codegen.wrapper.FortranToCWrapper.wrap`. - - Parameters - ---------- - *args : list - See FunctionDef. - - original_function : FunctionDef - The function from which the C-compatible version was created. - - **kwargs : dict - See FunctionDef. - - See Also - -------- - x2py.ast.core.FunctionDef - The class from which BindCFunctionDef inherits which contains all - details about the args and kwargs. - """ - - __slots__ = ("_original_function",) - _attribute_nodes = (*FunctionDef._attribute_nodes, "_original_function") - - def __init__(self, *args, original_function, **kwargs): - self._original_function = original_function - super().__init__(*args, **kwargs) - assert all(isinstance(a, FunctionDefArgument) for a in self._arguments) - - @property - def original_function(self): - """ - The function which is wrapped by this BindCFunctionDef. - - The original function which would be printed in pure Fortran which is not - compatible with C. - """ - return self._original_function - - def rename(self, newname): - """ - Rename the FunctionDef name->newname. - - Rename the FunctionDef name->newname. - - Parameters - ---------- - newname : str - New name for the FunctionDef. - """ - self._name = newname - - -# ======================================================================================= - - -class BindCVariable(Variable): - """ - A wrapper linking the new C-compatible variable to the original variable. - - A wrapper linking the new C-compatible variable to the variable that is accessible - via this information. This object is a variable which mimics the new variable so - it can be used in some of the same contexts but the underlying variables should be - extracted before manipulating them. - - Parameters - ---------- - new_var : Variable - The new C-compatible variable. - original_var : Variable - The original variable in the target language. - """ - - __slots__ = ("_new_var", "_original_var") - _attribute_nodes = (*Variable._attribute_nodes, "_new_var", "_original_var") - - def __init__(self, new_var, original_var): - self._new_var = new_var - self._original_var = original_var - super().__init__( - new_var.class_type, - new_var.name, - memory_handling=new_var.memory_handling, - is_optional=new_var.is_optional, - shape=new_var.shape, - ownership_decision=getattr(new_var, "ownership_decision", None) - or getattr(original_var, "ownership_decision", None), - ) - - @property - def new_var(self): - """ - The new C-compatible variable. - - The new C-compatible variable. - """ - return self._new_var - - @property - def original_var(self): - """ - The original variable in the target language. - - The original variable from the target language that was wrapped. - """ - return self._original_var - - -# ======================================================================================= -class BindCModule(Module): - """ - Represents a Module which only contains functions compatible with C. - - Represents a Module which provides the C-Fortran interface to another module. - Both functions and module variables are wrapped in order to be compatible with - C. - - Parameters - ---------- - *args : tuple - See `x2py.ast.core.Module`. - - original_module : Module - The Module being wrapped. - - variable_wrappers : list of BindCFunctionDef - A list containing all the functions which expose module variables to C. - - removed_functions : list of FunctionDef - A list of any functions which weren't translated to BindCFunctionDef - objects (e.g. private functions). - - **kwargs : dict - See `x2py.ast.core.Module`. - - See Also - -------- - x2py.ast.core.Module - The class from which BindCModule inherits which contains all details - about the args and kwargs. - """ - - __slots__ = ("_orig_mod", "_removed_functions", "_variable_wrappers") - _attribute_nodes = ( - *Module._attribute_nodes, - "_orig_mod", - "_variable_wrappers", - "_removed_functions", - ) - - def __init__( - self, - *args, - original_module, - variable_wrappers=(), - removed_functions=None, - **kwargs, - ): - self._orig_mod = original_module - self._variable_wrappers = variable_wrappers - self._removed_functions = removed_functions - super().__init__(*args, **kwargs) - - @property - def original_module(self): - """ - The module which was wrapped. - - The original module for which this object provides the C-Fortran interface. - """ - return self._orig_mod - - @property - def variable_wrappers(self): - """ - Get the wrappers which expose module variables to C. - - Get a list containing all the BindCFunctionDefs which expose module variables to C. - """ - return self._variable_wrappers - - @property - def removed_functions(self): - """ - Get the functions which weren't translated to BindCFunctionDef objects. - - Get a list of the functions which weren't translated to BindCFunctionDef objects. - This includes private functions and objects for which wrapper support is lacking. - """ - return self._removed_functions - - @property - def declarations(self): - """ - Get the declarations of all module variables. - - In the case of a BindCModule no variables should be declared. Plain variables - are used directly from the original module and more complex variables require - wrapper functions. - """ - return () - - -# ======================================================================================= - - -class BindCModuleVariable(Variable): - """ - A class which wraps a compatible variable from Fortran to make it available in C. - - A class which wraps a compatible module variable from Fortran to make it available - in C. A compatible variable is a variable which can be exposed to C simply using - iso_c_binding (i.e. no wrapper function is required). - - Parameters - ---------- - *args : tuple - See Variable. - - **kwargs : dict - See Variable. - - See Also - -------- - Variable : The super class. - """ - - __slots__ = () - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - - -class BindCModuleConstant(Variable): - """ - A Python-exported constant that has no mutable native storage. - """ - - __slots__ = () - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - - -class BindCAccessorModuleVariable(Variable): - """ - A module value exposed through generated getter and setter wrappers. - """ - - __slots__ = ("_getter_function", "_setter_function") - _attribute_nodes = ("_getter_function", "_setter_function") - - def __init__(self, *args, getter_function, setter_function, **kwargs): - self._getter_function = getter_function - self._setter_function = setter_function - super().__init__(*args, **kwargs) - - @property - def getter_function(self): - """Generated native getter for this module variable.""" - return self._getter_function - - @property - def setter_function(self): - """Generated native setter for this module variable.""" - return self._setter_function - - -# ======================================================================================= - - -class BindCArrayVariable(Variable): - """ - A class which wraps an array from Fortran to make it available in C. - - A class which wraps an array from Fortran to make it available in C. - - Parameters - ---------- - *args : tuple - See Variable. - - wrapper_function : FunctionDef - The function which can be used to access the array. - - original_variable : Variable - The original variable in the Fortran code. - - **kwargs : dict - See Variable. - - See Also - -------- - Variable : The super class. - """ - - __slots__ = ("_original_variable", "_wrapper_function") - _attribute_nodes = ("_wrapper_function", "_original_variable") - - def __init__(self, *args, wrapper_function, original_variable, **kwargs): - self._original_variable = original_variable - self._wrapper_function = wrapper_function - super().__init__(*args, **kwargs) - - @property - def original_variable(self): - """ - The original variable in the Fortran code. - - The original variable in the Fortran code. This is important in - order to access the correct type and other details about the - Variable. - """ - return self._original_variable - - @property - def wrapper_function(self): - """ - The function which can be used to access the array. - - The function which can be used to access the array. The function - must return the pointer to the raw data and information about - the shape. - """ - return self._wrapper_function - - -# ======================================================================================= - - -class BindCNativeArrayHandleVariable(Variable): - """ - Generated operation wrappers for a borrowed native array handle module variable. - """ - - __slots__ = ("_operation_function_items", "_original_variable") - _attribute_nodes = ("_original_variable",) - - def __init__(self, *args, operation_functions, original_variable, **kwargs): - items = operation_functions.items() if hasattr(operation_functions, "items") else operation_functions - self._operation_function_items = tuple((str(name), function) for name, function in items) - self._original_variable = original_variable - super().__init__(*args, **kwargs) - - @property - def operation_functions(self): - """Generated native operations used by the runtime handle object.""" - return dict(self._operation_function_items) - - @property - def operation_function_items(self): - """Generated native operation functions paired with their runtime operation names.""" - return self._operation_function_items - - @property - def original_variable(self): - """Original module variable exposed as a borrowed runtime handle.""" - return self._original_variable - - -# ======================================================================================= - - -class BindCNativeArrayHandleProperty: - """Generated operation methods for a borrowed native-array field handle.""" - - __slots__ = ( - "_class_type", - "_native_array_handle_policy", - "_operation_function_items", - "_original_variable", - "_owner_class", - "_python_name", - ) - _attribute_nodes = ("_original_variable",) - - def __init__( - self, - python_name, - *, - class_type, - operation_functions, - original_variable, - owner_class, - native_array_handle_policy, - ): - items = operation_functions.items() if hasattr(operation_functions, "items") else operation_functions - self._python_name = str(python_name) - self._class_type = class_type - self._operation_function_items = tuple((str(name), function) for name, function in items) - self._original_variable = original_variable - self._owner_class = owner_class - self._native_array_handle_policy = native_array_handle_policy - init_model_object(self) - - @property - def name(self): - return self._original_variable.name - - @property - def python_name(self): - return self._python_name - - @property - def class_type(self): - return self._class_type - - @property - def dtype(self): - return self._original_variable.dtype - - @property - def rank(self): - return self._original_variable.rank - - @property - def operation_functions(self): - return dict(self._operation_function_items) - - @property - def operation_function_items(self): - return self._operation_function_items - - @property - def original_variable(self): - return self._original_variable - - @property - def owner_class(self): - return self._owner_class - - @property - def native_array_handle_policy(self): - return self._native_array_handle_policy - - -# ======================================================================================= - - -class BindCClassProperty: - """ - A class which wraps a class attribute. - - A class which wraps a class attribute to make it accessible - from C. In the future this class will also be used to handle properties of - classes (i.e. functions marked with the `@property` decorator). - - Parameters - ---------- - python_name : str - The name of the attribute/property in the original Python code. - getter : FunctionDef - The function which collects the value of the class attribute. - setter : FunctionDef - The function which modifies the value of the class attribute. - class_type : Variable - The type of the class to which the attribute belongs. - docstring : Literal, optional - The docstring of the property. - getter_policy : object, optional - Completed policy for the getter result. - setter_policy : object, optional - Completed policy for setter availability and conversion. - """ - - __slots__ = ( - "_class_type", - "_docstring", - "_getter", - "_getter_policy", - "_python_name", - "_setter", - "_setter_policy", - ) - _attribute_nodes = ("_getter", "_setter") - - def __init__( - self, - python_name, - getter, - setter, - class_type, - docstring=None, - *, - getter_policy=None, - setter_policy=None, - ): - assert isinstance(getter, BindCFunctionDef) - assert isinstance(setter, BindCFunctionDef) or setter is None - self._python_name = python_name - self._getter = getter - self._setter = setter - self._getter_policy = getter_policy - self._setter_policy = setter_policy - self._class_type = class_type - self._docstring = docstring - init_model_object(self) - - @property - def getter(self): - """ - The BindCFunctionDef describing the getter function. - - The BindCFunctionDef describing the function which allows the user to collect - the value of the property. - """ - return self._getter - - @property - def getter_policy(self): - """Return the completed policy for reading this property.""" - return self._getter_policy - - @property - def setter(self): - """ - The BindCFunctionDef describing the setter function. - - The BindCFunctionDef describing the function which allows the user to modify - the value of the property. - """ - return self._setter - - @property - def setter_policy(self): - """Return the completed policy for writing this property.""" - return self._setter_policy - - @property - def class_type(self): - """ - The type of the class to which the attribute belongs. - - The type of the class to which the attribute belongs. - """ - return self._class_type - - @property - def python_name(self): - """ - The name of the attribute/property in the original Python code. - - The name of the attribute/property in the original Python code. - """ - return self._python_name - - @property - def docstring(self): - """ - The docstring of the property being wrapped. - - The docstring of the property being wrapped. - """ - return self._docstring - - -# ======================================================================================= - - -class BindCClassDef(ClassDef): - """ - Represents a class which is compatible with C. - - Represents a class which is compatible with C. This means that it stores - C-compatible versions of class methods and getters and setters for class - variables. - - Parameters - ---------- - original_class : ClassDef - The class being wrapped. - - new_func : BindCFunctionDef - The function which provides a new instance of the class. - - **kwargs : dict - See ClassDef. - """ - - __slots__ = ("_new_func", "_original_class") - - def __init__(self, original_class, new_func, **kwargs): - self._original_class = original_class - self._new_func = new_func - super().__init__(original_class.name, scope=original_class.scope, **kwargs) - - @property - def new_func(self): - """ - Get the wrapper for `__new__`. - - Get the wrapper for `__new__` which allocates the memory for the class instance. - """ - return self._new_func - - @property - def original_class(self): - """Return the source class wrapped by this Bind-C class.""" - return self._original_class - - -# ======================================================================================= -# Utility functions -# ======================================================================================= - - -class CLocFunc: - """ - Creates a C-compatible pointer to the argument. - - Class representing the iso_c_binding function cloc which returns a valid - C pointer to the location where an object can be found. - - Parameters - ---------- - argument : Variable - The object which should be pointed to. - - result : Variable of dtype BindCPointer - The variable where the C-compatible pointer should be stored. - """ - - __slots__ = ("_arg", "_result") - _attribute_nodes = () - - def __init__(self, argument, result): - self._arg = argument - self._result = result - assert result.dtype is BindCPointer() - init_model_object(self) - - @property - def arg(self): - """ - Pointer target. - - Object which will be pointed at by the result pointer. - """ - return self._arg - - @property - def result(self): - """ - The variable where the C-compatible pointer should be stored. - - The variable where the C-compatible pointer of dtype BindCPointer - should be stored. - """ - return self._result - - -# ======================================================================================= - - -class C_F_Pointer: - """ - Creates a Fortran array pointer from a C pointer and size information. - - Represents the iso_c_binding function C_F_Pointer which takes a pointer - to an object in C (with dtype BindCPointer) and a list of sizes and returns - a Fortran array pointer. - - Parameters - ---------- - c_expr : Variable of dtype BindCPointer - The Variable containing the C pointer. - - f_expr : Variable - The Variable containing the resulting array. - - shape : list of Variables - A list describing the Variables which dictate the size of the array in each dimension. - """ - - __slots__ = ("_c_expr", "_f_expr", "_shape") - _attribute_nodes = ("_c_expr", "_f_expr", "_shape") - - def __init__(self, c_expr, f_expr, shape=None): - self._c_expr = c_expr - self._f_expr = f_expr - self._shape = shape - init_model_object(self) - - @property - def c_pointer(self): - """ - The Variable containing the C pointer. - - The Variable of dtype BindCPointer which contains the C pointer. - """ - return self._c_expr - - @property - def f_array(self): - """ - The Variable containing the resulting array. - - The Variable where the array pointer will be stored. - """ - return self._f_expr - - @property - def shape(self): - """ - A list of the sizes of the array in each dimension. - - A list describing the Variables which are passed as arguments, in order to - determine the size of the array in each dimension. - """ - return self._shape - - -class DeallocatePointer(Deallocate): - """ - Represents memory deallocation for memory only stored in a pointer. - - Represents memory deallocation for memory only stored in a pointer. Usually - `deallocate` is not called on pointers so as not to delete the target values - however this capability is necessary in the wrapper. - - Parameters - ---------- - variable : x2py.ast.core.Variable - The typed variable (usually an array) that needs memory deallocation. - """ - - __slots__ = () - - -class BindCSizeOf(Function): - """ - Represents a call to a function which can calculate the size of an object in bits. - - Represents a call to a function which can calculate the size of an object in bits. - - Parameters - ---------- - element : model object - The object whose type should be determined. - """ - - __slots__ = () - _class_type = NumpyInt64Type() - _shape = None - - def __init__(self, element): - super().__init__(element) - - -class FortranTransfer(Function): - """Represent the Fortran ``transfer(source, mold[, size])`` intrinsic.""" - - __slots__ = ("_class_type", "_shape") - - def __init__(self, source, mold, size=None): - self._class_type = mold.class_type - self._shape = mold.shape - args = (source, mold) if size is None else (source, mold, size) - super().__init__(*args) - - @property - def source(self): - return self.args[0] - - @property - def mold(self): - return self.args[1] - - @property - def size(self): - return self.args[2] if len(self.args) == 3 else None - - -class C_NULL_CHAR: - """ - A class representing the C_NULL_CHAR character from the iso_c_binding module. - - A class representing the C_NULL_CHAR character from the iso_c_binding module. - This object should be appended to strings before returning them from Fortran - to C. - """ - - __slots__ = () - _class_type = StringType() - _shape = (convert_to_literal(1),) - _attribute_nodes = () - - def __init__(self): - init_model_object(self) - - -c_malloc = FunctionDef( - "c_malloc", - (FunctionDefArgument(Variable(NumpyInt64Type(), "size")),), - (), - FunctionDefResult(Variable(BindCPointer(), "ptr")), -) - - -for _model_cls in ( - BindCClassProperty, - BindCNativeArrayHandleProperty, - BindCNativeArrayHandleVariable, - CLocFunc, - C_F_Pointer, - C_NULL_CHAR, - FortranTransfer, -): - register_model_class(_model_cls) - -del _model_cls diff --git a/x2py/codegen/binding_pipeline.py b/x2py/codegen/binding_pipeline.py deleted file mode 100644 index 20e7ea8e4..000000000 --- a/x2py/codegen/binding_pipeline.py +++ /dev/null @@ -1,151 +0,0 @@ -""" -Module containing the BindingPipeline class. - -This module coordinates the generation of bridge and binding files required -to expose generated code to Python. -""" - -from pathlib import Path - -from .models.core import ModuleHeader -from .printers.cpythoncode import CPythonCodePrinter -from .printers.fcode import FCodePrinter -from .bindings.c_to_python import CPythonBindingGenerator -from .bridges.fortran_to_c import FortranToCBridgeGenerator - -_EXTENSIONS = {"fortran": "f90", "c": "c"} -_HEADER_EXTENSIONS = {"fortran": None, "c": "h"} - -binding_pipeline_registry = { - "fortran": [FortranToCBridgeGenerator, CPythonBindingGenerator], -} - -printer_registry = { - FortranToCBridgeGenerator: FCodePrinter, - CPythonBindingGenerator: CPythonCodePrinter, -} - - -class BindingPipeline: - """ - Pipeline responsible for generating bridge and binding files. - - Parameters - ---------- - codegen : Codegen - The code generator which produced the translated AST. - name : str - Name of the generated module or program. - language : str - Source language accepted by the runtime wrapper pipeline. - verbose : int - The level of verbosity. - """ - - def __init__(self, codegen, name, language, verbose): - self._ast = codegen.ast - self._name = name - self._verbose = verbose - self._generated_asts = [] - - self._pipeline_steps = binding_pipeline_registry[language] - self._printer_types = [printer_registry[w] for w in self._pipeline_steps] - self._additional_imports = [{} for _ in self._pipeline_steps] - - def generate(self, sharedlib_dirpath): - """ - Generate the bridge and binding ASTs. - - Run each step of the binding pipeline in order. Each step receives the AST - generated by the previous step and returns a new AST. - - - Parameters - ---------- - sharedlib_dirpath : str - The folder where the generated .so file will be located. - """ - ast = self._ast - for Step in self._pipeline_steps: - if self._verbose: - print( - f">> Building {Step.start_language}-{Step.target_language} interface :: ", - self._name, - ) - - step = Step(sharedlib_dirpath, verbose=self._verbose) - - ast = step.generate(ast) - self._generated_asts.append(ast) - - def write(self, dirpath): - """ - Write the generated bridge and binding source files. - - Write the AST objects generated by a call to generate(). - - Parameters - ---------- - dirpath : str | Path - The path to the directory where files should be printed. - - Returns - ------- - list[Path] - A list of the source files printed by this function (this is not equivalent - to all files printed by this function as headers are excluded). - """ - dirpath = Path(dirpath) - files = [ - dirpath / f"{ast.name}_wrapper.{_EXTENSIONS[Step.start_language.lower()]}" - for ast, Step in zip(self._generated_asts, self._pipeline_steps, strict=False) - ] - for i, (filepath, ast, Printer) in enumerate( - zip(files, self._generated_asts, self._printer_types, strict=False) - ): - header_ext = _HEADER_EXTENSIONS[Printer.language.lower()] - - if self._verbose: - print(">>> Printing :: ", filepath) - printer = Printer(ast.name, verbose=self._verbose) - # print module - code = printer.doprint(ast) - - with open(filepath, "w", encoding="utf-8") as f: - f.write(code) - - # print module header - if header_ext is not None: - header_filename = dirpath / f"{ast.name}_wrapper.{header_ext}" - module_header = ModuleHeader(ast) - if self._verbose: - print(">>> Printing :: ", header_filename) - code = printer.doprint(module_header) - with open(header_filename, "w", encoding="utf-8") as f: - f.write(code) - - self._additional_imports[i] = printer.get_additional_imports().copy() - - return files - - def get_additional_imports(self): - """ - Get the objects that were imported by the codeprinters. - - Get the objects that were imported by the codeprinters. - These imports may affect the necessary compiler commands. - - Returns - ------- - list[dict[str, Import]] - A dictionary for each printed wrapper file, - mapping the include strings to the import module. - """ - return self._additional_imports - - @property - def generated_languages(self): - """ - Get the languages of the generated bridge and binding files. - """ - return [Printer.language.lower() for Printer in self._printer_types] diff --git a/x2py/codegen/bindings/c_concepts.py b/x2py/codegen/bindings/c_concepts.py deleted file mode 100644 index 3fdddb1eb..000000000 --- a/x2py/codegen/bindings/c_concepts.py +++ /dev/null @@ -1,512 +0,0 @@ -""" -Module representing concepts that are only applicable to C code (e.g. ObjectAddress). -""" - -from functools import cache - -from ..models.datatypes import ( - CharType, - convert_to_literal, - FixedSizeType, - FixedSizeNumericType, - init_model_object, - is_model_object, - PrimitiveIntegerType, - register_model_class, - Type, -) -from ..models.core import Function - -__all__ = ( - "CFIDescriptorAllocate", - "CFIDescriptorDeallocate", - "CFIDescriptorDimField", - "CFIDescriptorEstablish", - "CFIDescriptorField", - "CFIDescriptorStorageSize", - "CFIDescriptorStorageType", - "CFIDescriptorType", - "CFIDimensionType", - "CNativeInt", - "CStrStr", - "ObjectAddress", - "PointerCast", -) - -# ------------------------------------------------------------------------------ - - -class CNativeInt(FixedSizeNumericType): - """ - Class representing C's native integer type. - - Class representing C's native integer type. - """ - - __slots__ = () - _name = "int" - _primitive_type = PrimitiveIntegerType() - _precision = None - - -# ------------------------------------------------------------------------------ -class CFIDescriptorType(FixedSizeType): - """TS 29113 ``CFI_cdesc_t`` descriptor record.""" - - __slots__ = () - _name = "CFI_cdesc_t" - - -class CFIDescriptorStorageType(Type): - """Rank-specific storage emitted with the standard ``CFI_CDESC_T`` macro.""" - - __slots__ = ("_rank",) - _name = "CFI_CDESC_T" - - @classmethod - def get_new(cls, rank): - """Return cached descriptor storage for one non-negative rank.""" - rank = int(rank) - if rank < 0: - raise ValueError("CFI descriptor storage rank must be non-negative") - return cls._get_new(rank) - - @property - def rank(self): - """Descriptor storage is a scalar C object.""" - return 0 - - @property - def descriptor_rank(self): - """Descriptor rank reserved by this storage type.""" - return self._rank - - @property - def container_rank(self): - """Descriptor storage is a scalar C object.""" - return 0 - - @property - def order(self): - """Descriptor storage has no array order.""" - return None - - @property - def datatype(self): - """Descriptor storage is its own datatype.""" - return self - - @classmethod - @cache - def _get_new(cls, rank): - """Create the cached rank-specific descriptor storage type.""" - - def __init__(self): - self._rank = rank - Type.__init__(self) - - return type(f"CFIDescriptorStorage{rank}DType", (CFIDescriptorStorageType,), {"__init__": __init__})() - - -class CFIDimensionType(FixedSizeType): - """TS 29113 ``CFI_dim_t`` dimension record.""" - - __slots__ = () - _name = "CFI_dim_t" - - -class CFIDescriptorField: - """Typed field access on a ``CFI_cdesc_t*`` descriptor pointer.""" - - __slots__ = ("_class_type", "_field", "_owner", "_shape") - _attribute_nodes = ("_owner",) - - def __init__(self, owner, field, dtype): - """Initialize one descriptor field expression.""" - if not is_model_object(owner): - raise TypeError("owner must be a model object") - if not isinstance(field, str) or not field: - raise TypeError("field must be a non-empty string") - if not isinstance(dtype, Type): - raise TypeError("dtype must be a codegen Type") - self._owner = owner - self._field = field - self._class_type = dtype - self._shape = None - init_model_object(self) - - @property - def owner(self): - """Descriptor pointer whose field is read.""" - return self._owner - - @property - def field(self): - """Descriptor field name.""" - return self._field - - -class CFIDescriptorDimField: - """Typed field access on ``CFI_cdesc_t.dim[index]``.""" - - __slots__ = ("_class_type", "_field", "_index", "_owner", "_shape") - _attribute_nodes = ("_owner", "_index") - - def __init__(self, owner, index, field, dtype): - """Initialize one descriptor dimension field expression.""" - if not is_model_object(owner): - raise TypeError("owner must be a model object") - if isinstance(index, int): - index = convert_to_literal(index) - if not is_model_object(index): - raise TypeError("index must be an integer or model object") - if not isinstance(field, str) or not field: - raise TypeError("field must be a non-empty string") - if not isinstance(dtype, Type): - raise TypeError("dtype must be a codegen Type") - self._owner = owner - self._index = index - self._field = field - self._class_type = dtype - self._shape = None - init_model_object(self) - - @property - def owner(self): - """Descriptor pointer whose dimension field is read.""" - return self._owner - - @property - def index(self): - """Zero-based descriptor dimension index.""" - return self._index - - @property - def field(self): - """Descriptor dimension field name.""" - return self._field - - -class CFIDescriptorEstablish: - """Standard ``CFI_establish`` call for an initially disassociated pointer.""" - - __slots__ = ( - "_attribute", - "_base_address", - "_class_type", - "_descriptor", - "_element_length", - "_element_type", - "_extents", - "_rank", - "_shape", - ) - _attribute_nodes = ("_base_address", "_descriptor", "_element_length", "_extents") - - def __init__( - self, - descriptor, - element_type, - rank, - *, - base_address=None, - attribute="pointer", - element_length=None, - extents=(), - ): - """Initialize one standard descriptor-establishment expression.""" - if not is_model_object(descriptor): - raise TypeError("descriptor must be a model object") - if not isinstance(element_type, Type): - raise TypeError("element_type must be a codegen Type") - rank = int(rank) - if rank < 0: - raise ValueError("CFI descriptor rank must be non-negative") - if attribute not in {"allocatable", "other", "pointer"}: - raise ValueError("CFI descriptor attribute must be allocatable, other, or pointer") - extents = tuple(extents) - if extents and len(extents) != rank: - raise ValueError("CFI descriptor extents must match the declared rank") - self._descriptor = descriptor - self._element_type = element_type - self._rank = rank - self._base_address = base_address - self._attribute = attribute - self._element_length = element_length - self._extents = extents - self._class_type = CNativeInt() - self._shape = None - init_model_object(self) - - @property - def descriptor(self): - """Pointer to the descriptor record being established.""" - return self._descriptor - - @property - def element_type(self): - """Declared array element type.""" - return self._element_type - - @property - def rank(self): - """Declared descriptor rank.""" - return self._rank - - @property - def base_address(self): - """Optional native data address used to establish the descriptor.""" - return self._base_address - - @property - def attribute(self): - """Standard CFI descriptor attribute spelling.""" - return self._attribute - - @property - def element_length(self): - """Optional runtime element length expression.""" - return self._element_length - - @property - def extents(self): - """Runtime extent expressions used for an associated descriptor.""" - return self._extents - - -class CFIDescriptorAllocate: - """Standard ``CFI_allocate`` call for persistent allocatable storage.""" - - __slots__ = ("_class_type", "_descriptor", "_element_length", "_lower_bounds", "_shape", "_upper_bounds") - _attribute_nodes = ("_descriptor", "_element_length", "_lower_bounds", "_upper_bounds") - - def __init__(self, descriptor, lower_bounds, upper_bounds, element_length): - """Initialize one persistent descriptor-allocation expression.""" - lower_bounds = tuple(lower_bounds) - upper_bounds = tuple(upper_bounds) - if not is_model_object(descriptor): - raise TypeError("descriptor must be a model object") - if len(lower_bounds) != len(upper_bounds): - raise ValueError("CFI allocation lower and upper bounds must have equal rank") - if not all(is_model_object(bound) for bound in (*lower_bounds, *upper_bounds)): - raise TypeError("CFI allocation bounds must be model objects") - if not is_model_object(element_length): - raise TypeError("CFI allocation element length must be a model object") - self._descriptor = descriptor - self._lower_bounds = lower_bounds - self._upper_bounds = upper_bounds - self._element_length = element_length - self._class_type = CNativeInt() - self._shape = None - init_model_object(self) - - @property - def descriptor(self): - """Descriptor pointer whose payload is allocated.""" - return self._descriptor - - @property - def lower_bounds(self): - """Inclusive lower bounds passed to ``CFI_allocate``.""" - return self._lower_bounds - - @property - def upper_bounds(self): - """Inclusive upper bounds passed to ``CFI_allocate``.""" - return self._upper_bounds - - @property - def element_length(self): - """Runtime element length passed to ``CFI_allocate``.""" - return self._element_length - - -class CFIDescriptorDeallocate: - """Standard ``CFI_deallocate`` call for persistent allocatable storage.""" - - __slots__ = ("_class_type", "_descriptor", "_shape") - _attribute_nodes = ("_descriptor",) - - def __init__(self, descriptor): - """Initialize one persistent descriptor-deallocation expression.""" - if not is_model_object(descriptor): - raise TypeError("descriptor must be a model object") - self._descriptor = descriptor - self._class_type = CNativeInt() - self._shape = None - init_model_object(self) - - @property - def descriptor(self): - """Descriptor pointer whose payload is deallocated.""" - return self._descriptor - - -class CFIDescriptorStorageSize: - """Size in bytes of rank-specific ``CFI_CDESC_T`` storage.""" - - __slots__ = ("_class_type", "_rank", "_shape") - _attribute_nodes = () - - def __init__(self, rank): - """Initialize storage-size lookup for one descriptor rank.""" - rank = int(rank) - if rank < 0: - raise ValueError("CFI descriptor storage rank must be non-negative") - self._rank = rank - self._class_type = CNativeInt() - self._shape = None - init_model_object(self) - - @property - def rank(self): - """Descriptor rank whose storage size is requested.""" - return self._rank - - -# ------------------------------------------------------------------------------ -class ObjectAddress: - """ - Class representing the address of an object. - - Class representing the address of an object. In most situations it will not be - necessary to use this object explicitly. E.g. if you assign a pointer to a - target then the pointer will be printed using `AliasAssign`. However for the - `_visit_AliasAssign` function to print neatly, this class will be used. - - Parameters - ---------- - obj : model object - The object whose address should be printed. - - Examples - -------- - >>> CCodePrinter._visit(ObjectAddress(Variable(NumpyInt64Type(),'a'))) - '&a' - >>> CCodePrinter._visit(ObjectAddress(Variable(NumpyInt64Type(),'a', memory_handling='alias'))) - 'a' - """ - - __slots__ = ("_class_type", "_obj", "_shape") - _attribute_nodes = ("_obj",) - - def __init__(self, obj): - """Initialize one ``ObjectAddress`` model instance.""" - if not is_model_object(obj): - raise TypeError("object must be a model object") - self._obj = obj - self._shape = obj.shape - self._class_type = obj.class_type - init_model_object(self) - - @property - def obj(self): - """The object whose address is of interest""" - return self._obj - - @property - def is_alias(self): - """ - Indicate that an ObjectAddress uses alias memory handling. - - Indicate that an ObjectAddress uses alias memory handling. - """ - return True - - -# ------------------------------------------------------------------------------ -class PointerCast: - """ - A class which represents the casting of one pointer to another. - - A class which represents the casting of one pointer to another in C code. - This is useful for storing addresses in a void pointer. - Using this class is not strictly necessary to produce correct C code, - but avoids compiler warnings about the implicit conversion of pointers. - - Parameters - ---------- - obj : Variable - The pointer being cast. - cast_type : model object - A model object describing the object resulting from the cast. - """ - - __slots__ = ("_cast_type", "_class_type", "_obj", "_shape") - _attribute_nodes = ("_obj",) - - def __init__(self, obj, cast_type): - """Initialize one ``PointerCast`` model instance.""" - if not is_model_object(obj): - raise TypeError("object must be a model object") - assert getattr(obj, "is_alias", False) - self._obj = obj - self._shape = cast_type.shape - self._class_type = cast_type.class_type - self._cast_type = cast_type - init_model_object(self) - - @property - def obj(self): - """ - The object whose address is of interest. - - The object whose address is of interest. - """ - return self._obj - - @property - def cast_type(self): - """ - Get the model object which describes the object resulting from the cast. - - Get the model object which describes the object resulting from the cast. - """ - return self._cast_type - - @property - def is_argument(self): - """ - Indicates whether the variable is an argument. - - Indicates whether the variable is an argument. - """ - return self._obj.is_argument - - -class CStrStr(Function): - """ - A class which extracts a const char* from a literal string. - - A class which extracts a const char* from a literal string. This - is useful for calling C functions which were not designed for - STC. - - Parameters - ---------- - arg : model object - The object which should be passed as a const char*. - """ - - __slots__ = () - _class_type = CharType() - _shape = (None,) - - def __init__(self, arg): - """Initialize one ``CStrStr`` model instance.""" - super().__init__(arg) - - -for _model_cls in ( - CFIDescriptorAllocate, - CFIDescriptorDeallocate, - CFIDescriptorField, - CFIDescriptorDimField, - CFIDescriptorEstablish, - CFIDescriptorStorageSize, - ObjectAddress, - PointerCast, -): - register_model_class(_model_cls) - -del _model_cls diff --git a/x2py/codegen/bindings/c_to_python.py b/x2py/codegen/bindings/c_to_python.py deleted file mode 100644 index 93d73a0c0..000000000 --- a/x2py/codegen/bindings/c_to_python.py +++ /dev/null @@ -1,7999 +0,0 @@ -""" -Module describing the code-wrapping class : CToPythonWrapper -which creates an interface exposing C code to Python. -""" - -import ast -from functools import reduce - -from x2py.semantics.ownership import ( - CodegenAction, - DestructionPolicy, - DestructionPolicyDispatcher, - NativeBarrierAction, - ObjectKind, - PolicyActionDispatcher, - PolicyProjectionDispatcher, - PythonBarrierAction, - PythonBarrierDispatcher, - SetterAction, - SetterActionDispatcher, - StorageMode, - ownership_decision_for_codegen_variable, -) -from x2py.semantics.metadata import SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA -from x2py.semantics.models import ( - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA, - INTERNAL_MODULE_VARIABLE_NAME_METADATA, - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA, - INTERNAL_NATIVE_ARRAY_HANDLE_OWNER_CLASS_METADATA, - RUNTIME_HOLD_GIL_METADATA, - RUNTIME_RETAIN_RESULT_OWNER_METADATA, - RUNTIME_STATUS_ERROR_METADATA, -) -from x2py.semantics.native_array_handles import ( - ArrayInteropPolicyDispatcher, - NativeArrayHandlePolicyDispatcher, - NativeArrayOutputProjectionDispatcher, -) -from x2py.semantics.wrapper_policy import NativeStatusErrorPolicy, PythonExceptionKind - -from ..bind_c import ( - BindCArrayVariable, - BindCArrayType, - BindCClassProperty, - BindCFunctionDef, - BindCModuleVariable, - BindCNativeArrayHandleProperty, - BindCNativeArrayHandleVariable, - BindCPointer, - BindCResultTupleType, - BindCScalarDescriptorType, - BindCVariable, - native_array_descriptor_argument_type, -) -from ..models.core import PythonTuple -from .c_concepts import ( - CFIDescriptorAllocate, - CFIDescriptorDeallocate, - CFIDescriptorDimField, - CFIDescriptorEstablish, - CFIDescriptorField, - CFIDescriptorStorageSize, - CFIDescriptorStorageType, - CFIDescriptorType, - CNativeInt, - CStrStr, - ObjectAddress, - PointerCast, -) -from ..models.core import ( - AliasAssign, - Allocate, - AsName, - Assign, - AugAssign, - ClassDef, - CommentBlock, - Deallocate, - Declare, - FunctionAddress, - FunctionCall, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - get_enclosing_class, - If, - IfSection, - Import, - is_in_overload_set, - Module, - Return, -) -from .cpython_api import ( - C_to_Python, - Py_DECREF, - Py_INCREF, - Py_None, - Py_ssize_t, - PyArg_ParseTupleNode, - PyArgKeywords, - PyArgumentError, - PyAllowThreadsBegin, - PyAllowThreadsEnd, - PyAttributeError, - PyBuildValueNode, - PyCallbackContextPop, - PyCallbackContextPush, - PyCallbackValidate, - PyCapsule_Import, - PyCapsule_New, - PythonObjectType, - PythonTypeObjectType, - PyClassDef, - PyDict_New, - PyDict_SetItem, - PyErr_Occurred, - PyErr_SetString, - PyErr_SetObject, - PyFunctionDef, - PyGetSetDefElement, - PyFunctionOverloadSet, - PyImport_ImportModule, - PyList_Append, - PyList_GetItem, - PyList_New, - PyList_SetItem, - PyLong_AsLongLong, - PyLong_AsVoidPtr, - PyLong_Check, - PyLong_FromLong, - PyLong_FromLongLong, - PyLong_FromVoidPtr, - PyMemoryError, - PyModInitFunc, - PyModule, - PyModule_AddObject, - PyModule_Create, - PyModule_SetPropertyType, - PyObject_CallObject, - PyObject_GetAttrString, - PyRuntimeError, - PyObject_TypeCheck, - PySys_GetObject, - PyTuple_Pack, - PyTuple_GetItem, - PyType_Ready, - PyTypeError, - PyUnicode_AsUTF8, - PyUnicode_AsUTF8AndSize, - PyUnicode_Check, - PyUnicode_FromString, - WrapperCustomDataType, - check_type_registry, - c_memcpy, - c_memset, - c_strlen, - py_to_c_registry, - x2py_malloc, -) -from ..models.datatypes import ( - CharType, - CustomDataType, - DataTypeFactory, - FinalType, - FixedSizeNumericType, - NumpyBoolType, - PrimitiveComplexType, - PrimitiveFloatingPointType, - PrimitiveIntegerType, - StringType, - TupleType, - VoidType, - cast_to, - NIL, - convert_to_literal, -) -from ..models.core import Slice -from .numpy_cpython_api import ( - PyArray_Check, - PyArray_DATA, - PyArray_CHKFLAGS, - PyArray_ISNOTSWAPPED, - PyArray_ITEMSIZE, - PyArray_NDIM, - PyArray_SetBaseObject, - PyArray_TYPE, - NumpyArrayObjectType, - get_strides_and_shape_from_numpy_array, - import_array, - is_numpy_array, - no_order_check, - numpy_dtype_registry, - numpy_flag_aligned, - numpy_flag_c_contig, - numpy_flag_f_contig, - numpy_flag_writeable, - numpy_string_type, - pyarray_check, - require_any_contiguous, - require_c_contiguous, - require_f_contiguous, - to_numpy_bytes_array, - to_pyarray, -) -from ..models.datatypes import ( - NumpyInt32Type, - NumpyInt64Type, - NumpyNDArrayType, -) -from ..models.core import ( - Add, - And, - IfTernaryOperator, - Eq, - Ge, - Is, - IsNot, - Le, - Lt, - Minus, - Mul, - Ne, - Not, - Or, -) -from ..models.core import DottedVariable, IndexedElement, Variable -from ..scope import Scope - -from ..generator import BindingGenerator - -cpython_ndarray_imports = [ - Import("python_runtime_ndarrays", Module("python_runtime_ndarrays", (), ())), - Import("ndarrays", Module("ndarrays", (), ())), -] -_MAX_SUPPORTED_ASSUMED_RANK = 15 - -StackArrayClass = ClassDef("stack_array") - -magic_binary_funcs = ( - "__add__", - "__sub__", - "__mul__", - "__truediv__", - "__pow__", - "__lshift__", - "__rshift__", - "__and__", - "__or__", - "__iadd__", - "__isub__", - "__imul__", - "__itruediv__", - "__ipow__", - "__ilshift__", - "__irshift__", - "__iand__", - "__ior__", - "__getitem__", -) -magic_unary_funcs = ("__pos__", "__neg__", "__invert__") -magic_comparison_funcs = ("__eq__", "__ne__", "__lt__", "__le__", "__gt__", "__ge__") -magic_overload_funcs = (*magic_binary_funcs, *magic_unary_funcs, *magic_comparison_funcs) - - -class CPythonBindingGenerator(BindingGenerator): - """Create a Python-compatible binding AST for a C module. - - The class follows the same reading order as ``FortranParser``: - - - public generation entrypoint inherited from ``BindingGenerator``; - - module, function, argument, variable, and class visitors; - - Python argument conversion helpers; - - Python result conversion helpers; - - documentation, ownership, and low-level validation helpers. - - Model-node dispatch remains exclusively owned by ``_visit``. Datatype and - ownership conversions use explicit secondary dispatch tables. - - Parameters - ---------- - sharedlib_dirpath : str - The folder where the generated .so file will be located. - verbose : int - The level of verbosity. - """ - - target_language = "Python" - start_language = "C" - _PYTHON_BARRIER_DISPATCHER = PythonBarrierDispatcher( - { - PythonBarrierAction.SCALAR_VALUE: "_convert_python_scalar_value_argument", - PythonBarrierAction.SCALAR_STORAGE: "_convert_python_scalar_storage_argument", - PythonBarrierAction.ARRAY_STORAGE: "_convert_python_array_storage_argument", - PythonBarrierAction.STRING_VALUE: "_convert_python_string_value_argument", - PythonBarrierAction.STRING_STORAGE: "_convert_python_string_storage_argument", - PythonBarrierAction.RAW_ADDRESS: "_convert_python_raw_address_argument", - PythonBarrierAction.WRAPPER_INSTANCE: "_convert_python_wrapper_instance_argument", - } - ) - _ARGUMENT_DETAIL_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_direct_argument_detail_lines", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT): "_call_local_argument_detail_lines", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_in_place_argument_detail_lines", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_identity_output_detail_lines", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_replacement_value_detail_lines", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT): "_call_local_argument_detail_lines", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT): "_in_place_argument_detail_lines", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT): "_discarded_identity_output_detail_lines", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_replacement_value_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_call_local_argument_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_in_place_argument_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_identity_output_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_replacement_array_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT): "_call_local_argument_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_in_place_argument_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_identity_output_detail_lines", - } - ) - _ARGUMENT_CAST_GUARD_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_append_checked_argument_cast", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT): "_append_checked_argument_cast", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_append_checked_argument_cast", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_append_unchecked_argument_cast", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_append_replacement_argument_cast", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT): "_append_checked_argument_cast", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT): "_append_checked_argument_cast", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT): "_append_checked_argument_cast", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_append_replacement_argument_cast", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_append_checked_argument_cast", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_append_checked_argument_cast", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_append_checked_argument_cast", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_append_replacement_argument_cast", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT): "_append_checked_argument_cast", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_append_checked_argument_cast", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_append_checked_argument_cast", - } - ) - _RESULT_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_convert_snapshot_policy_scalar_result", - (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_convert_policy_scalar_result", - (ObjectKind.STRING, CodegenAction.COPY_OUT): "_convert_policy_string_result", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_policy_string_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_convert_policy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_policy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_convert_policy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_convert_policy_array_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_convert_policy_custom_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_convert_policy_custom_result", - } - ) - _RESULT_DETAIL_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_default_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_detail_lines", - (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_default_result_detail_lines", - (ObjectKind.STRING, CodegenAction.COPY_OUT): "_default_result_detail_lines", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_default_result_detail_lines", - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_default_result_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_default_result_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_default_result_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_default_result_detail_lines", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_default_result_detail_lines", - } - ) - _RESULT_NOTE_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_empty_result_notes", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_empty_result_notes", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_notes", - (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_empty_result_notes", - (ObjectKind.STRING, CodegenAction.COPY_OUT): "_empty_result_notes", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_empty_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_copy_return_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_empty_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_copy_return_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_snapshot_copy_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_borrowed_view_result_notes", - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_copy_return_result_notes", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_empty_result_notes", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_empty_result_notes", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_empty_result_notes", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_empty_result_notes", - } - ) - _PROPERTY_SETTER_POLICY_DISPATCHER = SetterActionDispatcher( - { - SetterAction.WRITE_THROUGH: "_build_writable_property_setter", - SetterAction.REJECT_REPLACEMENT: "_build_blocked_property_setter", - } - ) - _BORROWED_GETTER_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_incref_borrowed_array_getter", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_incref_borrowed_custom_getter", - } - ) - _NATIVE_ARRAY_HANDLE_DISPATCHER = NativeArrayHandlePolicyDispatcher( - { - ("allocatable", "argument_descriptor"): "_bind_allocatable_descriptor_argument", - ("allocatable", "borrowed_field_descriptor"): "_bind_borrowed_native_array_field_handle", - ("allocatable", "borrowed_module_descriptor"): "_bind_borrowed_native_array_module_handle", - ("allocatable", "optional_absent_handle"): "_bind_optional_native_array_handle", - ("allocatable", "owned_result_descriptor"): "_bind_owned_allocatable_result_handle", - ("pointer", "argument_descriptor"): "_bind_pointer_descriptor_argument", - ("pointer", "borrowed_field_descriptor"): "_bind_borrowed_native_array_field_handle", - ("pointer", "borrowed_module_descriptor"): "_bind_borrowed_native_array_module_handle", - ("pointer", "optional_absent_handle"): "_bind_optional_native_array_handle", - } - ) - _NATIVE_ARRAY_DESCRIPTOR_RESULT_DISPATCHER = PolicyProjectionDispatcher( - { - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True): ( - "_bind_projected_native_array_handle_result" - ), - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE, True): ( - "_bind_materialized_native_array_handle_result" - ), - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE, False): ( - "_bind_materialized_native_array_handle_result" - ), - } - ) - _NATIVE_ARRAY_DESCRIPTOR_ARGUMENT_DISPATCHER = NativeArrayOutputProjectionDispatcher( - { - "none": "_bind_fact_packed_native_array_descriptor_argument", - "projected_handle": "_bind_direct_native_array_descriptor_argument", - } - ) - _ARRAY_INTEROP_POLICY_DISPATCHER = ArrayInteropPolicyDispatcher( - { - ("argument", "data_buffer"): "_bind_data_buffer_argument", - ("argument", "descriptor"): "_bind_descriptor_argument", - ("result", "data_buffer"): "_bind_data_buffer_result", - ("result", "descriptor"): "_bind_descriptor_result", - } - ) - _ARGUMENT_RETURN_PROJECTION_DISPATCHER = PolicyProjectionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE, False): "_skip_argument_return_projection", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE, True): "_project_native_argument_return", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.STRING, CodegenAction.COPY_OUT, True): "_project_native_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, True): "_project_visible_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT, True): "_project_native_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT, True): "_project_native_argument_return", - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE, True): "_project_native_argument_return", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_argument_return_projection", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_argument_return_projection", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, False): "_skip_argument_return_projection", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, True): "_project_visible_argument_return", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, True): "_project_visible_argument_return", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE, True): "_project_native_argument_return", - } - ) - _PROJECTED_ARGUMENT_OBJECT_DISPATCHER = PolicyProjectionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE, False): "_skip_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT, False): "_skip_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT, True): "_record_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT, True): "_record_projected_argument_object", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT, True): "_skip_projected_argument_object", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_projected_argument_object", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_projected_argument_object", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, False): "_skip_projected_argument_object", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT, True): "_record_projected_argument_object", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT, True): "_record_projected_argument_object", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT, True): "_skip_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT, True): "_record_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT, False): "_skip_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT, True): "_record_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT, True): "_record_projected_argument_object", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT, True): "_skip_projected_argument_object", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT, False): "_skip_projected_argument_object", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, False): "_skip_projected_argument_object", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, False): "_skip_projected_argument_object", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT, True): "_record_projected_argument_object", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT, True): "_record_projected_argument_object", - } - ) - _ARRAY_ACCESS_VALIDATION_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_writable_array_access_validation", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_readable_array_access_validation", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_writable_array_access_validation", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_writable_array_access_validation", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_readable_array_access_validation", - } - ) - _ARRAY_RELEASE_POLICY_DISPATCHER = DestructionPolicyDispatcher( - { - DestructionPolicy.PYTHON_REFCOUNT: "_release_python_owned_array_memory", - DestructionPolicy.CALLER: "_borrow_array_memory", - DestructionPolicy.WRAPPER_DEALLOC: "_borrow_array_memory", - DestructionPolicy.NATIVE_OWNER: "_borrow_array_memory", - DestructionPolicy.CALL_LOCAL: "_borrow_array_memory", - DestructionPolicy.NONE: "_borrow_array_memory", - DestructionPolicy.BLOCKED: "_blocked_array_release_policy", - } - ) - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, sharedlib_dirpath, verbose): - """Initialize state collected while building one extension module.""" - # A map used to find the Python-compatible Variable equivalent to an object in the AST - self._python_object_map = {} - # The object that should be returned to indicate an error - self._error_exit_code = NIL - - self._sharedlib_dirpath = sharedlib_dirpath - super().__init__(verbose) - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_Module(self, expr): - """ - Build a `PyModule` from a `Module`. - - Create a `PyModule` which wraps a C-compatible `Module`. - - Parameters - ---------- - expr : Module - The module which can be called from C. - - Returns - ------- - PyModule - The module which can be called from Python. - """ - # Define scope - scope = expr.scope - original_mod = expr.original_module - original_mod_name = original_mod.scope.get_python_name(original_mod.name) - self._native_array_owned_operation_functions = [] - self._native_array_result_operation_module_name = str(original_mod_name) - - mod_scope = Scope( - name=original_mod_name, - used_symbols=scope.local_used_symbols.copy(), - original_symbols=scope.python_names.copy(), - naming_policy=scope.naming_policy, - public_namespace=scope.public_namespace, - symbol_language=self.start_language, - scope_type="module", - ) - self.scope = mod_scope - - imports = [self._visit(i) for i in original_mod.imports] - imports = [i for i in imports if i] - - # Ensure all class types are declared - for c in expr.classes: - name = c.name - python_name = c.scope.get_python_name(name) - struct_name = self.scope.get_new_name(f"Py{python_name}Object") - dtype = DataTypeFactory( - struct_name, - self.scope.get_python_name(struct_name), - BaseClass=WrapperCustomDataType, - )() - - type_name = self.scope.get_new_name(f"Py{python_name}Type") - superclasses = tuple( - self.scope.find(base.scope.get_python_name(base.name), "classes", raise_if_missing=True) - for base in c.superclasses - ) - wrapped_class = PyClassDef( - c, - struct_name, - type_name, - self.scope.new_child_scope(name, "class"), - docstring=self._class_docstring(c), - class_type=dtype, - superclasses=superclasses, - ) - - orig_cls_dtype = c.scope.parent_scope.cls_constructs[python_name] - self._python_object_map[c] = wrapped_class - self._python_object_map[c.original_class] = wrapped_class - self._python_object_map[orig_cls_dtype] = dtype - - self.scope.insert_class(wrapped_class, python_name) - - # Wrap classes - classes = [self._visit(i) for i in expr.classes] - - funcs, interfaces, python_exports = self._wrap_module_callables(expr) - owned_operation_functions = self._native_array_owned_operation_functions - funcs.extend(function for function, _export_name in owned_operation_functions) - if python_exports is not None: - python_exports.update( - {id(function): (((), export_name),) for function, export_name in owned_operation_functions} - ) - if python_exports is not None: - python_exports.update( - { - id(wrapped): expr.get_python_exports(source) - for source, wrapped in zip(expr.classes, classes, strict=True) - } - ) - - module_def_name = self.scope.get_new_name("module") - namespace_module_defs = self._namespace_module_definitions(expr) - module_properties = self._module_variable_properties(expr, funcs) - init_func = self._build_module_init_function( - expr, - imports, - module_def_name, - namespace_module_defs, - module_properties, - ) - - API_var, import_func = self._build_module_import_function(expr) - - self.exit_scope() - - original_mod_name = mod_scope.get_python_name(original_mod.name) - return PyModule( - original_mod_name, - [API_var], - funcs, - imports=imports, - overload_sets=interfaces, - classes=classes, - scope=mod_scope, - init_func=init_func, - import_func=import_func, - module_def_name=module_def_name, - module_properties=module_properties, - namespace_module_defs=namespace_module_defs, - python_exports=python_exports, - ) - - def _wrap_module_callables(self, expr): - """Wrap module functions, overloads, and generated variable getters.""" - funcs_to_wrap = [ - function - for function in expr.funcs - if function not in (expr.init_func, expr.free_func) and function.is_semantic and not function.is_private - ] - funcs_to_wrap.extend(expr.removed_functions) - funcs = [self._visit(function) for function in funcs_to_wrap] - python_exports = self._callable_python_exports(expr, funcs_to_wrap, funcs) - self._append_allocatable_variable_getters(expr, funcs, python_exports) - - source_interfaces = [interface for interface in expr.overload_sets if not interface.is_private] - interfaces = [self._visit(interface) for interface in source_interfaces] - if python_exports is not None: - python_exports.update( - { - id(wrapped): expr.get_python_exports(source) - for source, wrapped in zip(source_interfaces, interfaces, strict=True) - } - ) - return funcs, interfaces, python_exports - - @staticmethod - def _callable_python_exports(expr, source_functions, wrapped_functions): - """Map wrapped callables to their explicit Python export paths.""" - if not expr.has_explicit_python_exports: - return None - return { - id(wrapped): expr.get_python_exports(source) - for source, wrapped in zip(source_functions, wrapped_functions, strict=True) - } - - @staticmethod - def _release_python_owned_array_memory(_subject, _decision): - """Tell NumPy to release Python-owned array result storage.""" - return convert_to_literal(True) - - @staticmethod - def _borrow_array_memory(_subject, _decision): - """Tell NumPy that some non-NumPy owner releases the array storage.""" - return convert_to_literal(False) - - @staticmethod - def _blocked_array_release_policy(subject, decision): - """Reject blocked release policy if it reaches binding generation.""" - name = getattr(subject, "name", type(subject).__name__) - raise ValueError(f"Array result {name!r} has blocked release policy: {decision.blocker}") - - def _append_allocatable_variable_getters(self, expr, funcs, python_exports): - """Add heap-backed module array getters to callable wrappers.""" - for variable in expr.variable_wrappers: - if not isinstance(variable, BindCArrayVariable): - continue - decision = ownership_decision_for_codegen_variable(variable) - if decision.storage_mode is not StorageMode.HEAP: - continue - getter = self._get_allocatable_module_array_getter(variable) - funcs.append(getter) - if python_exports is not None: - source_name = getter.original_function.name - python_exports[id(getter)] = tuple( - (namespace, str(self.scope.get_python_name(source_name))) - for namespace, _ in expr.get_python_exports(variable) - ) - - self._append_native_array_handle_operation_wrappers(expr, funcs, python_exports) - - def _append_native_array_handle_operation_wrappers(self, expr, funcs, python_exports): - """Add private generated operation wrappers used by native array handles.""" - for variable in expr.variable_wrappers: - if not isinstance(variable, BindCNativeArrayHandleVariable): - continue - for operation_name, operation in variable.operation_function_items: - wrapped = self._wrap_native_array_handle_operation(variable, operation_name, operation) - funcs.append(wrapped) - if python_exports is not None: - source_name = operation.original_function.name - python_exports[id(wrapped)] = (((), str(self.scope.get_python_name(source_name))),) - - def _wrap_native_array_handle_operation(self, variable, operation_name, operation): - """Build the private Python callable for one generated handle operation.""" - if self._uses_native_array_descriptor_view_operation_wrapper(variable, operation_name): - return self._native_array_descriptor_view_operation_wrapper(variable, operation) - if self._uses_native_array_pointer_operation_wrapper(operation_name, operation): - return self._native_array_pointer_operation_wrapper(variable, operation) - return self._visit(operation) - - @staticmethod - def _uses_native_array_descriptor_view_operation_wrapper(variable, operation_name): - """Return whether a generated handle operation needs CFI descriptor decoding.""" - policy = variable.native_array_handle_policy - return ( - operation_name in {"descriptor", "to_numpy"} - and policy is not None - and policy.descriptor_kind == "pointer" - and policy.requires_pointer_c_descriptor_interop - and (operation_name == "descriptor" or policy.to_numpy == "descriptor_view") - ) - - @staticmethod - def _uses_native_array_pointer_operation_wrapper(operation_name, operation): - """Return whether a generated handle operation returns a raw native pointer address.""" - result = getattr(getattr(operation, "results", None), "var", None) - return operation_name in {"array_actual", "descriptor"} and getattr(result, "dtype", None) is BindCPointer() - - def _native_array_pointer_operation_wrapper(self, variable, operation): - """Wrap a generated native pointer operation as a Python integer address.""" - wrapper_name = self.scope.get_new_name(operation.name + "_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(wrapper_name, "function") - self.scope = func_scope - pointer = Variable( - BindCPointer(), - func_scope.get_new_name(f"{variable.name}_pointer"), - memory_handling="alias", - ) - func_scope.insert_variable(pointer) - py_result = self._new_python_object(f"{variable.name}_pointer_address") - func_args, body, call_args = self._native_array_operation_wrapper_arguments(operation) - body.extend( - [ - Assign(pointer, operation(*call_args)), - AliasAssign(py_result, PyLong_FromVoidPtr(pointer)), - If(IfSection(Is(py_result, NIL), [Return(self._error_exit_code)])), - Return(py_result), - ] - ) - self.exit_scope() - function = PyFunctionDef( - wrapper_name, - [FunctionDefArgument(arg) for arg in func_args], - body, - FunctionDefResult(py_result), - scope=func_scope, - original_function=operation.original_function, - ) - self.scope.insert_function(function, func_scope.get_python_name(wrapper_name)) - self._python_object_map[operation] = function - return function - - def _native_array_operation_wrapper_arguments(self, operation): - """Unpack an optional bound owner and convert it for one private operation.""" - original = getattr(operation, "original_function", operation) - owner_class = original.decorators.get(INTERNAL_NATIVE_ARRAY_HANDLE_OWNER_CLASS_METADATA) - original_arguments = original.arguments if owner_class is not None else () - if owner_class is not None: - self.scope.insert_symbol(operation.arguments[0].var.original_var.name) - func_args, body = self._unpack_python_args( - original_arguments, - None if owner_class is None else owner_class.class_type, - ) - if owner_class is None: - return func_args, body, [] - owner_argument = operation.arguments[0] - self._python_object_map[owner_argument] = func_args[0] - converted = self._visit(owner_argument) - self._python_object_map.pop(owner_argument) - body.extend(converted["body"]) - return func_args, body, converted["args"] - - def _native_array_descriptor_view_operation_wrapper(self, variable, operation): - """Wrap a generated pointer descriptor-view operation as a decoded mapping.""" - original_function = getattr(operation, "original_function", operation) - wrapper_name = self.scope.get_new_name(operation.name + "_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(wrapper_name, "function") - self.scope = func_scope - - func_args, body, call_args = self._native_array_operation_wrapper_arguments(operation) - descriptor_storage = Variable( - CFIDescriptorStorageType.get_new(variable.rank), - func_scope.get_new_name(f"{variable.name}_descriptor_storage"), - ) - descriptor_pointer = Variable( - CFIDescriptorType(), - func_scope.get_new_name(f"{variable.name}_descriptor"), - memory_handling="alias", - ) - establish_status = Variable( - CNativeInt(), - func_scope.get_new_name(f"{variable.name}_descriptor_status"), - ) - func_scope.insert_variable(descriptor_storage) - func_scope.insert_variable(descriptor_pointer) - func_scope.insert_variable(establish_status) - descriptor_result = self._new_python_object(f"{variable.name}_descriptor_view") - - body.extend( - [ - AliasAssign( - descriptor_pointer, - PointerCast(ObjectAddress(descriptor_storage), descriptor_pointer), - ), - Assign( - establish_status, - CFIDescriptorEstablish(descriptor_pointer, variable.dtype, variable.rank), - ), - If( - IfSection( - Ne(establish_status, Variable(CNativeInt(), "CFI_SUCCESS")), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr(convert_to_literal("failed to establish pointer C descriptor")), - ), - Return(self._error_exit_code), - ], - ) - ), - operation(*call_args, ObjectAddress(descriptor_pointer)), - ] - ) - body.extend(self._native_array_descriptor_view_body(descriptor_pointer, descriptor_result, rank=variable.rank)) - body.append(Return(descriptor_result)) - - self.exit_scope() - function = PyFunctionDef( - wrapper_name, - [FunctionDefArgument(arg) for arg in func_args], - body, - FunctionDefResult(descriptor_result), - scope=func_scope, - docstring="", - original_function=original_function, - ) - self.scope.insert_function(function, func_scope.get_python_name(wrapper_name)) - self._python_object_map[operation] = function - return function - - def _namespace_module_definitions(self, expr): - """Create generated module-definition names for nested exports.""" - namespaces = set() - objects = (*expr.funcs, *expr.overload_sets, *expr.classes, *expr.variables) - for obj in objects: - for namespace, _ in expr.get_python_exports(obj): - namespaces.update(tuple(namespace[:index]) for index in range(1, len(namespace) + 1)) - return { - namespace: self.scope.get_new_name(f"module_{'_'.join(namespace)}", object_type="wrapper") - for namespace in sorted(namespaces, key=lambda item: (len(item), item)) - } - - def _module_variable_properties(self, expr, funcs): - """Group internal module-variable accessors by exported namespace.""" - properties = {} - source_variables = {str(variable.name): variable for variable in expr.original_module.variables} - for function in funcs: - decorators = getattr(getattr(function, "original_function", None), "decorators", {}) - variable_name = decorators.get(INTERNAL_MODULE_VARIABLE_NAME_METADATA) - access = decorators.get(INTERNAL_MODULE_VARIABLE_ACCESS_METADATA) - if not isinstance(variable_name, str) or access not in {"get", "set"}: - continue - source = source_variables[variable_name] - for namespace, export_name in expr.original_module.get_python_exports(source): - descriptor = properties.setdefault( - namespace, - { - "setup_name": self.scope.get_new_name( - f"{'_'.join(namespace) or 'root'}_module_property_setup", - object_type="wrapper", - ), - "items": {}, - }, - ) - item = descriptor["items"].setdefault(export_name, {"get": None, "set": None}) - item[access] = function - return properties - - def _visit_BindCModule(self, expr): - """ - Build a `PyModule` from a `BindCModule`. - - Create a `PyModule` which wraps a C-compatible `BindCModule`. This function calls the - more general `_visit_Module` however additional steps are required to ensure that the - Fortran functions and variables are declared in C. - - Parameters - ---------- - expr : Module - The module which can be called from C. - - Returns - ------- - PyModule - The module which can be called from Python. - """ - pymod = self._visit_Module(expr) - - # Add declarations for C-compatible variables - decs = [ - Declare(v.clone(v.name.lower()), module_variable=True, external=True) - for v in expr.variables - if not v.is_private and isinstance(v, BindCModuleVariable) - ] - pymod.declarations = decs - - external_funcs = [] - # Add external functions for functions wrapping array variables - for v in expr.variable_wrappers: - for f in self._bind_c_variable_wrapper_functions(v): - external_funcs.append(FunctionDef(f.name, f.arguments, [], f.results, is_header=True, scope=f.scope)) - - # Add external functions for normal functions - external_funcs.extend( - FunctionDef( - f.name, - f.arguments, - [], - f.results, - is_header=True, - scope=f.scope, - ) - for f in expr.funcs - ) - external_funcs.extend( - FunctionDef( - f.name, - f.arguments, - [], - f.results, - is_header=True, - scope=f.scope, - ) - for i in expr.overload_sets - for f in i.functions - ) - - for c in expr.classes: - m = c.new_func - external_funcs.append(FunctionDef(m.name, m.arguments, [], m.results, is_header=True, scope=m.scope)) - for m in c.methods: - external_funcs.append( - FunctionDef( - m.name, - m.arguments, - [], - m.results, - is_header=True, - scope=m.scope, - ) - ) - for i in c.overload_sets: - for f in i.functions: - external_funcs.append( - FunctionDef( - f.name, - f.arguments, - [], - f.results, - is_header=True, - scope=f.scope, - ) - ) - for a in c.attributes: - functions = ( - tuple(function for _name, function in a.operation_function_items) - if isinstance(a, BindCNativeArrayHandleProperty) - else (a.getter, a.setter) - ) - for f in functions: - if f: - external_funcs.append( - FunctionDef( - f.name, - f.arguments, - [], - f.results, - is_header=True, - scope=f.scope, - ) - ) - pymod.external_funcs = external_funcs - - return pymod - - @staticmethod - def _bind_c_variable_wrapper_functions(variable): - """Return generated Bind-C functions attached to one module-variable wrapper.""" - if isinstance(variable, BindCArrayVariable): - return (variable.wrapper_function,) - if isinstance(variable, BindCNativeArrayHandleVariable): - return tuple(function for _name, function in variable.operation_function_items) - return () - - def _visit_FunctionOverloadSet(self, expr): - """ - Build a `PyFunctionOverloadSet` from an `FunctionOverloadSet`. - - Create a `PyFunctionOverloadSet` which wraps a C-compatible `FunctionOverloadSet`. The `PyFunctionOverloadSet` - should take three arguments (`self`, `args`, and `kwargs`) and return a - `PythonObjectType`. The arguments are unpacked into multiple `PythonObjectType`s - which are passed to `PyFunctionDef`s describing each of the internal - `FunctionDef` objects. The appropriate `PyFunctionDef` is chosen using an - additional function which calculates an integer type_indicator. - - Parameters - ---------- - expr : FunctionOverloadSet - The interface which can be called from C. - - Returns - ------- - PyFunctionOverloadSet - The interface which can be called from Python. - - See Also - -------- - CToPythonWrapper._get_type_check_function : The function which defines the calculation - of the type_indicator. - """ - # Initialise the scope - func_name = self.scope.get_new_name(expr.name + "_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - original_funcs = expr.functions - example_func = original_funcs[0] - class_base = get_enclosing_class(expr) - has_bound_arg = bool(expr.arguments and expr.arguments[0].bound_argument) - class_dtype = class_base.class_type if class_base and has_bound_arg else None - is_magic = expr.name in magic_overload_funcs - - for f in original_funcs: - self._visit(f) - - # Add the variables to the expected symbols in the scope - for a in example_func.arguments: - func_scope.insert_symbol(a.var.name) - - # Create necessary arguments - python_args = example_func.arguments - if is_magic: - func_args = self._get_python_argument_variables(python_args) - body = [] - if expr.name == "__pow__": - modulo = self._new_python_object("modulo") - func_args.append(modulo) - body.append( - If( - IfSection( - IsNot(modulo, Py_None), - [ - PyErr_SetString( - PyTypeError, - CStrStr(convert_to_literal("pow() with a modulus is not supported")), - ), - Return(self._error_exit_code), - ], - ) - ) - ) - else: - func_args, body = self._unpack_python_args(python_args, class_dtype) - - # Get python arguments which will be passed to FunctionDefs - python_arg_objs = [self._python_object_map[a] for a in python_args] - if expr.native_name.casefold() == "assignment(=)" and len(python_arg_objs) == 2: - body.append( - If( - IfSection( - Is(python_arg_objs[0], python_arg_objs[1]), - [ - Py_INCREF(python_arg_objs[0]), - Return(python_arg_objs[0]), - ], - ) - ) - ) - - type_indicator = Variable(NumpyInt64Type(), self.scope.get_new_name("type_indicator")) - self.scope.insert_variable(type_indicator) - - self.exit_scope() - - # Determine flags which indicate argument type - type_check_name = self.scope.get_new_name(expr.name + "_type_check", object_type="wrapper") - type_check_func, argument_type_flags = self._get_type_check_function( - type_check_name, - python_arg_objs, - original_funcs, - allow_native_scalars=is_magic, - ) - - self.scope = func_scope - # Build the body of the function - body.append(Assign(type_indicator, type_check_func(*python_arg_objs))) - - functions = [] - if_sections = [] - returns_assignment_target = expr.native_name.casefold() == "assignment(=)" and len(python_arg_objs) == 2 - assignment_result = ( - self._new_python_object("assignment_result", is_temp=True) if returns_assignment_target else None - ) - for func, index in argument_type_flags.items(): - # Add an IfSection calling the appropriate function if the type_indicator matches the index - wrapped_func = self._python_object_map[func] - section_body = ( - self._assignment_dispatch_return_body(wrapped_func, python_arg_objs, assignment_result) - if assignment_result is not None - else [Return(wrapped_func(*python_arg_objs))] - ) - if_sections.append( - IfSection( - Eq(type_indicator, convert_to_literal(index)), - section_body, - ) - ) - functions.append(wrapped_func) - if_sections.append( - IfSection( - Eq(type_indicator, convert_to_literal(-1)), - [Return(self._error_exit_code)], - ) - ) - if_sections.append( - IfSection( - convert_to_literal(True), - [ - PyErr_SetString( - PyTypeError, - CStrStr(convert_to_literal("Unexpected type combination")), - ), - Return(self._error_exit_code), - ], - ) - ) - body.append(If(*if_sections)) - result_var = self._new_python_object("result", is_temp=True) - self.exit_scope() - - dispatcher_func = FunctionDef( - func_name, - [FunctionDefArgument(a) for a in func_args], - body, - FunctionDefResult(result_var), - scope=func_scope, - ) - for a in python_args: - self._python_object_map.pop(a) - - return PyFunctionOverloadSet(func_name, functions, dispatcher_func, type_check_func, expr) - - def _assignment_dispatch_return_body(self, wrapped_func, python_arg_objs, result_var): - """Call a private assignment target and return the mutated left-hand object.""" - return [ - AliasAssign(result_var, wrapped_func(*python_arg_objs)), - If(IfSection(Is(result_var, NIL), [Return(self._error_exit_code)])), - Py_DECREF(result_var), - Py_INCREF(python_arg_objs[0]), - Return(python_arg_objs[0]), - ] - - def _visit_FunctionDef(self, expr): - """ - Build a `PyFunctionDef` from a `FunctionDef`. - - Create a `PyFunctionDef` which wraps a C-compatible `FunctionDef`. - The `PyFunctionDef` should take three arguments (`self`, `args`, - and `kwargs`) and return a `PythonObjectType`. If the function is - called from an FunctionOverloadSet then the arguments are `PythonObjectType`s - describing each of the arguments of the C-compatible function. - - Parameters - ---------- - expr : FunctionDef - The function which can be called from C. - - Returns - ------- - PyFunctionDef - The function which can be called from Python. - """ - original_func = getattr(expr, "original_function", expr) - func_name = self.scope.get_new_name(expr.name + "_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - original_func_name = original_func.scope.get_python_name(original_func.name) - - class_base = original_func.decorators.get(INTERNAL_NATIVE_ARRAY_HANDLE_OWNER_CLASS_METADATA) - if class_base is None: - class_base = get_enclosing_class(expr) - has_bound_arg = bool(expr.arguments and expr.arguments[0].bound_argument) - class_dtype = class_base.class_type if class_base and has_bound_arg else None - - is_bind_c_function_def = isinstance(expr, BindCFunctionDef) - - # Add the variables to the expected symbols in the scope - for a in expr.arguments: - a_var = a.var - func_scope.insert_symbol(getattr(a_var, "original_var", a_var).name) - - in_overload_set = is_in_overload_set(expr) - - # Get variables describing the arguments and results that are seen from Python - python_args = expr.arguments - python_results = expr.results - - # Get the arguments of the PyFunctionDef - func_args, body = self._python_wrapper_arguments( - original_func, - original_func_name, - python_args, - class_dtype, - in_overload_set, - func_scope, - ) - - # Get the code required to extract the C-compatible arguments from the Python arguments - wrapped_args = [self._visit(a) for a in python_args] - body += [line for arg in wrapped_args for line in arg["body"]] - callback_setup = [line for arg in wrapped_args for line in arg.get("callback_setup", ())] - callback_cleanup = [line for arg in reversed(wrapped_args) for line in arg.get("callback_cleanup", ())] - - # Get the code required to wrap the C-compatible results into Python objects - # This function creates variables so it must be called before extracting them from the scope. - wrapped_results = self._wrapped_python_results( - original_func_name, - func_args, - python_results, - is_bind_c_function_def, - expr, - body, - ) - - # Get the arguments and results which should be used to call the c-compatible function - func_call_args = [ca for a in wrapped_args for ca in a["args"]] - - # Get the names of the results collected from the C-compatible function - body.extend(wrapped_results.get("setup", ())) - c_results = wrapped_results["c_results"] - python_result_variable = wrapped_results["py_result"] - - if class_dtype: - body.extend(self._save_referenced_objects(expr, func_args)) - - # Call the C-compatible function - body.extend(callback_setup) - body.extend(self._native_call_nodes(expr, original_func, func_call_args, c_results, wrapped_args)) - body.extend(callback_cleanup) - - # Deallocate the C equivalent of any array arguments - # The C equivalent is the same variable that is passed to the function unless the target language is Fortran. - # In this case known-size stack arrays are used which are automatically deallocated when they go out of scope. - self._append_array_argument_cleanup(python_args, body) - python_result_variable = self._project_wrapper_result( - expr, - original_func, - original_func_name, - func_args, - python_result_variable, - c_results, - wrapped_results, - wrapped_args, - body, - ) - body.extend(ai for arg in wrapped_args for ai in arg["clean_up"]) - - # Pack the Python compatible results of the function into one argument. - res, func_results = self._python_wrapper_function_result(original_func_name, python_result_variable) - body.append(Return(res)) - - self.exit_scope() - self._drop_python_argument_mappings(python_args) - - function = PyFunctionDef( - func_name, - func_args, - body, - func_results, - scope=func_scope, - docstring=self._function_docstring(original_func_name, expr, original_func), - original_function=original_func, - ) - - self.scope.insert_function(function, func_scope.get_python_name(func_name)) - self._python_object_map[expr] = function - - return function - - def _python_wrapper_arguments( - self, - original_func, - original_func_name, - python_args, - class_dtype, - in_overload_set, - func_scope, - ): - """Build Python-visible wrapper arguments and their unpacking body.""" - if in_overload_set or original_func_name in magic_binary_funcs or original_func_name == "__len__": - raw_args = self._get_python_argument_variables(python_args) - return [FunctionDefArgument(argument) for argument in raw_args], [] - python_arg_names = [self._function_argument_python_name(original_func, arg) for arg in python_args] - raw_args, body = self._unpack_python_args( - python_args, - class_dtype, - python_arg_names=python_arg_names, - ) - return [FunctionDefArgument(argument) for argument in raw_args], body - - def _wrapped_python_results( - self, - original_func_name, - func_args, - python_results, - is_bind_c_function_def, - expr, - body, - ): - """Build result conversion metadata for a Python wrapper.""" - if original_func_name not in magic_binary_funcs or not original_func_name.startswith("__i"): - owner_object = func_args[0].var if func_args else None - return self._convert_result(python_results.var, is_bind_c_function_def, expr, owner_object=owner_object) - result = func_args[0].var.clone(self.scope.get_new_name(func_args[0].var.name), is_argument=False) - body.extend((AliasAssign(result, func_args[0].var), Py_INCREF(result))) - return {"c_results": [], "py_result": result, "body": []} - - def _append_array_argument_cleanup(self, python_args, body) -> None: - """Append cleanup for temporary C array arguments.""" - for argument in python_args: - orig_var = argument.var - if isinstance(orig_var, FunctionAddress) or not orig_var.is_ndarray: - continue - variable = self.scope.find(orig_var.name, category="variables", raise_if_missing=True) - if variable.is_optional: - body.append(If(IfSection(IsNot(variable, NIL), [Deallocate(variable)]))) - else: - body.append(Deallocate(variable)) - - def _project_wrapper_result( - self, - expr, - original_func, - original_func_name, - func_args, - python_result_variable, - c_results, - wrapped_results, - wrapped_args, - body, - ): - """Project native results onto the public Python return contract.""" - if original_func_name == "__len__": - self.scope.remove_variable(python_result_variable) - return c_results[0] - body.extend(wrapped_results["body"]) - if original_func_name in magic_binary_funcs and original_func_name.startswith("__i"): - return python_result_variable - native_py_results = wrapped_results.get( - "py_results", - [] if python_result_variable is Py_None else [python_result_variable], - ) - if original_func.decorators.get(RUNTIME_RETAIN_RESULT_OWNER_METADATA): - if not func_args or len(native_py_results) != 1: - raise ValueError( - f"Native array operation {original_func_name!r} needs one owner argument and one array result" - ) - body.extend( - self._incref_borrowed_array_getter( - original_func.results.var, - ownership_decision_for_codegen_variable(original_func.results.var), - func_args[0].var, - native_py_results[0], - ) - ) - native_owned_results = wrapped_results.get("owned_py_results", [True] * len(native_py_results)) - wrapped_arg_cleanup = [item for arg in wrapped_args for item in arg["clean_up"]] - body.extend( - self._status_error_check( - original_func, - wrapped_results, - native_py_results, - native_owned_results, - wrapped_arg_cleanup, - ) - ) - projected_return = self._project_python_return( - expr, - original_func, - native_py_results, - native_owned_results, - excluded_output_names=self._status_error_output_names(original_func), - ) - body.extend(projected_return["body"]) - return projected_return["result"] - - def _python_wrapper_function_result(self, original_func_name, python_result_variable): - """Build the wrapper return expression and result declaration.""" - if original_func_name == "__len__": - result = cast_to(python_result_variable, Py_ssize_t()) - definition = FunctionDefResult(Variable(Py_ssize_t(), self.scope.get_new_name(), is_temp=True)) - return result, definition - if python_result_variable is Py_None: - return Py_None, FunctionDefResult(self._new_python_object("result", is_temp=True)) - return python_result_variable, FunctionDefResult(python_result_variable) - - def _drop_python_argument_mappings(self, python_args) -> None: - """Remove temporary Python-object mappings for unbound arguments.""" - for argument in python_args: - if not argument.bound_argument: - self._python_object_map.pop(argument) - - def _visit_FunctionDefArgument(self, expr): - """ - Get the code which translates a Python `FunctionDefArgument` to a C-compatible `Variable`. - - Get the code necessary to transform a Variable passed as an argument in Python, from an object with - datatype `PythonObjectType` to a Variable that can be used in C code. - - The relevant `PythonObjectType` is collected from `self._python_object_map`. - - The necessary steps are: - - Create a variable to store the C-compatible result. - - Initialise the variable to any provided default value. - - Cast the Python object to the C object using utility functions. - - Raise any useful errors (this is not necessary if the FunctionDef is in an interface as errors are - raised while determining which function to call). - - Parameters - ---------- - expr : FunctionDefArgument - The argument of the C function. - - Returns - ------- - dict[str, Any] - A dictionary with the keys: - - body : a list of model objects containing the code which translates the `PythonObjectType` - to a C-compatible variable. - - args : a list of Variables which should be passed to call the function being wrapped. - """ - collect_arg = self._python_object_map[expr] - in_overload_set = is_in_overload_set(expr) - is_bind_c_argument = isinstance(expr.var, BindCVariable) - - orig_var = getattr(expr.var, "original_var", expr.var) - bound_argument = expr.bound_argument - - if isinstance(orig_var, FunctionAddress): - trampoline = FunctionAddress( - self.scope.get_new_name(f"{self.scope.name}_{orig_var.name}_trampoline"), - orig_var.arguments, - orig_var.results, - decorators={ - **orig_var.decorators, - "x2py_callback_trampoline": True, - }, - scope=orig_var.scope, - ) - return { - "body": [PyCallbackValidate(trampoline, collect_arg, self._error_exit_code)], - "args": [trampoline], - "callback_setup": [PyCallbackContextPush(trampoline, collect_arg)], - "callback_cleanup": [PyCallbackContextPop(trampoline)], - "clean_up": [], - } - - # Collect the function which casts from a Python object to a C object - arg_extraction = self._convert_argument(orig_var, collect_arg, bound_argument, is_bind_c_argument) - decision = ownership_decision_for_codegen_variable(orig_var) - - body = [] - cast = arg_extraction["body"] - arg_vars = arg_extraction["args"] - nullable_scalar = bool(arg_extraction.get("nullable_scalar")) - optional_scalar_descriptor = bool(arg_extraction.get("optional_scalar_descriptor")) - - # Initialise to any default value - if expr.has_default or nullable_scalar: - if "default_init" in arg_extraction: - for i, line in enumerate(arg_extraction["default_init"]): - body.insert(i, line) - else: - assert len(arg_vars) == 1 - arg_var = arg_vars[0] - default_val = NIL if nullable_scalar else expr.value - if default_val is NIL: - body.insert(0, AliasAssign(arg_var, default_val)) - else: - body.insert(0, Assign(arg_var, default_val)) - - body.extend(arg_extraction.get("pre_check_body", ())) - - # Create any necessary type checks and errors - if expr.has_default or nullable_scalar: - check_func, err = self._get_type_check_condition( - collect_arg, orig_var, True, body, allow_empty_arrays=is_bind_c_argument - ) - cast_condition = IsNot(collect_arg, Py_None) - if optional_scalar_descriptor: - cast_condition = And(IsNot(collect_arg, NIL), cast_condition) - body.append( - If( - IfSection( - cast_condition, - [ - If( - IfSection(check_func, cast), - IfSection(convert_to_literal(True), [*err, Return(self._error_exit_code)]), - ) - ], - ) - ) - ) - elif not (in_overload_set or bound_argument): - self._ARGUMENT_CAST_GUARD_DISPATCHER.dispatch_decision( - self, - orig_var, - decision, - collect_arg, - cast, - body, - arg_extraction, - is_bind_c_argument, - ) - else: - body.extend(cast) - - return { - "body": body, - "args": arg_vars, - "clean_up": arg_extraction.get("clean_up", ()), - } - - def _append_checked_argument_cast( - self, - orig_var, - _decision, - collect_arg, - cast, - body, - _arg_extraction, - is_bind_c_argument, - ): - """Append type checks before the selected argument conversion body.""" - if self._argument_conversion_owns_validation(orig_var) or _arg_extraction.get("owns_type_check"): - body.extend(cast) - return - check_func, err = self._get_type_check_condition( - collect_arg, orig_var, True, body, allow_empty_arrays=is_bind_c_argument - ) - body.append(If(IfSection(Not(check_func), [*err, Return(self._error_exit_code)]))) - body.extend(cast) - - @staticmethod - def _argument_conversion_owns_validation(orig_var) -> bool: - """Return whether the selected conversion emits the full public contract check.""" - decision = ownership_decision_for_codegen_variable(orig_var) - return decision.python_barrier_action in { - PythonBarrierAction.SCALAR_STORAGE, - PythonBarrierAction.STRING_STORAGE, - PythonBarrierAction.RAW_ADDRESS, - } - - def _append_unchecked_argument_cast( - self, - _orig_var, - _decision, - _collect_arg, - cast, - body, - _arg_extraction, - _is_bind_c_argument, - ): - """Append a conversion body that already contains its own validation.""" - body.extend(cast) - - def _append_replacement_argument_cast( - self, - orig_var, - decision, - collect_arg, - cast, - body, - arg_extraction, - is_bind_c_argument, - ): - """Append replacement argument conversion, including nullable default storage.""" - if decision.nullable and "default_init" in arg_extraction: - check_func, err = self._get_type_check_condition( - collect_arg, orig_var, True, body, allow_empty_arrays=is_bind_c_argument - ) - body.extend(arg_extraction["default_init"]) - body.append( - If( - IfSection( - IsNot(collect_arg, Py_None), - [ - If( - IfSection(check_func, cast), - IfSection(convert_to_literal(True), [*err, Return(self._error_exit_code)]), - ) - ], - ) - ) - ) - return - self._append_checked_argument_cast( - orig_var, - decision, - collect_arg, - cast, - body, - arg_extraction, - is_bind_c_argument, - ) - - def _visit_BindCArrayVariable(self, expr): - """ - Get the code which translates a Fortran array module variable to an object with datatype `PythonObjectType`. - - Get the code which translates a Fortran array module variable to an object with datatype `PythonObjectType` - which can be used as a Python module variable. This new object is saved into self._python_object_map. - Fortran arrays are not compatible with C, but objects of type `BindCArrayVariable` contain wrapper - functions which can be used to retrieve C-compatible variables. - - The necessary steps are: - - Create the variables necessary to retrieve array objects from Fortran. - - Call the bind c wrapper function to initialise these objects. - - Pack the results into a C-compatible `ndarray`. - - Use `self._visit_Variable` to get the object with datatype `PythonObjectType`. - - Correct the key in self._python_object_map initialised by `self._wrap_Variable`. - - Parameters - ---------- - expr : BindCArrayVariable - The array module variable. - - Returns - ------- - list of codegen model object - The code which translates the Variable to a Python-compatible variable. - """ - v = expr.original_variable - - # Get pointer to store raw array data - data_var = self.scope.get_temporary_variable( - dtype_or_var=VoidType(), name=v.name + "_data", memory_handling="alias" - ) - itemsize_var = ( - self.scope.get_temporary_variable(NumpyInt64Type(), name=v.name + "_itemsize") - if self._is_character_array(v) - else None - ) - # Create variables to store the shape of the array - shape_var = self.scope.get_temporary_variable( - NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), - name=v.name + "_size", - shape=(v.rank,), - ) - shape = [IndexedElement(shape_var, i) for i in range(v.rank)] - # Get the bind_c function which wraps a fortran array and returns c objects - var_wrapper = expr.wrapper_function - # Call bind_c function - c_results = [ObjectAddress(data_var)] - if itemsize_var is not None: - c_results.append(itemsize_var) - c_results.extend(shape) - call = Assign(PythonTuple(*c_results), var_wrapper()) - - # Create the resulting Variable with datatype `PythonObjectType` - py_equiv = self._new_python_object(f"{v.name}_obj", dtype=v.dtype) - self._python_object_map[expr] = py_equiv - - decision = ownership_decision_for_codegen_variable(expr) - release_memory = self._ARRAY_RELEASE_POLICY_DISPATCHER.dispatch(self, expr, decision) - unallocated_guard = self._return_none_if_unallocated(data_var) if decision.nullable else [] - # Save the ndarray to vars_to_wrap to be handled as if it came from C - create_array = AliasAssign( - py_equiv, self._array_to_python_call(v, data_var, shape_var, itemsize_var, release_memory) - ) - return [ - call, - *unallocated_guard, - create_array, - ] - - def _visit_BindCNativeArrayHandleVariable(self, expr): - """Dispatch module-handle construction from completed native-array policy.""" - owner_module = getattr(self, "_native_array_handle_owner_module", None) - if owner_module is None: - raise ValueError(f"Native array handle module variable {expr.name!r} needs a module owner object") - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - expr, - expr.native_array_handle_policy, - owner_module, - ) - - def _bind_borrowed_native_array_module_handle(self, expr, _policy, owner_module): - """Build a borrowed module handle from its generated operation wrappers.""" - body, py_equiv = self._native_array_handle_creation(expr, owner_module) - self._python_object_map[expr] = py_equiv - return body - - def _native_array_handle_creation( - self, - expr, - owner_object, - *, - operation_owner=None, - operation_function_items=None, - failure_cleanup=(), - call_failure_cleanup=(), - ): - """Create one runtime handle from operations found on its retained owner.""" - policy = expr.native_array_handle_policy - if policy is None: - raise ValueError(f"Native array handle {expr.name!r} is missing completed policy") - - runtime_module = self._new_python_object(f"{expr.name}_runtime_handles") - helper = self._new_python_object(f"{expr.name}_handle_helper") - descriptor_kind = self._new_python_object(f"{expr.name}_descriptor_kind") - dtype = self._new_python_object(f"{expr.name}_dtype") - rank = self._new_python_object(f"{expr.name}_rank") - ops = self._new_python_object(f"{expr.name}_ops") - descriptor_ownership = self._new_python_object(f"{expr.name}_descriptor_ownership") - to_numpy_policy = self._new_python_object(f"{expr.name}_to_numpy_policy") - generation = self._new_python_object(f"{expr.name}_generation") - helper_args = self._new_python_object(f"{expr.name}_handle_helper_args") - py_equiv = self._new_python_object(f"{expr.name}_handle", dtype=expr.dtype) - operation_owner = owner_object if operation_owner is None else operation_owner - retained_owner = owner_object - operation_function_items = ( - expr.operation_function_items if operation_function_items is None else tuple(operation_function_items) - ) - owner_setup = [] - if not isinstance(owner_object.dtype, PythonObjectType): - operation_owner = self._new_python_object(f"{expr.name}_owner") - owner_setup.append(AliasAssign(operation_owner, PointerCast(owner_object, operation_owner))) - retained_owner = operation_owner - - owned_args = [ - descriptor_kind, - dtype, - rank, - ops, - descriptor_ownership, - to_numpy_policy, - generation, - ] - body = [ - *owner_setup, - AliasAssign(runtime_module, PyImport_ImportModule(CStrStr(convert_to_literal("x2py.runtime.handles")))), - If(IfSection(Is(runtime_module, NIL), [*failure_cleanup, Return(self._error_exit_code)])), - AliasAssign( - helper, - PyObject_GetAttrString( - runtime_module, - CStrStr(convert_to_literal("_native_array_handle_from_generated_ops")), - ), - ), - If( - IfSection( - Is(helper, NIL), - [Py_DECREF(runtime_module), *failure_cleanup, Return(self._error_exit_code)], - ) - ), - AliasAssign(descriptor_kind, PyUnicode_FromString(CStrStr(convert_to_literal(policy.descriptor_kind)))), - *self._native_array_descriptor_dtype_object(expr, dtype), - AliasAssign(rank, PyLong_FromLong(convert_to_literal(expr.rank, dtype=CNativeInt()))), - AliasAssign(ops, PyDict_New()), - AliasAssign( - descriptor_ownership, - PyUnicode_FromString(CStrStr(convert_to_literal(policy.descriptor_ownership))), - ), - AliasAssign(to_numpy_policy, PyUnicode_FromString(CStrStr(convert_to_literal(policy.to_numpy)))), - AliasAssign(generation, Py_None), - Py_INCREF(Py_None), - ] - body.extend( - self._return_if_any_native_array_helper_arg_failed( - runtime_module, - helper, - owned_args, - failure_cleanup=failure_cleanup, - ) - ) - body.extend( - self._populate_native_array_handle_ops( - expr, - operation_owner, - operation_function_items, - ops, - [*owned_args, helper, runtime_module], - failure_cleanup=failure_cleanup, - ) - ) - body.extend( - [ - AliasAssign( - helper_args, - PyTuple_Pack( - ObjectAddress(descriptor_kind), - ObjectAddress(dtype), - ObjectAddress(rank), - ObjectAddress(ops), - ObjectAddress(retained_owner), - ObjectAddress(descriptor_ownership), - ObjectAddress(to_numpy_policy), - ObjectAddress(generation), - ), - ), - If( - IfSection( - Is(helper_args, NIL), - [ - *self._decref_all([*owned_args, helper, runtime_module]), - *failure_cleanup, - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(py_equiv, PyObject_CallObject(helper, helper_args)), - Py_DECREF(helper_args), - *self._decref_all([*owned_args, helper, runtime_module]), - If( - IfSection( - Is(py_equiv, NIL), - [*call_failure_cleanup, Return(self._error_exit_code)], - ) - ), - ] - ) - return body, py_equiv - - def _populate_native_array_handle_ops( - self, - expr, - operation_owner, - operation_function_items, - ops, - cleanup, - *, - failure_cleanup=(), - ): - """Populate the generated handle operation dictionary.""" - body = [] - for operation_name, function in operation_function_items: - key = self._new_python_object(f"{expr.name}_{operation_name}_op_key") - operation = self._new_python_object(f"{expr.name}_{operation_name}_op") - body.extend( - [ - AliasAssign(key, PyUnicode_FromString(CStrStr(convert_to_literal(operation_name)))), - If( - IfSection( - Is(key, NIL), - [*self._decref_all(cleanup), *failure_cleanup, Return(self._error_exit_code)], - ) - ), - AliasAssign( - operation, - PyObject_GetAttrString( - operation_owner, - CStrStr(convert_to_literal(self._native_array_handle_operation_export_name(function))), - ), - ), - If( - IfSection( - Is(operation, NIL), - [ - Py_DECREF(key), - *self._decref_all(cleanup), - *failure_cleanup, - Return(self._error_exit_code), - ], - ) - ), - If( - IfSection( - Lt(PyDict_SetItem(ops, key, operation), convert_to_literal(0)), - [ - Py_DECREF(operation), - Py_DECREF(key), - *self._decref_all(cleanup), - *failure_cleanup, - Return(self._error_exit_code), - ], - ) - ), - Py_DECREF(operation), - Py_DECREF(key), - ] - ) - return body - - @staticmethod - def _native_array_handle_operation_export_name(function): - """Return the generated Python attribute name for a handle operation wrapper.""" - original = getattr(function, "original_function", function) - scope = getattr(original, "scope", None) - if scope is not None: - try: - return str(scope.get_python_name(original.name)) - except RuntimeError: - pass - return str(original.name) - - def _visit_BindCModuleConstant(self, expr): - """Convert the ``BindCModuleConstant`` model node.""" - py_equiv = self.scope.get_temporary_variable(PythonObjectType(), memory_handling="alias") - self._python_object_map[expr] = py_equiv - dtype = expr.class_type.underlying_type if isinstance(expr.class_type, FinalType) else expr.class_type - c_value = self.scope.get_temporary_variable(dtype, name=f"{expr.name}_value") - return [ - Assign(c_value, self._module_constant_literal(expr)), - AliasAssign(py_equiv, FunctionCall(C_to_Python(c_value), [c_value])), - ] - - def _visit_BindCNativeArrayHandleProperty(self, expr): - """Dispatch field-handle construction from completed native-array policy.""" - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - expr, - expr.native_array_handle_policy, - ) - - def _bind_borrowed_native_array_field_handle(self, expr, policy): - """Build a borrowed field-handle getter and its private bound operations.""" - wrapped_class = self._python_object_map[expr.owner_class] - for operation_name, operation in expr.operation_function_items: - wrapped_class.add_new_method(self._wrap_native_array_handle_operation(expr, operation_name, operation)) - - getter_name = self.scope.get_new_name( - f"{expr.class_type.name}_{expr.python_name}_handle_getter", - object_type="wrapper", - ) - getter_scope = self.scope.new_child_scope(getter_name, "function") - self.scope = getter_scope - getter_args = [ - self._new_python_object("self_obj", dtype=expr.class_type), - getter_scope.get_temporary_variable(VoidType(), memory_handling="alias"), - ] - body, handle = self._native_array_handle_creation(expr, getter_args[0]) - body.append(Return(handle)) - self.exit_scope() - - getter = PyFunctionDef( - getter_name, - [FunctionDefArgument(argument) for argument in getter_args], - body, - FunctionDefResult(handle), - scope=getter_scope, - original_function=None, - ) - docstring = CStrStr( - convert_to_literal( - f"{policy.descriptor_kind} array descriptor handle; owner retention: {policy.owner_retention}." - ) - ) - self._error_exit_code = NIL - return PyGetSetDefElement(expr.python_name, getter, None, docstring) - - def _visit_BindCClassProperty(self, expr): - """ - Create a PyGetSetDefElement to expose a class attribute/property to Python. - - Create getter and setter functions which are compatible with the expected prototype for - `PyGetSetDef` and which call the getter and setter functions contained in the - BindCClassProperty. The result is returned in a PyGetSetDefElement. - See - for more information about the necessary prototypes. - - Parameters - ---------- - expr : BindCClassProperty - The object containing the getter and setter functions to be wrapped. - - Returns - ------- - PyGetSetDefElement - An object which contains the new getter and setter functions that should be - described in the array of PyGetSetDef objects. - """ - class_type = expr.class_type - name = expr.python_name - # ---------------------------------------------------------------------------------- - # Create getter - # ---------------------------------------------------------------------------------- - getter_name = self.scope.get_new_name(f"{class_type.name}_{name}_getter", object_type="wrapper") - getter_scope = self.scope.new_child_scope(getter_name, "function") - self.scope = getter_scope - - get_val_arg = expr.getter.arguments[0] - self.scope.insert_symbol(get_val_arg.var.original_var.name) - get_val_result = expr.getter.results - - getter_args = [ - self._new_python_object("self_obj", dtype=class_type), - getter_scope.get_temporary_variable(VoidType(), memory_handling="alias"), - ] - - self._python_object_map[get_val_arg] = getter_args[0] - - wrapped_args = self._visit(get_val_arg) - arg_code = wrapped_args["body"] - class_obj = wrapped_args["args"][0] - - # Cast the C variable into a Python variable - get_val_result_var = getattr(get_val_result, "original_function_result_variable", get_val_result.var) - result_wrapping = self._convert_result(get_val_result_var, True, expr.getter) - res_wrapper = result_wrapping["body"] - c_results = result_wrapping["c_results"] - getter_result = result_wrapping["py_result"] - setup = result_wrapping.get("setup", ()) - - call = self._call_wrapped_function(expr.getter, (class_obj,), c_results) - - if expr.getter_policy is not None: - wrapped_var = expr.getter.original_function - if expr.getter_policy.borrowed: - res_wrapper.extend( - self._BORROWED_GETTER_POLICY_DISPATCHER.dispatch_decision( - self, - wrapped_var, - expr.getter_policy, - getter_args[0], - getter_result, - ) - ) - else: - wrapped_var = expr.getter.original_function.results.var - - getter_body = [*setup, *arg_code, call, *res_wrapper, Return(getter_result)] - self.exit_scope() - - args = [FunctionDefArgument(a) for a in getter_args] - getter = PyFunctionDef( - getter_name, - args, - getter_body, - FunctionDefResult(getter_result), - original_function=expr.getter, - scope=getter_scope, - ) - - setter = ( - self._build_policy_property_setter(expr, class_type, name) - if expr.setter_policy is not None and expr.setter_policy.setter_action is not SetterAction.OMIT - else None - ) - - self._error_exit_code = NIL - - docstring = convert_to_literal( - "\n".join(expr.docstring.comments) - if expr.docstring - else self._attribute_docstring( - expr.python_name, - wrapped_var, - expr.getter_policy, - expr.setter_policy, - ) - ) - return PyGetSetDefElement(expr.python_name, getter, setter, CStrStr(docstring)) - - def _bind_allocatable_descriptor_argument(self, subject, policy, *args, **kwargs): - """Pack an allocatable handle argument into the Bind-C descriptor ABI.""" - return self._bind_native_array_descriptor_argument(subject, policy, *args, **kwargs) - - def _bind_owned_allocatable_result_handle( - self, - subject, - policy, - wrapped_var, - is_bind_c, - funcdef, - _owner_object=None, - ): - """Transfer a copied native result into persistent standard CFI storage.""" - if not is_bind_c or not isinstance(wrapped_var.class_type, BindCArrayType): - raise ValueError(f"Owned allocatable result {subject.name!r} requires the Bind-C array result ABI") - if funcdef is None: - raise ValueError(f"Owned allocatable result {subject.name!r} requires its containing function") - - data_var, itemsize_var, shape_var, c_results = self._owned_allocatable_result_parts(subject) - descriptor_storage = Variable( - VoidType(), - self.scope.get_new_name(f"{subject.name}_owner_storage"), - memory_handling=StorageMode.ALIAS.value, - ) - descriptor_pointer = Variable( - CFIDescriptorType(), - self.scope.get_new_name(f"{subject.name}_owner_descriptor"), - memory_handling=StorageMode.ALIAS.value, - ) - establish_status = Variable( - CNativeInt(), - self.scope.get_new_name(f"{subject.name}_owner_establish_status"), - ) - allocate_status = Variable( - CNativeInt(), - self.scope.get_new_name(f"{subject.name}_owner_allocate_status"), - ) - for variable in (descriptor_storage, descriptor_pointer, establish_status, allocate_status): - self.scope.insert_variable(variable) - - shape = tuple(IndexedElement(shape_var, index) for index in range(subject.rank)) - element_length = ( - itemsize_var - if itemsize_var is not None - else CFIDescriptorField(descriptor_pointer, "elem_len", NumpyInt64Type()) - ) - lower_bounds = tuple(convert_to_literal(0, dtype=NumpyInt64Type()) for _ in shape) - upper_bounds = tuple(Minus(extent, convert_to_literal(1, dtype=NumpyInt64Type())) for extent in shape) - payload_size = reduce(Mul, (element_length, *shape)) - - owner_object = self._new_python_object(f"{subject.name}_native_owner") - operation_module = self._new_python_object(f"{subject.name}_operation_module") - operation_items = self._owned_allocatable_result_operation_items(subject, policy) - cleanup_storage = [Deallocate(descriptor_storage)] - cleanup_result = [Deallocate(data_var)] - body = [ - Assign(ObjectAddress(descriptor_storage), x2py_malloc(CFIDescriptorStorageSize(subject.rank))), - If( - IfSection( - Is(descriptor_storage, NIL), - [ - PyErr_SetString( - PyMemoryError, - CStrStr(convert_to_literal("Unable to allocate owned native array descriptor storage.")), - ), - *cleanup_result, - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(descriptor_pointer, PointerCast(descriptor_storage, descriptor_pointer)), - Assign( - establish_status, - CFIDescriptorEstablish( - descriptor_pointer, - subject.dtype, - subject.rank, - attribute="allocatable", - element_length=itemsize_var, - ), - ), - *self._owned_cfi_status_guard( - establish_status, - "failed to establish owned allocatable C descriptor", - [*cleanup_storage, *cleanup_result], - ), - If( - IfSection( - IsNot(data_var, NIL), - [ - Assign( - allocate_status, - CFIDescriptorAllocate( - descriptor_pointer, - lower_bounds, - upper_bounds, - element_length, - ), - ), - *self._owned_cfi_status_guard( - allocate_status, - "failed to allocate owned allocatable C descriptor payload", - [*cleanup_storage, *cleanup_result], - ), - c_memcpy( - CFIDescriptorField(descriptor_pointer, "base_addr", BindCPointer()), - data_var, - payload_size, - ), - Deallocate(data_var), - ], - ) - ), - AliasAssign(owner_object, PyLong_FromVoidPtr(descriptor_pointer)), - If( - IfSection( - Is(owner_object, NIL), - [ - *self._owned_descriptor_release_body(descriptor_pointer, allocate_status), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign( - operation_module, - PyImport_ImportModule(CStrStr(convert_to_literal(self._native_array_result_operation_module_name))), - ), - If( - IfSection( - Is(operation_module, NIL), - [ - Py_DECREF(owner_object), - *self._owned_descriptor_release_body(descriptor_pointer, allocate_status), - Return(self._error_exit_code), - ], - ) - ), - ] - handle_body, handle = self._native_array_handle_creation( - subject, - owner_object, - operation_owner=operation_module, - operation_function_items=operation_items, - failure_cleanup=[ - Py_DECREF(operation_module), - Py_DECREF(owner_object), - *self._owned_descriptor_release_body(descriptor_pointer, allocate_status), - ], - call_failure_cleanup=[Py_DECREF(operation_module), Py_DECREF(owner_object)], - ) - body.extend([*handle_body, Py_DECREF(operation_module), Py_DECREF(owner_object)]) - return { - "c_results": PythonTuple(*c_results), - "py_result": handle, - "py_results": [handle], - "owned_py_results": [True], - "body": body, - } - - def _owned_allocatable_result_parts(self, subject): - """Create C result variables for the bridge-local copied array payload.""" - data_var = Variable( - VoidType(), - self.scope.get_new_name(f"{subject.name}_data"), - memory_handling=StorageMode.ALIAS.value, - ) - shape_var = Variable( - NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), - self.scope.get_new_name(f"{subject.name}_shape"), - shape=(subject.rank,), - memory_handling=StorageMode.ALIAS.value, - ) - itemsize_var = ( - Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_itemsize")) - if self._is_character_array(subject) - else None - ) - for variable in (data_var, shape_var, itemsize_var): - if variable is not None: - self.scope.insert_variable(variable) - c_results = [ObjectAddress(data_var)] - if itemsize_var is not None: - c_results.append(itemsize_var) - c_results.extend(IndexedElement(shape_var, index) for index in range(subject.rank)) - return data_var, itemsize_var, shape_var, c_results - - def _owned_allocatable_result_operation_items(self, subject, policy): - """Generate private C operations over one persistent CFI descriptor owner.""" - descriptor_fields = self._owned_allocatable_descriptor_operation(subject, "descriptor_fields") - items = [ - ("shape", descriptor_fields), - ("array_actual", self._owned_allocatable_pointer_operation(subject)), - ("descriptor", self._owned_allocatable_descriptor_pointer_operation(subject)), - ("allocated", self._owned_allocatable_state_operation(subject)), - ("native_byte_order", self._owned_allocatable_true_operation(subject, "native_byte_order")), - ("aligned", self._owned_allocatable_true_operation(subject, "aligned")), - ("writeable", self._owned_allocatable_true_operation(subject, "writeable")), - ("layout", self._owned_allocatable_layout_operation(subject)), - ("to_numpy", descriptor_fields), - ("destroy", self._owned_allocatable_destroy_operation(subject)), - ] - if policy.allows("deallocate"): - items.append(("deallocate", self._owned_allocatable_deallocate_operation(subject))) - if policy.allows("resize"): - items.append(("resize", self._owned_allocatable_resize_operation(subject))) - return tuple(items) - - @staticmethod - def _owned_cfi_status_guard(status, message, cleanup=()): - """Return a RuntimeError branch for a failed standard CFI operation.""" - return [ - If( - IfSection( - Ne(status, Variable(CNativeInt(), "CFI_SUCCESS")), - [ - PyErr_SetString(PyRuntimeError, CStrStr(convert_to_literal(message))), - *cleanup, - Return(NIL), - ], - ) - ) - ] - - def _owned_allocatable_operation_context(self, subject, operation, *, extent_count=0): - """Enter a private C operation wrapper over persistent descriptor storage.""" - outer_scope = self.scope - module_scope = outer_scope - while module_scope.parent_scope is not None: - module_scope = module_scope.parent_scope - original_name = module_scope.get_new_name( - f"private__x2py_owned_{subject.name}_{operation}", - object_type="function", - ) - export_name = str(module_scope.get_python_name(original_name)) - wrapper_name = module_scope.get_new_name(f"{original_name}_wrapper", object_type="wrapper") - func_scope = module_scope.new_child_scope(wrapper_name, "function") - self.scope = func_scope - self._error_exit_code = NIL - - owner_arg = FunctionDefArgument( - Variable(BindCPointer(), "owner_address", is_argument=True, memory_handling=StorageMode.ALIAS.value) - ) - extent_args = tuple( - FunctionDefArgument(Variable(NumpyInt64Type(), f"extent_{index + 1}", is_argument=True)) - for index in range(extent_count) - ) - original_args = (owner_arg, *extent_args) - func_args, body = self._unpack_python_args(original_args) - descriptor_pointer = Variable( - CFIDescriptorType(), - func_scope.get_new_name(f"{subject.name}_owner_descriptor"), - memory_handling=StorageMode.ALIAS.value, - ) - func_scope.insert_variable(descriptor_pointer) - owner_object = self._python_object_map[owner_arg] - body.extend( - [ - AliasAssign(descriptor_pointer, PyLong_AsVoidPtr(owner_object)), - If( - IfSection( - Is(descriptor_pointer, NIL), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr(convert_to_literal("owned native array descriptor pointer is NULL")), - ), - Return(self._error_exit_code), - ], - ) - ), - ] - ) - extents = [] - for argument in extent_args: - extent = Variable(NumpyInt64Type(), func_scope.get_new_name(argument.var.name)) - func_scope.insert_variable(extent) - body.extend( - [ - Assign(extent, PyLong_AsLongLong(self._python_object_map[argument])), - If( - IfSection( - And(Eq(extent, convert_to_literal(-1, dtype=NumpyInt64Type())), PyErr_Occurred()), - [Return(self._error_exit_code)], - ) - ), - ] - ) - extents.append(extent) - original_function = FunctionDef( - original_name, - original_args, - [], - FunctionDefResult(NIL), - scope=module_scope, - decorators={INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA: True}, - is_private=True, - ) - return { - "outer_scope": outer_scope, - "module_scope": module_scope, - "func_scope": func_scope, - "func_args": func_args, - "body": body, - "descriptor": descriptor_pointer, - "extents": tuple(extents), - "original_args": original_args, - "original_function": original_function, - "wrapper_name": wrapper_name, - "export_name": export_name, - } - - def _finish_owned_allocatable_operation(self, context, body, result): - """Finish one private owned-descriptor operation and expose it on the module.""" - function = PyFunctionDef( - context["wrapper_name"], - [FunctionDefArgument(argument) for argument in context["func_args"]], - [*context["body"], *body, Return(result)], - FunctionDefResult(result), - scope=context["func_scope"], - original_function=context["original_function"], - ) - context["module_scope"].insert_function(function, context["export_name"]) - self._native_array_owned_operation_functions.append((function, context["export_name"])) - for argument in context["original_args"]: - self._python_object_map.pop(argument, None) - self.scope = context["outer_scope"] - return function - - def _owned_allocatable_descriptor_operation(self, subject, operation): - """Return decoded standard descriptor facts for one owned result.""" - context = self._owned_allocatable_operation_context(subject, operation) - result = self._new_python_object(f"{subject.name}_{operation}_descriptor") - body = self._native_array_descriptor_view_body(context["descriptor"], result, rank=subject.rank) - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_descriptor_pointer_operation(self, subject): - """Return the persistent standard C descriptor address for mutation.""" - context = self._owned_allocatable_operation_context(subject, "descriptor") - result = self._new_python_object(f"{subject.name}_descriptor_address") - body = [ - AliasAssign(result, PyLong_FromVoidPtr(context["descriptor"])), - If(IfSection(Is(result, NIL), [Return(self._error_exit_code)])), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_pointer_operation(self, subject): - """Return the current owned payload address as a Python integer.""" - context = self._owned_allocatable_operation_context(subject, "array_actual") - result = self._new_python_object(f"{subject.name}_array_actual") - body = [ - AliasAssign( - result, - PyLong_FromVoidPtr(CFIDescriptorField(context["descriptor"], "base_addr", BindCPointer())), - ), - If(IfSection(Is(result, NIL), [Return(self._error_exit_code)])), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_state_operation(self, subject): - """Return whether persistent descriptor payload storage is allocated.""" - context = self._owned_allocatable_operation_context(subject, "allocated") - result = self._new_python_object(f"{subject.name}_allocated") - body = [ - AliasAssign( - result, - PyLong_FromLong(IsNot(CFIDescriptorField(context["descriptor"], "base_addr", BindCPointer()), NIL)), - ), - If(IfSection(Is(result, NIL), [Return(self._error_exit_code)])), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_true_operation(self, subject, operation): - """Return a completed true storage fact for owned CFI allocations.""" - context = self._owned_allocatable_operation_context(subject, operation) - result = self._new_python_object(f"{subject.name}_{operation}") - body = [ - AliasAssign(result, PyLong_FromLong(convert_to_literal(1, dtype=CNativeInt()))), - If(IfSection(Is(result, NIL), [Return(self._error_exit_code)])), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_layout_operation(self, subject): - """Return the Fortran-contiguous layout of a CFI-allocated result.""" - context = self._owned_allocatable_operation_context(subject, "layout") - result = self._new_python_object(f"{subject.name}_layout") - body = [ - AliasAssign(result, PyUnicode_FromString(CStrStr(convert_to_literal("F")))), - If(IfSection(Is(result, NIL), [Return(self._error_exit_code)])), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_deallocate_operation(self, subject): - """Deallocate the payload while retaining the persistent descriptor record.""" - context = self._owned_allocatable_operation_context(subject, "deallocate") - status = Variable(CNativeInt(), context["func_scope"].get_new_name("deallocate_status")) - context["func_scope"].insert_variable(status) - result = self._new_python_object(f"{subject.name}_deallocate_result") - body = [ - If( - IfSection( - IsNot(CFIDescriptorField(context["descriptor"], "base_addr", BindCPointer()), NIL), - [ - Assign(status, CFIDescriptorDeallocate(context["descriptor"])), - *self._owned_cfi_status_guard( - status, - "failed to deallocate owned allocatable descriptor payload", - ), - ], - ) - ), - AliasAssign(result, Py_None), - Py_INCREF(result), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_resize_operation(self, subject): - """Replace owned payload storage with a newly allocated requested shape.""" - context = self._owned_allocatable_operation_context(subject, "resize", extent_count=subject.rank) - status = Variable(CNativeInt(), context["func_scope"].get_new_name("resize_status")) - context["func_scope"].insert_variable(status) - result = self._new_python_object(f"{subject.name}_resize_result") - descriptor = context["descriptor"] - lower_bounds = tuple(convert_to_literal(0, dtype=NumpyInt64Type()) for _ in context["extents"]) - upper_bounds = tuple( - Minus(extent, convert_to_literal(1, dtype=NumpyInt64Type())) for extent in context["extents"] - ) - body = [ - If( - IfSection( - IsNot(CFIDescriptorField(descriptor, "base_addr", BindCPointer()), NIL), - [ - Assign(status, CFIDescriptorDeallocate(descriptor)), - *self._owned_cfi_status_guard( - status, - "failed to release owned allocatable payload before resize", - ), - ], - ) - ), - Assign( - status, - CFIDescriptorAllocate( - descriptor, - lower_bounds, - upper_bounds, - CFIDescriptorField(descriptor, "elem_len", NumpyInt64Type()), - ), - ), - *self._owned_cfi_status_guard(status, "failed to resize owned allocatable descriptor payload"), - AliasAssign(result, Py_None), - Py_INCREF(result), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - def _owned_allocatable_destroy_operation(self, subject): - """Deallocate payload and descriptor storage for an owned result handle.""" - context = self._owned_allocatable_operation_context(subject, "destroy") - status = Variable(CNativeInt(), context["func_scope"].get_new_name("destroy_status")) - context["func_scope"].insert_variable(status) - result = self._new_python_object(f"{subject.name}_destroy_result") - body = [ - *self._owned_descriptor_release_body(context["descriptor"], status), - AliasAssign(result, Py_None), - Py_INCREF(result), - ] - return self._finish_owned_allocatable_operation(context, body, result) - - @staticmethod - def _owned_descriptor_release_body(descriptor, status): - """Release an owned descriptor payload and its persistent record.""" - return [ - If( - IfSection( - IsNot(CFIDescriptorField(descriptor, "base_addr", BindCPointer()), NIL), - [ - Assign(status, CFIDescriptorDeallocate(descriptor)), - If( - IfSection( - Ne(status, Variable(CNativeInt(), "CFI_SUCCESS")), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr( - convert_to_literal("failed to release owned allocatable descriptor payload") - ), - ), - Deallocate(descriptor), - Return(NIL), - ], - ) - ), - ], - ) - ), - Deallocate(descriptor), - ] - - def _bind_pointer_descriptor_argument(self, subject, policy, *args, **kwargs): - """Pack a pointer handle argument into the Bind-C descriptor ABI.""" - return self._bind_native_array_descriptor_argument(subject, policy, *args, **kwargs) - - def _bind_optional_native_array_handle(self, subject, policy, *args, **kwargs): - """Pack an optional absent-handle argument into the Bind-C descriptor ABI.""" - return self._bind_native_array_descriptor_argument(subject, policy, *args, **kwargs) - - def _bind_native_array_descriptor_argument( - self, - subject, - policy, - collect_arg, - bound_argument, - _is_bind_c_argument, - *, - arg_var=None, - ): - """Dispatch descriptor binding from completed output-projection policy.""" - return self._NATIVE_ARRAY_DESCRIPTOR_ARGUMENT_DISPATCHER.dispatch( - self, - subject, - policy, - collect_arg, - bound_argument, - _is_bind_c_argument, - arg_var=arg_var, - ) - - def _bind_fact_packed_native_array_descriptor_argument( - self, - subject, - policy, - collect_arg, - bound_argument, - _is_bind_c_argument, - *, - arg_var=None, - ): - """Validate a handle and establish call-local CFI storage from facts.""" - if bound_argument or arg_var is not None: - raise ValueError(f"Native array descriptor argument {subject.name!r} requires a standalone value slot") - descriptor_type = self._native_array_descriptor_argument_type(policy) - descriptor_arg = Variable( - descriptor_type, - self.scope.get_new_name(subject.name), - shape=(convert_to_literal(len(descriptor_type)),), - ) - descriptor_storage = Variable( - CFIDescriptorStorageType.get_new(subject.rank), - self.scope.get_new_name(f"{subject.name}_descriptor_storage"), - ) - descriptor_pointer = Variable( - CFIDescriptorType(), - self.scope.get_new_name(f"{subject.name}_descriptor"), - memory_handling=StorageMode.ALIAS.value, - ) - base_address = Variable( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_base_address"), - memory_handling=StorageMode.ALIAS.value, - ) - element_length = Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_element_length")) - descriptor_rank = Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_descriptor_rank")) - lower_bounds = tuple( - Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_lower_bound_{index + 1}")) - for index in range(subject.rank) - ) - extents = tuple( - Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_extent_{index + 1}")) - for index in range(subject.rank) - ) - strides = tuple( - Variable(NumpyInt64Type(), self.scope.get_new_name(f"{subject.name}_stride_{index + 1}")) - for index in range(subject.rank) - ) - establish_status = Variable( - CNativeInt(), - self.scope.get_new_name(f"{subject.name}_descriptor_status"), - ) - self.scope.insert_variable(descriptor_storage) - self.scope.insert_variable(descriptor_pointer) - self.scope.insert_variable(base_address) - self.scope.insert_variable(element_length) - self.scope.insert_variable(descriptor_rank) - self.scope.insert_variable(establish_status) - for field in (*lower_bounds, *extents, *strides): - self.scope.insert_variable(field) - self.scope.insert_symbolic_alias( - IndexedElement(descriptor_arg, convert_to_literal(0)), - ObjectAddress(descriptor_pointer), - ) - presence_pointer = None - if descriptor_type.has_presence: - presence_pointer = Variable( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_present"), - memory_handling=StorageMode.ALIAS.value, - ) - self.scope.insert_variable(presence_pointer) - self.scope.insert_symbolic_alias(IndexedElement(descriptor_arg, convert_to_literal(1)), presence_pointer) - body = self._native_array_descriptor_argument_body( - subject, - policy, - collect_arg, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - presence_pointer, - ) - default_init = [AliasAssign(descriptor_pointer, NIL)] - if presence_pointer is not None: - default_init.append(AliasAssign(presence_pointer, NIL)) - return { - "body": body, - "args": [descriptor_arg], - "default_init": default_init, - "owns_type_check": True, - } - - def _bind_direct_native_array_descriptor_argument( - self, - subject, - policy, - collect_arg, - bound_argument, - _is_bind_c_argument, - *, - arg_var=None, - ): - """Pass persistent standard descriptor storage for projected mutation.""" - if bound_argument or arg_var is not None: - raise ValueError(f"Native array descriptor argument {subject.name!r} requires a standalone value slot") - descriptor_type = self._native_array_descriptor_argument_type(policy) - descriptor_arg = Variable( - descriptor_type, - self.scope.get_new_name(subject.name), - shape=(convert_to_literal(len(descriptor_type)),), - ) - descriptor_pointer = Variable( - CFIDescriptorType(), - self.scope.get_new_name(f"{subject.name}_descriptor"), - memory_handling=StorageMode.ALIAS.value, - ) - self.scope.insert_variable(descriptor_pointer) - self.scope.insert_symbolic_alias( - IndexedElement(descriptor_arg, convert_to_literal(0)), - ObjectAddress(descriptor_pointer), - ) - presence_pointer = None - if descriptor_type.has_presence: - presence_pointer = Variable( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_present"), - memory_handling=StorageMode.ALIAS.value, - ) - self.scope.insert_variable(presence_pointer) - self.scope.insert_symbolic_alias(IndexedElement(descriptor_arg, convert_to_literal(1)), presence_pointer) - body = self._direct_native_array_descriptor_argument_body( - subject, - policy, - collect_arg, - descriptor_pointer, - presence_pointer, - ) - default_init = [AliasAssign(descriptor_pointer, NIL)] - if presence_pointer is not None: - default_init.append(AliasAssign(presence_pointer, NIL)) - return { - "body": body, - "args": [descriptor_arg], - "default_init": default_init, - "owns_type_check": True, - } - - def _direct_native_array_descriptor_argument_body( - self, - subject, - policy, - collect_arg, - descriptor_pointer, - presence_pointer, - ): - """Build the runtime call that returns a persistent CFI descriptor pointer.""" - runtime_module = self._new_python_object(f"{subject.name}_runtime_handles") - helper = self._new_python_object(f"{subject.name}_descriptor_helper") - descriptor_kind = self._new_python_object(f"{subject.name}_descriptor_kind") - dtype = self._new_python_object(f"{subject.name}_dtype") - rank = self._new_python_object(f"{subject.name}_rank") - expected_shape = self._new_python_object(f"{subject.name}_shape") - optional_absent = self._new_python_object(f"{subject.name}_optional") - helper_args = self._new_python_object(f"{subject.name}_descriptor_helper_args") - packed = self._new_python_object(f"{subject.name}_descriptor_handoff") - descriptor_item = self._new_python_object(f"{subject.name}_descriptor_item") - owned_args = [descriptor_kind, dtype, rank, expected_shape, optional_absent] - body = [ - AliasAssign(runtime_module, PyImport_ImportModule(CStrStr(convert_to_literal("x2py.runtime.handles")))), - If(IfSection(Is(runtime_module, NIL), [Return(self._error_exit_code)])), - AliasAssign( - helper, - PyObject_GetAttrString( - runtime_module, - CStrStr(convert_to_literal("_native_array_descriptor_handoff_for_binding_positional")), - ), - ), - If(IfSection(Is(helper, NIL), [Py_DECREF(runtime_module), Return(self._error_exit_code)])), - AliasAssign(descriptor_kind, PyUnicode_FromString(CStrStr(convert_to_literal(policy.descriptor_kind)))), - *self._native_array_descriptor_dtype_object(subject, dtype), - AliasAssign(rank, PyLong_FromLong(convert_to_literal(subject.rank, dtype=CNativeInt()))), - *self._native_array_descriptor_shape_object(subject, expected_shape), - AliasAssign( - optional_absent, - PyLong_FromLong(convert_to_literal(1 if policy.optional_absent else 0, dtype=CNativeInt())), - ), - ] - body.extend(self._return_if_any_native_array_helper_arg_failed(runtime_module, helper, owned_args)) - body.extend( - [ - AliasAssign( - helper_args, - PyTuple_Pack( - ObjectAddress(collect_arg), - ObjectAddress(descriptor_kind), - ObjectAddress(dtype), - ObjectAddress(rank), - ObjectAddress(expected_shape), - ObjectAddress(optional_absent), - ), - ), - If( - IfSection( - Is(helper_args, NIL), - [ - *self._decref_all([*owned_args, helper, runtime_module]), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(packed, PyObject_CallObject(helper, helper_args)), - Py_DECREF(helper_args), - *self._decref_all([*owned_args, helper, runtime_module]), - If(IfSection(Is(packed, NIL), [Return(self._error_exit_code)])), - AliasAssign(descriptor_item, PyTuple_GetItem(packed, convert_to_literal(0))), - If(IfSection(Is(descriptor_item, NIL), [Py_DECREF(packed), Return(self._error_exit_code)])), - *self._assign_native_array_descriptor_pointer( - descriptor_pointer, - descriptor_item, - packed, - allow_none=policy.optional_absent, - ), - ] - ) - if presence_pointer is not None: - presence_item = self._new_python_object(f"{subject.name}_presence_item") - body.extend( - [ - AliasAssign(presence_item, PyTuple_GetItem(packed, convert_to_literal(1))), - If(IfSection(Is(presence_item, NIL), [Py_DECREF(packed), Return(self._error_exit_code)])), - *self._assign_native_array_descriptor_pointer( - presence_pointer, - presence_item, - packed, - allow_none=True, - ), - ] - ) - body.append(Py_DECREF(packed)) - return body - - def _native_array_descriptor_argument_body( - self, - subject, - policy, - collect_arg, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - presence_pointer, - ): - """Build CPython calls into the runtime descriptor-argument packer.""" - runtime_module = self._new_python_object(f"{subject.name}_runtime_handles") - helper = self._new_python_object(f"{subject.name}_descriptor_helper") - descriptor_kind = self._new_python_object(f"{subject.name}_descriptor_kind") - dtype = self._new_python_object(f"{subject.name}_dtype") - rank = self._new_python_object(f"{subject.name}_rank") - expected_shape = self._new_python_object(f"{subject.name}_shape") - optional_absent = self._new_python_object(f"{subject.name}_optional") - helper_args = self._new_python_object(f"{subject.name}_descriptor_helper_args") - packed = self._new_python_object(f"{subject.name}_descriptor_fields") - owned_args = [descriptor_kind, dtype, rank, expected_shape, optional_absent] - body = [ - AliasAssign(runtime_module, PyImport_ImportModule(CStrStr(convert_to_literal("x2py.runtime.handles")))), - If(IfSection(Is(runtime_module, NIL), [Return(self._error_exit_code)])), - AliasAssign( - helper, - PyObject_GetAttrString( - runtime_module, - CStrStr(convert_to_literal("_native_array_descriptor_argument_for_binding_positional")), - ), - ), - If(IfSection(Is(helper, NIL), [Py_DECREF(runtime_module), Return(self._error_exit_code)])), - AliasAssign(descriptor_kind, PyUnicode_FromString(CStrStr(convert_to_literal(policy.descriptor_kind)))), - *self._native_array_descriptor_dtype_object(subject, dtype), - AliasAssign(rank, PyLong_FromLong(convert_to_literal(subject.rank, dtype=CNativeInt()))), - *self._native_array_descriptor_shape_object(subject, expected_shape), - AliasAssign( - optional_absent, - PyLong_FromLong(convert_to_literal(1 if policy.optional_absent else 0, dtype=CNativeInt())), - ), - ] - body.extend(self._return_if_any_native_array_helper_arg_failed(runtime_module, helper, owned_args)) - body.extend( - [ - AliasAssign( - helper_args, - PyTuple_Pack( - ObjectAddress(collect_arg), - ObjectAddress(descriptor_kind), - ObjectAddress(dtype), - ObjectAddress(rank), - ObjectAddress(expected_shape), - ObjectAddress(optional_absent), - ), - ), - If( - IfSection( - Is(helper_args, NIL), - [ - *self._decref_all([*owned_args, helper, runtime_module]), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(packed, PyObject_CallObject(helper, helper_args)), - Py_DECREF(helper_args), - *self._decref_all([*owned_args, helper, runtime_module]), - If(IfSection(Is(packed, NIL), [Return(self._error_exit_code)])), - ] - ) - body.extend( - self._assign_native_array_cfi_descriptor( - subject, - policy, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - presence_pointer, - packed, - ) - ) - body.append(Py_DECREF(packed)) - return body - - def _assign_native_array_cfi_descriptor( - self, - subject, - policy, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - presence_pointer, - packed, - ): - """Establish one call-local CFI descriptor from validated runtime fields.""" - field_targets = [base_address, element_length, descriptor_rank] - for lower_bound, extent, stride in zip(lower_bounds, extents, strides, strict=True): - field_targets.extend((lower_bound, extent, stride)) - items = [ - self._new_python_object(f"{subject.name}_descriptor_field_{index}") for index in range(len(field_targets)) - ] - body = [] - for index, item in enumerate(items): - body.extend( - [ - AliasAssign(item, PyTuple_GetItem(packed, convert_to_literal(index))), - If(IfSection(Is(item, NIL), [Py_DECREF(packed), Return(self._error_exit_code)])), - ] - ) - - if presence_pointer is not None: - presence_item = self._new_python_object(f"{subject.name}_presence_item") - body.extend( - [ - AliasAssign(presence_item, PyTuple_GetItem(packed, convert_to_literal(len(items)))), - If(IfSection(Is(presence_item, NIL), [Py_DECREF(packed), Return(self._error_exit_code)])), - *self._assign_native_array_descriptor_pointer( - presence_pointer, - presence_item, - packed, - allow_none=True, - ), - ] - ) - - present_body = self._establish_native_array_cfi_descriptor( - subject, - policy, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - items, - packed, - ) - if policy.optional_absent: - body.append( - If( - IfSection(Is(items[0], Py_None), [AliasAssign(descriptor_pointer, NIL)]), - IfSection(convert_to_literal(True), present_body), - ) - ) - else: - body.extend(present_body) - return body - - def _establish_native_array_cfi_descriptor( - self, - subject, - policy, - descriptor_storage, - descriptor_pointer, - base_address, - element_length, - descriptor_rank, - lower_bounds, - extents, - strides, - establish_status, - items, - packed, - ): - """Convert present descriptor fields and initialize standard CFI storage.""" - body = self._assign_native_array_descriptor_pointer(base_address, items[0], packed) - for target, item in zip( - [ - element_length, - descriptor_rank, - *[value for triple in zip(lower_bounds, extents, strides, strict=True) for value in triple], - ], - items[1:], - strict=True, - ): - body.extend(self._assign_native_array_actual_int64(target, item, packed)) - body.extend( - [ - If( - IfSection( - Ne(descriptor_rank, convert_to_literal(subject.rank, dtype=NumpyInt64Type())), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr(convert_to_literal("native array descriptor rank changed after validation")), - ), - Py_DECREF(packed), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign( - descriptor_pointer, - PointerCast(ObjectAddress(descriptor_storage), descriptor_pointer), - ), - Assign( - establish_status, - CFIDescriptorEstablish( - descriptor_pointer, - subject.dtype, - subject.rank, - base_address=base_address, - attribute=policy.descriptor_kind, - element_length=element_length, - extents=extents, - ), - ), - If( - IfSection( - Ne(establish_status, Variable(CNativeInt(), "CFI_SUCCESS")), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr(convert_to_literal("failed to establish native array C descriptor")), - ), - Py_DECREF(packed), - Return(self._error_exit_code), - ], - ) - ), - ] - ) - for index, (lower_bound, stride) in enumerate(zip(lower_bounds, strides, strict=True)): - body.extend( - [ - Assign( - CFIDescriptorDimField(descriptor_pointer, index, "lower_bound", NumpyInt64Type()), - lower_bound, - ), - Assign( - CFIDescriptorDimField(descriptor_pointer, index, "sm", NumpyInt64Type()), - stride, - ), - ] - ) - return body - - def _native_array_descriptor_dtype_object(self, subject, dtype): - """Return body nodes that create the runtime dtype argument object.""" - dtype_name = self._native_array_descriptor_dtype_name(subject) - if dtype_name is None: - return [AliasAssign(dtype, Py_None), Py_INCREF(Py_None)] - return [AliasAssign(dtype, PyUnicode_FromString(CStrStr(convert_to_literal(dtype_name))))] - - @staticmethod - def _native_array_descriptor_dtype_name(subject): - """Return the NumPy dtype spelling accepted by the runtime handle helper.""" - dtype = getattr(subject, "dtype", None) - if isinstance(dtype, CharType): - return None - name = str(dtype) - return name.removeprefix("numpy.") - - def _native_array_descriptor_shape_object(self, subject, expected_shape): - """Return body nodes that create the runtime expected-shape argument.""" - fixed_shape = self._rank_one_fixed_extent(subject) - if fixed_shape is None: - return [AliasAssign(expected_shape, Py_None), Py_INCREF(Py_None)] - return [ - AliasAssign(expected_shape, PyLong_FromLong(convert_to_literal(fixed_shape, dtype=CNativeInt()))), - ] - - @staticmethod - def _rank_one_fixed_extent(subject): - """Return fixed rank-one extent when it can be represented as a Python int argument.""" - shape = getattr(subject, "alloc_shape", None) - if getattr(subject, "rank", None) != 1 or not shape or len(shape) != 1: - return None - value = getattr(shape[0], "python_value", shape[0]) - if isinstance(value, bool) or not isinstance(value, int): - return None - return int(value) - - def _return_if_any_native_array_helper_arg_failed( - self, - runtime_module, - helper, - owned_args, - *, - failure_cleanup=(), - ): - """Return if any Python object needed for the helper call failed to allocate.""" - condition = None - for item in owned_args: - item_failed = Is(item, NIL) - condition = item_failed if condition is None else Or(condition, item_failed) - return [ - If( - IfSection( - condition, - [ - *self._decref_non_null(owned_args), - Py_DECREF(helper), - Py_DECREF(runtime_module), - *failure_cleanup, - Return(self._error_exit_code), - ], - ) - ) - ] - - def _assign_native_array_descriptor_pointer(self, pointer_var, item, packed, *, allow_none=False): - """Convert one Python integer-or-None descriptor ABI field to void*.""" - convert_field = [ - AliasAssign(pointer_var, PyLong_AsVoidPtr(item)), - If( - IfSection( - And(Is(pointer_var, NIL), PyErr_Occurred()), - [Py_DECREF(packed), Return(self._error_exit_code)], - ) - ), - ] - if not allow_none: - return convert_field - return [ - If( - IfSection(Is(item, Py_None), [AliasAssign(pointer_var, NIL)]), - IfSection(convert_to_literal(True), convert_field), - ) - ] - - @staticmethod - def _decref_all(items): - """Return Py_DECREF calls for known-owned PyObject references.""" - return [Py_DECREF(item) for item in items] - - @staticmethod - def _decref_non_null(items): - """Return Py_DECREF calls guarded for partially-created owned references.""" - return [If(IfSection(IsNot(item, NIL), [Py_DECREF(item)])) for item in items] - - @staticmethod - def _native_array_descriptor_argument_type(policy): - """Return the Bind-C tuple shape selected for a handle descriptor argument.""" - return native_array_descriptor_argument_type(policy) - - @staticmethod - def _validate_descriptor_array_interop_policy(subject, policy) -> None: - """Require descriptor ABI dispatch to carry completed native handle policy.""" - handle_policy = subject.native_array_handle_policy - if handle_policy is None: - raise ValueError(f"Descriptor array interop for {subject.name!r} is missing completed handle policy") - if policy.descriptor_kind != handle_policy.descriptor_kind or policy.handle_kind != handle_policy.handle_kind: - raise ValueError( - f"Descriptor array interop for {subject.name!r} disagrees with completed handle policy: " - f"{policy.descriptor_kind}/{policy.handle_kind} != " - f"{handle_policy.descriptor_kind}/{handle_policy.handle_kind}" - ) - - def _native_array_descriptor_view_body(self, descriptor_pointer, descriptor_result, *, rank, cleanup=()): - """Decode a ``CFI_cdesc_t*`` into the runtime descriptor-view mapping shape.""" - if not isinstance(rank, int) or rank < 0: - raise ValueError("native array descriptor view rank must be a non-negative integer") - cleanup = tuple(cleanup) - dim_list = self._new_python_object(f"{descriptor_result.name}_dim") - body = [ - If( - IfSection( - Is(descriptor_pointer, NIL), - [ - PyErr_SetString( - PyRuntimeError, - CStrStr(convert_to_literal("native array descriptor pointer is NULL")), - ), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(descriptor_result, PyDict_New()), - If(IfSection(Is(descriptor_result, NIL), [*self._decref_non_null(cleanup), Return(self._error_exit_code)])), - ] - body.extend( - self._set_native_array_descriptor_view_item( - descriptor_result, - "base_addr", - CFIDescriptorField(descriptor_pointer, "base_addr", BindCPointer()), - PyLong_FromVoidPtr, - cleanup, - ) - ) - for name in ("elem_len", "rank"): - body.extend( - self._set_native_array_descriptor_view_item( - descriptor_result, - name, - CFIDescriptorField(descriptor_pointer, name, NumpyInt64Type()), - PyLong_FromLongLong, - cleanup, - ) - ) - body.extend( - [ - AliasAssign(dim_list, PyList_New(convert_to_literal(0, dtype=NumpyInt64Type()))), - If( - IfSection( - Is(dim_list, NIL), - [Py_DECREF(descriptor_result), *self._decref_non_null(cleanup), Return(self._error_exit_code)], - ) - ), - ] - ) - for index in range(rank): - body.extend( - self._append_native_array_descriptor_dimension( - descriptor_pointer, - dim_list, - index, - [descriptor_result, *cleanup], - ) - ) - body.extend( - self._set_native_array_descriptor_view_pyobject( - descriptor_result, - "dim", - dim_list, - cleanup, - decref_value=True, - ) - ) - return body - - def _append_native_array_descriptor_dimension(self, descriptor_pointer, dim_list, index, cleanup): - """Append one decoded ``CFI_cdesc_t.dim[index]`` mapping.""" - dim = self._new_python_object(f"{dim_list.name}_{index}") - body = [ - AliasAssign(dim, PyDict_New()), - If( - IfSection( - Is(dim, NIL), - [ - Py_DECREF(dim_list), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - ] - for name in ("lower_bound", "extent", "sm"): - body.extend( - self._set_native_array_descriptor_view_item( - dim, - name, - CFIDescriptorDimField(descriptor_pointer, index, name, NumpyInt64Type()), - PyLong_FromLongLong, - [dim_list, *cleanup], - ) - ) - body.extend( - [ - If( - IfSection( - Lt(PyList_Append(dim_list, dim), convert_to_literal(0)), - [ - Py_DECREF(dim), - Py_DECREF(dim_list), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - Py_DECREF(dim), - ] - ) - return body - - def _set_native_array_descriptor_view_item(self, mapping, key_text, value_expr, converter, cleanup): - """Convert one descriptor field and store it in a Python mapping.""" - value = self._new_python_object(f"{mapping.name}_{key_text}_value") - return [ - AliasAssign(value, converter(value_expr)), - If( - IfSection( - Is(value, NIL), - [ - Py_DECREF(mapping), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - *self._set_native_array_descriptor_view_pyobject(mapping, key_text, value, cleanup, decref_value=True), - ] - - def _set_native_array_descriptor_view_pyobject(self, mapping, key_text, value, cleanup, *, decref_value): - """Store one owned Python object in a descriptor-view mapping.""" - key = self._new_python_object(f"{mapping.name}_{key_text}_key") - failure_cleanup = [Py_DECREF(value)] if decref_value else [] - return [ - AliasAssign(key, PyUnicode_FromString(CStrStr(convert_to_literal(key_text)))), - If( - IfSection( - Is(key, NIL), - [ - *failure_cleanup, - Py_DECREF(mapping), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - If( - IfSection( - Lt(PyDict_SetItem(mapping, key, value), convert_to_literal(0)), - [ - Py_DECREF(key), - *failure_cleanup, - Py_DECREF(mapping), - *self._decref_non_null(cleanup), - Return(self._error_exit_code), - ], - ) - ), - Py_DECREF(key), - *((Py_DECREF(value),) if decref_value else ()), - ] - - def _build_policy_property_setter(self, expr, class_type, name): - """Build a writable or rejecting setter from completed accessor policy.""" - self._error_exit_code = convert_to_literal(-1, dtype=CNativeInt()) - setter_name = self.scope.get_new_name(f"{class_type.name}_{name}_setter", object_type="wrapper") - setter_scope = self.scope.new_child_scope(setter_name, "function") - self.scope = setter_scope - setter_args = [ - self._new_python_object("self_obj", dtype=class_type), - self._new_python_object(f"{name}_obj"), - setter_scope.get_temporary_variable(VoidType(), memory_handling="alias"), - ] - setter_body = self._PROPERTY_SETTER_POLICY_DISPATCHER.dispatch( - self, - expr, - expr.setter_policy, - setter_args, - ) - setter_result = FunctionDefResult(setter_scope.get_temporary_variable(CNativeInt())) - self.exit_scope() - return PyFunctionDef( - setter_name, - [FunctionDefArgument(arg) for arg in setter_args], - setter_body, - setter_result, - original_function=expr, - scope=setter_scope, - ) - - def _build_writable_property_setter(self, expr, _decision, setter_args): - """Build a property setter which calls the completed native setter.""" - if expr.setter is None: - raise ValueError(f"Writable property {expr.python_name!r} has no generated native setter") - original_args = expr.setter.arguments - self_arg, set_val_arg = original_args - for argument in original_args: - self.scope.insert_symbol(argument.var.name) - self.scope.insert_symbol(self_arg.var.original_var.name) - self.scope.insert_symbol(set_val_arg.var.original_var.name) - self._python_object_map[self_arg] = setter_args[0] - self._python_object_map[set_val_arg] = setter_args[1] - wrapped_args = [self._visit(argument) for argument in original_args] - arg_code = [line for argument in wrapped_args for line in argument["body"]] - func_call_args = [converted for argument in wrapped_args for converted in argument["args"]] - return [ - *arg_code, - expr.setter(*func_call_args), - *self._save_referenced_objects(expr.setter, setter_args), - Return(convert_to_literal(0, dtype=CNativeInt())), - ] - - def _build_blocked_property_setter(self, _expr, _decision, _setter_args): - """Build the stable Python error for a policy-blocked property write.""" - return [ - PyErr_SetString( - PyAttributeError, - CStrStr(convert_to_literal("Can't reallocate memory via Python interface.")), - ), - Return(self._error_exit_code), - ] - - def _visit_ClassDef(self, expr): - """ - Get the code which exposes a class definition to Python. - - Get the code which exposes a class definition to Python. - - Parameters - ---------- - expr : ClassDef - The class definition being wrapped. - - Returns - ------- - PyClassDef - The wrapped class definition. - """ - name = expr.name - python_name = expr.scope.get_python_name(name) - - orig_cls_dtype = expr.scope.parent_scope.cls_constructs[python_name] - wrapped_class = self._python_object_map[expr] - - orig_scope = expr.scope - has_initialiser = False - - for f in expr.methods: - if not f.is_semantic: - continue - if f.is_private: - continue - orig_f = getattr(f, "original_function", f) - name = orig_f.name - python_name = orig_scope.get_python_name(name) - if python_name == "__del__": - wrapped_class.add_new_method(self._get_class_destructor(f, orig_cls_dtype, wrapped_class.scope)) - elif python_name == "__init__": - has_initialiser = True - wrapped_class.add_new_method(self._get_class_initialiser(f, orig_cls_dtype)) - elif python_name in (*magic_binary_funcs, "__len__"): - wrapped_class.add_new_magic_method(self._visit(f)) - else: - wrapped_class.add_new_method(self._visit(f)) - - for i in expr.overload_sets: - if i.is_private: - continue - for f in i.functions: - self._visit(f) - wrapped_overload_set = self._visit(i) - if i.name in magic_overload_funcs: - wrapped_class.add_new_magic_method(wrapped_overload_set) - else: - wrapped_class.add_new_overload_set(wrapped_overload_set) - - wrapped_class.add_alloc_method(self._get_class_allocator(orig_cls_dtype, expr.new_func)) - - for a in expr.attributes: - if isinstance(a.class_type, TupleType): - raise NotImplementedError("Tuples cannot yet be exposed to Python.") - wrapped_class.add_property(self._visit(a)) - - if not has_initialiser: - if self._suppresses_default_class_initialiser(expr): - wrapped_class.add_new_method(self._get_blocked_class_initialiser(wrapped_class, orig_cls_dtype)) - else: - wrapped_class.add_new_method(self._get_default_class_initialiser(wrapped_class, orig_cls_dtype)) - - return wrapped_class - - def _visit_Import(self, expr): - """ - Examine an Import statement and collect any relevant objects. - - Examine an Import statement used in the module being wrapped. If it imports a class - from a module then a PyClassDef is added to the scope imports to ensure that its - description is available for functions wishing to use this type for an argument - or return value. - - Parameters - ---------- - expr : Import - The import found in the module being wrapped. - - Returns - ------- - Import | None - The import needed in the wrapper, or None if none is necessary. - """ - if expr.source_module is None: - return None - - # Imports do not use collision handling as there is not enough context available. - # This should be fixed when stub files and proper pickling is added - import_wrapper = False - import_scope = None - for as_name in expr.target: - t = as_name.object - if isinstance(t, ClassDef): - if import_scope is None: - import_scope = Scope( - name=expr.source_module.name, - used_symbols=expr.source_module.scope.local_used_symbols.copy(), - original_symbols=expr.source_module.scope.python_names.copy(), - naming_policy=self.scope.naming_policy, - symbol_language=self.scope.symbol_language, - scope_type="module", - ) - name = t.scope.get_python_name(t.name) - struct_name = import_scope.get_new_name(f"Py{name}Object") - dtype = DataTypeFactory(struct_name, struct_name, BaseClass=WrapperCustomDataType)() - type_name = import_scope.get_new_name(f"Py{name}Type") - wrapped_class = PyClassDef( - t, - struct_name, - type_name, - Scope( - name=name, - naming_policy=self.scope.naming_policy, - symbol_language=self.scope.symbol_language, - scope_type="class", - ), - class_type=dtype, - ) - self._python_object_map[t] = wrapped_class - self._python_object_map[t.class_type] = dtype - self.scope.imports["classes"][name] = wrapped_class - import_wrapper = True - - if import_wrapper: - wrapper_name = f"{expr.source}_wrapper" - mod_spoof_scope = Scope( - name=expr.source_module.name, - naming_policy=self.scope.naming_policy, - symbol_language=self.scope.symbol_language, - scope_type="module", - ) - mod_import_func = FunctionDef( - mod_spoof_scope.get_new_name("import"), - (), - (), - FunctionDefResult(Variable(CNativeInt(), "_", is_temp=True)), - ) - mod_spoof = PyModule( - expr.source_module.name, - (), - (), - scope=mod_spoof_scope, - module_def_name=mod_spoof_scope.get_new_name("module"), - import_func=mod_import_func, - ) - return Import(wrapper_name, AsName(mod_spoof, expr.source), mod=mod_spoof) - return None - - # ------------------------------------------------------------------ - # Datatype conversion - # ------------------------------------------------------------------ - - def _convert_argument(self, orig_var, collect_arg, bound_argument, is_bind_c_argument, *, arg_var=None): - """ - Extract the C-compatible FunctionDefArgument from the PythonObject. - - Extract the C-compatible FunctionDefArgument from the PythonObject. - The C-compatible argument is extracted from collect_arg which holds a Python - object into arg_var. - - The explicit datatype dispatch table selects the conversion helper. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - bound_argument : bool - True if the argument is the self argument of a class method. False otherwise. - This should always be False for this function. - - is_bind_c_argument : bool - True if the argument was defined in a BindCFunctionDef. False otherwise. - - arg_var : Variable | IndexedElement, optional - A variable or an element of the variable representing the argument that - will be passed to the low-level function call. - - Returns - ------- - dict - A dictionary describing the objects necessary to access the argument. - """ - if orig_var.array_interop_policy is not None: - return self._ARRAY_INTEROP_POLICY_DISPATCHER.dispatch( - self, - orig_var, - orig_var.array_interop_policy, - "argument", - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - ) - return self._bind_non_array_argument(orig_var, collect_arg, bound_argument, is_bind_c_argument, arg_var=arg_var) - - def _bind_data_buffer_argument( - self, - subject, - _policy, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Convert an ordinary array argument through the data-buffer ABI.""" - return self._bind_non_array_argument(subject, collect_arg, bound_argument, is_bind_c_argument, arg_var=arg_var) - - def _bind_descriptor_argument( - self, - subject, - policy, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Convert a native array handle argument through descriptor ABI.""" - self._validate_descriptor_array_interop_policy(subject, policy) - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - subject, - subject.native_array_handle_policy, - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - ) - - def _bind_non_array_argument(self, orig_var, collect_arg, bound_argument, is_bind_c_argument, *, arg_var=None): - """Convert a Python argument without native descriptor-handle routing.""" - return self._PYTHON_BARRIER_DISPATCHER.dispatch( - self, - orig_var, - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - ) - - def _convert_python_scalar_value_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Read a Python scalar value for the completed scalar barrier.""" - if getattr(orig_var, "is_optional", False) or (decision.descriptor_boundary and decision.nullable): - return self._convert_nullable_scalar_argument( - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - ) - return self._convert_scalar_argument( - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - bind_c_stack_alias=decision.storage_mode is StorageMode.ALIAS, - ) - - def _convert_nullable_scalar_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Convert an optional or descriptor scalar through one nullable C pointer path.""" - if bound_argument or arg_var is not None: - raise ValueError(f"Nullable scalar argument {orig_var.name!r} requires a standalone value slot") - value_var = orig_var.clone( - self.scope.get_new_name(f"{orig_var.name}_value"), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=StorageMode.STACK.value, - ) - self.scope.insert_variable(value_var) - converted = self._convert_scalar_argument( - orig_var, - decision, - collect_arg, - False, - is_bind_c_argument, - arg_var=value_var, - bind_c_stack_alias=False, - ) - pointer_var = Variable( - BindCPointer(), - self.scope.get_new_name(f"{orig_var.name}_nullable"), - memory_handling=StorageMode.ALIAS.value, - ) - self.scope.insert_variable(pointer_var) - if self._uses_optional_scalar_descriptor_presence(orig_var, decision): - presence_var = Variable( - BindCPointer(), - self.scope.get_new_name(f"{orig_var.name}_present"), - memory_handling=StorageMode.ALIAS.value, - ) - self.scope.insert_variable(presence_var) - descriptor_var = Variable( - BindCScalarDescriptorType(), - self.scope.get_new_name(f"{orig_var.name}_descriptor"), - shape=(convert_to_literal(2),), - ) - self.scope.insert_symbolic_alias(IndexedElement(descriptor_var, convert_to_literal(0)), pointer_var) - self.scope.insert_symbolic_alias(IndexedElement(descriptor_var, convert_to_literal(1)), presence_var) - return { - "body": [*converted["body"], Assign(pointer_var, ObjectAddress(value_var))], - "args": [descriptor_var], - "clean_up": [], - "default_init": [AliasAssign(pointer_var, NIL), AliasAssign(presence_var, NIL)], - "nullable_scalar": True, - "optional_scalar_descriptor": True, - "pre_check_body": [ - If(IfSection(IsNot(collect_arg, NIL), [Assign(presence_var, ObjectAddress(value_var))])) - ], - } - return { - "body": [*converted["body"], Assign(pointer_var, ObjectAddress(value_var))], - "args": [pointer_var], - "clean_up": [], - "nullable_scalar": True, - } - - @staticmethod - def _uses_optional_scalar_descriptor_presence(orig_var, decision): - """Return whether this scalar descriptor must preserve omitted vs None.""" - return CPythonBindingGenerator._is_optional_scalar_descriptor_var(orig_var, decision) - - @staticmethod - def _is_optional_scalar_descriptor_var(orig_var, decision=None): - """Return whether ``orig_var`` is an optional nullable scalar descriptor.""" - if decision is None: - decision = getattr(orig_var, "ownership_decision", None) - return bool( - getattr(orig_var, "is_optional", False) - and decision is not None - and decision.kind is ObjectKind.SCALAR - and decision.descriptor_boundary - and decision.nullable - ) - - def _convert_python_scalar_storage_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Validate rank-0 NumPy scalar storage for the completed barrier.""" - assert not bound_argument - arg_var = self._scalar_argument_target( - orig_var, - is_bind_c_argument, - arg_var=arg_var, - bind_c_stack_alias=False, - ) - read_initial = decision.codegen_action is not CodegenAction.IDENTITY_OUTPUT - return self._convert_scalar_storage_argument( - orig_var, decision, collect_arg, arg_var, read_initial=read_initial - ) - - def _convert_python_raw_address_argument( - self, - orig_var, - _decision, - collect_arg, - _bound_argument, - _is_bind_c_argument, - *, - arg_var=None, - ): - """Read a Python raw address value for the completed barrier.""" - if arg_var is not None: - raise ValueError(f"Raw address argument {orig_var.name!r} cannot reuse an existing conversion target") - return self._convert_raw_address_argument(orig_var, collect_arg) - - def _convert_python_string_storage_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - _is_bind_c_argument, - *, - arg_var=None, - ): - """Validate rank-0 NumPy bytes storage for a fixed-length string contract.""" - assert not bound_argument - if arg_var is not None: - raise ValueError(f"String storage argument {orig_var.name!r} cannot reuse an existing conversion target") - data_var = Variable( - VoidType(), - self.scope.get_new_name(f"{orig_var.name}_data"), - memory_handling="alias", - ) - self.scope.insert_variable(data_var) - pyarray = PointerCast(collect_arg, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - pyarray_address = ObjectAddress(pyarray) - check = pyarray_check( - CStrStr(convert_to_literal(orig_var.name)), - collect_arg, - numpy_string_type, - convert_to_literal(0), - no_order_check, - convert_to_literal(False), - ) - itemsize = cast_to(PyArray_ITEMSIZE(pyarray_address), NumpyInt64Type()) - body = [ - If(IfSection(Not(check), [Return(self._error_exit_code)])), - *self._array_itemsize_validation(orig_var, itemsize, collect_arg), - ] - if decision.mutates_native: - body.extend(self._writable_array_access_validation(orig_var, decision, collect_arg)) - else: - body.extend(self._readable_array_access_validation(orig_var, decision, collect_arg)) - body.append(AliasAssign(data_var, PyArray_DATA(pyarray_address))) - return {"body": body, "args": [data_var], "clean_up": []} - - def _convert_scalar_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - bind_c_stack_alias, - ): - """ - Extract the C-compatible scalar FunctionDefArgument from the PythonObject. - - Extract the C-compatible scalar FunctionDefArgument from the PythonObject. - The C-compatible argument is extracted from collect_arg which holds a Python - object into arg_var. - - The extraction is done by calling a function from the C-Python API. These functions - are indexed in the dictionary `py_to_c_registry`. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - bound_argument : bool - True if the argument is the self argument of a class method. False otherwise. - This should always be False for this function. - - is_bind_c_argument : bool - True if the argument was defined in a BindCFunctionDef. False otherwise. - - arg_var : Variable | IndexedElement - A variable or an element of the variable representing the argument that - will be passed to the low-level function call. - - Returns - ------- - dict - A dictionary describing the objects necessary to access the argument. - """ - assert not bound_argument - supplied_arg_var = arg_var is not None - arg_var = self._scalar_argument_target( - orig_var, - is_bind_c_argument, - arg_var=arg_var, - bind_c_stack_alias=bind_c_stack_alias, - ) - - dtype = orig_var.dtype - try: - cast_function = py_to_c_registry[(dtype.primitive_type, dtype.precision)] - except KeyError: - raise TypeError(f"No Python-to-C cast registered for {dtype}") from None - cast_func = FunctionDef( - name=cast_function, - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(dtype, name="v")), - ) - - body = [Assign(arg_var, cast_func(collect_arg))] - - if getattr(orig_var, "is_optional", False) and not supplied_arg_var: - memory_var = self.scope.get_temporary_variable( - arg_var, - name=arg_var.name + "_memory", - is_optional=False, - memory_handling="stack", - ) - body.insert(0, AliasAssign(arg_var, memory_var)) - - return {"body": body, "args": [arg_var]} - - def _scalar_argument_target(self, orig_var, is_bind_c_argument, *, arg_var, bind_c_stack_alias): - """Create or reuse the scalar C argument target selected by Python-barrier dispatch.""" - if arg_var is not None: - return arg_var - class_type = orig_var.class_type - if isinstance(class_type, FinalType): - class_type = class_type.underlying_type - kwargs = { - "new_class": Variable, - "is_argument": False, - "class_type": class_type, - } - if getattr(orig_var, "is_optional", False): - kwargs["memory_handling"] = "alias" - elif is_bind_c_argument and bind_c_stack_alias: - kwargs["memory_handling"] = "stack" - arg_var = orig_var.clone( - self.scope.get_expected_name(orig_var.name), - **kwargs, - ) - self.scope.insert_variable(arg_var, orig_var.name) - return arg_var - - def _convert_scalar_storage_argument(self, orig_var, decision, collect_arg, arg_var, *, read_initial=True): - """Use caller-supplied rank-0 NumPy storage for a scalar contract.""" - try: - type_ref = numpy_dtype_registry[orig_var.dtype] - except KeyError: - raise TypeError(f"Can't check the type of scalar storage {orig_var.dtype}") from None - pyarray = PointerCast(collect_arg, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - check = pyarray_check( - CStrStr(convert_to_literal(orig_var.name)), - collect_arg, - type_ref, - convert_to_literal(0), - no_order_check, - convert_to_literal(False), - ) - data_value = PointerCast(PyArray_DATA(ObjectAddress(pyarray)), arg_var) - body = [If(IfSection(Not(check), [Return(self._error_exit_code)]))] - if decision.mutates_native: - body.extend(self._writable_array_access_validation(orig_var, decision, collect_arg)) - else: - body.extend(self._readable_array_access_validation(orig_var, decision, collect_arg)) - if orig_var.is_optional: - body.append(AliasAssign(arg_var, data_value)) - return {"body": body, "args": [arg_var], "clean_up": []} - if read_initial: - body.append(Assign(arg_var, data_value)) - clean_up = [Assign(data_value, arg_var)] if decision.mutates_native else [] - return {"body": body, "args": [arg_var], "clean_up": clean_up} - - def _convert_raw_address_argument(self, orig_var, collect_arg): - """Convert a Python integer address into a raw C pointer argument.""" - address_var = Variable( - BindCPointer(), - self.scope.get_new_name(f"{orig_var.name}_addr"), - memory_handling="alias", - ) - self.scope.insert_variable(address_var) - body = [ - AliasAssign(address_var, PyLong_AsVoidPtr(collect_arg)), - If(IfSection(And(Is(address_var, NIL), PyErr_Occurred()), [Return(self._error_exit_code)])), - ] - return {"body": body, "args": [ObjectAddress(address_var)], "clean_up": []} - - @staticmethod - def _uses_python_raw_address(var) -> bool: - """Return whether completed policy extracts this argument as an address.""" - return ownership_decision_for_codegen_variable(var).python_barrier_action is PythonBarrierAction.RAW_ADDRESS - - @staticmethod - def _uses_python_scalar_storage(var) -> bool: - """Return whether completed policy expects rank-0 NumPy scalar storage.""" - return ownership_decision_for_codegen_variable(var).python_barrier_action is PythonBarrierAction.SCALAR_STORAGE - - def _convert_python_wrapper_instance_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """ - Extract the C-compatible class FunctionDefArgument from the PythonObject. - - Extract the C-compatible class FunctionDefArgument from the PythonObject. - The C-compatible argument is extracted from collect_arg which holds a Python - object into arg_var. - - The extraction is done by accessing the pointer from the `instance` attribute of the - X2py generated class definition. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - bound_argument : bool - True if the argument is the self argument of a class method. False otherwise. - This should always be False for this function. - - is_bind_c_argument : bool - True if the argument was defined in a BindCFunctionDef. False otherwise. - - arg_var : Variable | IndexedElement, optional - A variable or an element of the variable representing the argument that - will be passed to the low-level function call. - - Returns - ------- - dict - A dictionary describing the objects necessary to access the argument. - """ - if arg_var is None: - kwargs = { - "is_argument": False, - "memory_handling": decision.boundary_storage_mode.value, - } - if is_bind_c_argument: - kwargs["class_type"] = VoidType() - - arg_var = orig_var.clone( - self.scope.get_expected_name(orig_var.name), - new_class=Variable, - **kwargs, - ) - self.scope.insert_variable(arg_var, orig_var.name) - - dtype = orig_var.dtype - python_cls_base = self.scope.find(dtype.name, "classes", raise_if_missing=True) - scope = python_cls_base.scope - attribute = scope.find("instance", "variables", raise_if_missing=True) - if bound_argument: - cast_type = collect_arg - cast = [] - else: - cast_type = Variable( - self._python_object_map[dtype], - self.scope.get_new_name(collect_arg.name), - memory_handling="alias", - cls_base=self.scope.find(dtype.name, "classes", raise_if_missing=True), - ) - self.scope.insert_variable(cast_type) - cast = [AliasAssign(cast_type, PointerCast(collect_arg, cast_type))] - c_res = attribute.clone(attribute.name, new_class=DottedVariable, lhs=cast_type) - cast_c_res = PointerCast(c_res, orig_var) - cast.append(AliasAssign(arg_var, cast_c_res)) - return {"body": cast, "args": [arg_var]} - - def _convert_python_array_storage_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """ - Extract the C-compatible NumPy array FunctionDefArgument from the PythonObject. - - Extract the C-compatible NumPy array FunctionDefArgument from the PythonObject. - The C-compatible argument is extracted from collect_arg which holds a Python - object into arg_var. - - The extraction is done by calling the function `pyarray_to_ndarray` from the stdlib. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - bound_argument : bool - True if the argument is the self argument of a class method. False otherwise. - This should always be False for this function. - - is_bind_c_argument : bool - True if the argument was defined in a BindCFunctionDef. False otherwise. - - arg_var : Variable | IndexedElement, optional - A variable or an element of the variable representing the argument that - will be passed to the low-level function call. - - Returns - ------- - dict - A dictionary describing the objects necessary to access the argument. - """ - assert arg_var is None - parts = self._get_array_parts(orig_var, collect_arg) - body = parts["body"] - shape = parts["shape"] - itemsize = parts["itemsize"] - strides = parts["strides"] - ubounds = parts["ubounds"] - descriptor_rank = self._array_descriptor_rank(orig_var) - shape_elems = [IndexedElement(shape, i) for i in range(descriptor_rank)] - stride_elems = [IndexedElement(strides, i) for i in range(descriptor_rank)] - ubound_elems = [IndexedElement(ubounds, i) for i in range(descriptor_rank)] - args = [parts["data"], *shape_elems, *stride_elems] - body.extend(self._array_shape_validation(orig_var, shape_elems)) - body.extend(self._array_itemsize_validation(orig_var, itemsize, collect_arg)) - body.extend(self._array_access_validation(orig_var, decision, collect_arg)) - default_body = self._array_default_initializers(parts, shape_elems, ubound_elems, stride_elems) - - if is_bind_c_argument: - arg_var = self._bind_c_array_argument_descriptor(orig_var, parts, shape_elems, ubound_elems, stride_elems) - if self._bind_c_array_argument_uses_native_handle_fallback(orig_var): - type_check_body = [] - check_func, err = self._get_type_check_condition( - collect_arg, - orig_var, - True, - type_check_body, - allow_empty_arrays=True, - ) - numpy_body = [ - *type_check_body, - If(IfSection(Not(check_func), [*err, Return(self._error_exit_code)])), - *body, - ] - native_handle_body = self._native_array_actual_argument_body( - orig_var, - decision, - collect_arg, - parts, - shape_elems, - ubound_elems, - stride_elems, - arg_var.class_type, - ) - return { - "body": [ - If( - IfSection(PyArray_Check(collect_arg), numpy_body), - IfSection(convert_to_literal(True), native_handle_body), - ) - ], - "args": [arg_var], - "default_init": default_body, - "owns_type_check": True, - } - return {"body": body, "args": [arg_var], "default_init": default_body} - - class_type = orig_var.class_type - if isinstance(class_type, FinalType): - class_type = class_type.underlying_type - arg_var = orig_var.clone( - self.scope.get_new_name(orig_var.name), - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - class_type=class_type, - ) - self.scope.insert_variable(arg_var) - if orig_var.is_optional: - sliced_arg_var = orig_var.clone( - self.scope.get_new_name(orig_var.name), - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - class_type=class_type, - ) - self.scope.insert_variable(sliced_arg_var) - else: - sliced_arg_var = orig_var.clone( - self.scope.get_expected_name(orig_var.name), - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - class_type=class_type, - ) - self.scope.insert_variable(sliced_arg_var, orig_var.name) - - body.append(Allocate(arg_var, shape=tuple(shape_elems), status="unallocated", like=args[0])) - body.append( - AliasAssign( - sliced_arg_var, - IndexedElement( - arg_var, - *[Slice(None, u, s) for s, u in zip(stride_elems, ubound_elems, strict=False)], - ), - ) - ) - - collect_arg = sliced_arg_var - if orig_var.is_optional: - optional_arg_var = sliced_arg_var.clone(self.scope.get_expected_name(orig_var.name), is_optional=True) - self.scope.insert_variable(optional_arg_var) - body.append(AliasAssign(optional_arg_var, sliced_arg_var)) - default_body.append(AliasAssign(optional_arg_var, NIL)) - collect_arg = optional_arg_var - return {"body": body, "args": [collect_arg], "default_init": default_body} - - def _array_default_initializers(self, parts, shape_elems, ubound_elems, stride_elems): - """Return null descriptor defaults for optional/nullable array arguments.""" - itemsize = parts["itemsize"] - return ( - [AliasAssign(parts["data"], NIL)] - + ([Assign(parts["rank"], 0)] if parts["rank"] is not None else []) - + ([Assign(itemsize, 0)] if itemsize is not None else []) - + [Assign(shape, 0) for shape in shape_elems] - + [Assign(ubound, 0) for ubound in ubound_elems] - + [Assign(stride, 1) for stride in stride_elems] - ) - - def _bind_c_array_argument_descriptor(self, orig_var, parts, shape_elems, ubound_elems, stride_elems): - """Pack a Python NumPy argument into the bind-C array descriptor.""" - rank = self._array_descriptor_rank(orig_var) - allows_strides = orig_var.class_type.allows_strides - descriptor_type = BindCArrayType.get_new( - rank, - allows_strides, - has_rank=self._is_assumed_rank_array(orig_var), - has_itemsize=self._is_character_array(orig_var), - ) - arg_var = Variable( - descriptor_type, - self.scope.get_new_name(orig_var.name), - shape=(convert_to_literal(len(descriptor_type)),), - ) - offset = self._bind_c_array_descriptor_prefix(arg_var, parts) - self._bind_c_array_descriptor_shape(arg_var, shape_elems, offset) - if allows_strides: - self._bind_c_array_descriptor_strides(arg_var, rank, offset, ubound_elems, stride_elems) - return arg_var - - def _bind_c_array_descriptor_prefix(self, arg_var, parts): - """Alias pointer, runtime rank, and itemsize descriptor fields.""" - offset = 1 - self.scope.insert_symbolic_alias(IndexedElement(arg_var, convert_to_literal(0)), ObjectAddress(parts["data"])) - if parts["rank"] is not None: - self.scope.insert_symbolic_alias(IndexedElement(arg_var, convert_to_literal(offset)), parts["rank"]) - offset += 1 - if parts["itemsize"] is not None: - self.scope.insert_symbolic_alias(IndexedElement(arg_var, convert_to_literal(offset)), parts["itemsize"]) - offset += 1 - return offset - - def _bind_c_array_descriptor_shape(self, arg_var, shape_elems, offset): - """Alias shape fields in a bind-C array descriptor.""" - for index, shape in enumerate(shape_elems): - self.scope.insert_symbolic_alias(IndexedElement(arg_var, convert_to_literal(index + offset)), shape) - - def _bind_c_array_descriptor_strides(self, arg_var, rank, offset, ubound_elems, stride_elems): - """Alias upper-bound and stride fields in a bind-C array descriptor.""" - for index, ubound in enumerate(ubound_elems): - self.scope.insert_symbolic_alias(IndexedElement(arg_var, convert_to_literal(index + rank + offset)), ubound) - for index, stride in enumerate(stride_elems): - self.scope.insert_symbolic_alias( - IndexedElement(arg_var, convert_to_literal(index + 2 * rank + offset)), - stride, - ) - - def _bind_c_array_argument_uses_native_handle_fallback(self, orig_var): - """Return whether a generated Bind-C array argument can also accept native handles.""" - return not ( - getattr(orig_var, "is_optional", False) - or self._is_assumed_rank_array(orig_var) - or self._is_character_array(orig_var) - ) - - def _native_array_actual_argument_body( - self, - orig_var, - decision, - collect_arg, - parts, - shape_elems, - ubound_elems, - stride_elems, - descriptor_type, - ): - """Build CPython calls into the runtime normal-array argument packer.""" - runtime_module = self._new_python_object(f"{orig_var.name}_runtime_handles") - helper = self._new_python_object(f"{orig_var.name}_array_actual_helper") - dtype = self._new_python_object(f"{orig_var.name}_dtype") - rank = self._new_python_object(f"{orig_var.name}_rank") - expected_shape = self._new_python_object(f"{orig_var.name}_shape") - expected_layout = self._new_python_object(f"{orig_var.name}_layout") - require_writeable = self._new_python_object(f"{orig_var.name}_writeable") - require_native_byte_order = self._new_python_object(f"{orig_var.name}_native_byte_order") - require_aligned = self._new_python_object(f"{orig_var.name}_aligned") - include_rank = self._new_python_object(f"{orig_var.name}_include_rank") - include_itemsize = self._new_python_object(f"{orig_var.name}_include_itemsize") - include_strides = self._new_python_object(f"{orig_var.name}_include_strides") - require_contiguous = self._new_python_object(f"{orig_var.name}_require_contiguous") - helper_args = self._new_python_object(f"{orig_var.name}_array_actual_helper_args") - packed = self._new_python_object(f"{orig_var.name}_array_actual_fields") - owned_args = [ - dtype, - rank, - expected_shape, - expected_layout, - require_writeable, - require_native_byte_order, - require_aligned, - include_rank, - include_itemsize, - include_strides, - require_contiguous, - ] - body = [ - AliasAssign(runtime_module, PyImport_ImportModule(CStrStr(convert_to_literal("x2py.runtime.handles")))), - If(IfSection(Is(runtime_module, NIL), [Return(self._error_exit_code)])), - AliasAssign( - helper, - PyObject_GetAttrString( - runtime_module, - CStrStr(convert_to_literal("_native_array_actual_argument_for_binding_positional")), - ), - ), - If(IfSection(Is(helper, NIL), [Py_DECREF(runtime_module), Return(self._error_exit_code)])), - *self._native_array_descriptor_dtype_object(orig_var, dtype), - AliasAssign(rank, PyLong_FromLong(convert_to_literal(orig_var.rank, dtype=CNativeInt()))), - *self._native_array_descriptor_shape_object(orig_var, expected_shape), - *self._native_array_actual_expected_layout_object(orig_var, expected_layout), - AliasAssign( - require_writeable, - PyLong_FromLong(convert_to_literal(1 if decision.mutates_native else 0, dtype=CNativeInt())), - ), - AliasAssign(require_native_byte_order, PyLong_FromLong(convert_to_literal(1, dtype=CNativeInt()))), - AliasAssign(require_aligned, PyLong_FromLong(convert_to_literal(1, dtype=CNativeInt()))), - AliasAssign( - include_rank, - PyLong_FromLong(convert_to_literal(1 if descriptor_type.has_rank else 0, dtype=CNativeInt())), - ), - AliasAssign( - include_itemsize, - PyLong_FromLong(convert_to_literal(1 if descriptor_type.has_itemsize else 0, dtype=CNativeInt())), - ), - AliasAssign( - include_strides, - PyLong_FromLong(convert_to_literal(1 if ubound_elems or stride_elems else 0, dtype=CNativeInt())), - ), - AliasAssign(require_contiguous, PyLong_FromLong(convert_to_literal(1, dtype=CNativeInt()))), - ] - body.extend(self._return_if_any_native_array_helper_arg_failed(runtime_module, helper, owned_args)) - body.extend( - [ - AliasAssign( - helper_args, - PyTuple_Pack( - ObjectAddress(collect_arg), - ObjectAddress(dtype), - ObjectAddress(rank), - ObjectAddress(expected_shape), - ObjectAddress(expected_layout), - ObjectAddress(require_writeable), - ObjectAddress(require_native_byte_order), - ObjectAddress(require_aligned), - ObjectAddress(include_rank), - ObjectAddress(include_itemsize), - ObjectAddress(include_strides), - ObjectAddress(require_contiguous), - ), - ), - If( - IfSection( - Is(helper_args, NIL), - [ - *self._decref_all([*owned_args, helper, runtime_module]), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(packed, PyObject_CallObject(helper, helper_args)), - Py_DECREF(helper_args), - *self._decref_all([*owned_args, helper, runtime_module]), - If(IfSection(Is(packed, NIL), [Return(self._error_exit_code)])), - ] - ) - body.extend( - self._assign_native_array_actual_fields( - packed, - parts, - shape_elems, - ubound_elems, - stride_elems, - ) - ) - body.append(Py_DECREF(packed)) - return body - - def _native_array_actual_expected_layout_object(self, subject, expected_layout): - """Return body nodes that create the runtime expected-layout argument object.""" - layout = self._native_array_actual_expected_layout_name(subject) - if layout is None: - return [AliasAssign(expected_layout, Py_None), Py_INCREF(Py_None)] - return [AliasAssign(expected_layout, PyUnicode_FromString(CStrStr(convert_to_literal(layout))))] - - @staticmethod - def _native_array_actual_expected_layout_name(subject): - """Return the layout spelling enforced for a native handle array actual.""" - if getattr(subject, "rank", None) == 1: - return None - return getattr(subject, "order", None) - - def _assign_native_array_actual_fields( - self, - packed, - parts, - shape_elems, - ubound_elems, - stride_elems, - ): - """Copy packed runtime normal-array ABI fields into generated descriptor slots.""" - targets = [parts["data"]] - if parts["rank"] is not None: - targets.append(parts["rank"]) - if parts["itemsize"] is not None: - targets.append(parts["itemsize"]) - targets.extend(shape_elems) - targets.extend(ubound_elems) - targets.extend(stride_elems) - body = [] - for index, target in enumerate(targets): - item = self._new_python_object(f"array_actual_field_{index}") - body.extend( - [ - AliasAssign(item, PyTuple_GetItem(packed, convert_to_literal(index))), - If(IfSection(Is(item, NIL), [Py_DECREF(packed), Return(self._error_exit_code)])), - ] - ) - if index == 0: - body.extend(self._assign_native_array_descriptor_pointer(target, item, packed)) - else: - body.extend(self._assign_native_array_actual_int64(target, item, packed)) - return body - - def _assign_native_array_actual_int64(self, target, item, packed): - """Convert one Python integer normal-array ABI field to an int64 slot.""" - return [ - Assign(target, PyLong_AsLongLong(item)), - If( - IfSection( - And(Eq(target, convert_to_literal(-1, dtype=NumpyInt64Type())), PyErr_Occurred()), - [Py_DECREF(packed), Return(self._error_exit_code)], - ) - ), - ] - - def _convert_python_string_value_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - ): - """Read a Python string value for the completed barrier.""" - return self._convert_string_argument( - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - arg_var=arg_var, - projected_replacement=decision.codegen_action is CodegenAction.COPY_IN_OUT, - ) - - def _convert_string_argument( - self, - orig_var, - decision, - collect_arg, - bound_argument, - is_bind_c_argument, - *, - arg_var=None, - projected_replacement, - ): - """ - Extract the C-compatible string FunctionDefArgument from the PythonObject. - - Extract the C-compatible string FunctionDefArgument from the PythonObject. - The C-compatible argument is extracted from collect_arg which holds a Python - object into arg_var. - - The extraction is done by allocating an array and filling the elements with values - extracted from the indexed Python tuple in collect_arg. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - bound_argument : bool - True if the argument is the self argument of a class method. False otherwise. - This should always be False for this function. - - is_bind_c_argument : bool - True if the argument was saved in a BindCFunctionDefArgument. False otherwise. - - arg_var : Variable | IndexedElement, optional - A variable or an element of the variable representing the argument that - will be passed to the low-level function call. - - Returns - ------- - list[model object] - A list of expressions which extract the argument from collect_arg into arg_var. - """ - assert bound_argument is False - - if is_bind_c_argument: - writable = decision.mutates_native - if arg_var is not None: - raise NotImplementedError("Reusing an existing Bind-C string descriptor is not supported.") - data_var, size_var, arg_var = self._bind_c_string_arg_parts(orig_var, writable=writable) - - source_var, source_size, body = self._string_utf8_source(orig_var, collect_arg) - if writable: - payload_size = self._string_replacement_payload_size(orig_var, source_size) - fixed_length = payload_size is not source_size - body.extend( - [ - Assign(size_var, payload_size), - Assign(ObjectAddress(data_var), x2py_malloc(Add(payload_size, convert_to_literal(1)))), - If( - IfSection( - Is(data_var, NIL), - [ - PyErr_SetString( - PyMemoryError, - CStrStr( - convert_to_literal( - f"Unable to allocate mutable string buffer for argument {orig_var.name}." - ) - ), - ), - Return(self._error_exit_code), - ], - ) - ), - *self._string_replacement_copy_body( - data_var, - source_var, - source_size, - payload_size, - fixed_length=fixed_length, - ), - ] - ) - else: - body.extend([Assign(ObjectAddress(data_var), ObjectAddress(source_var)), Assign(size_var, source_size)]) - - default_init = [Assign(ObjectAddress(data_var), NIL), Assign(size_var, 0)] - clean_up = [] - if writable and not projected_replacement: - if getattr(orig_var, "is_optional", False): - clean_up.append(If(IfSection(IsNot(data_var, NIL), [Deallocate(data_var)]))) - else: - clean_up.append(Deallocate(data_var)) - else: - if arg_var is None: - kwargs = {"new_class": Variable, "is_argument": False} - if getattr(orig_var, "is_optional", False): - kwargs["memory_handling"] = "alias" - arg_var = orig_var.clone( - self.scope.get_expected_name(orig_var.name), - **kwargs, - ) - self.scope.insert_variable(arg_var, orig_var.name) - - body = [Assign(orig_var, cast_to(PyUnicode_AsUTF8(collect_arg), StringType()))] - - default_init = [AliasAssign(arg_var, NIL)] - if getattr(orig_var, "is_optional", False): - memory_var = self.scope.get_temporary_variable( - arg_var, - name=arg_var.name + "_memory", - is_optional=False, - memory_handling="stack", - ) - body.insert(0, AliasAssign(arg_var, memory_var)) - clean_up = [] - - return {"body": body, "args": [arg_var], "default_init": default_init, "clean_up": clean_up} - - def _convert_result(self, orig_var, is_bind_c, funcdef=None, *, owner_object=None): - """ - Get the code which translates a C-compatible `Variable` to a Python `FunctionDefResult`. - - Get the code necessary to transform a Variable returned from a C-compatible function written in - Fortran to an object with datatype `PythonObjectType`. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - - is_bind_c : bool - True if the result comes from a C-binding from another language. False otherwise. - - funcdef : FunctionDef - The function being wrapped. - - Returns - ------- - dict[str, Any] - A dictionary with the keys: - - body : a list of model objects containing the code which translates the C-compatible variable - to a `PythonObjectType`. - - c_results : a list of Variables which are returned from the function being wrapped. - - py_result : the Variable returned to Python. - - setup : An optional key containing a list of model objects with code which should be - run before calling the function being wrapped. - """ - if orig_var is NIL: - return {"c_results": [], "py_result": Py_None, "body": []} - - class_type = orig_var.original_var.class_type if isinstance(orig_var, BindCVariable) else orig_var.class_type - if isinstance(class_type, BindCResultTupleType): - return self._convert_result_tuple(orig_var, is_bind_c, funcdef) - original = getattr(orig_var, "original_var", orig_var) - if original.array_interop_policy is not None: - return self._ARRAY_INTEROP_POLICY_DISPATCHER.dispatch( - self, - original, - original.array_interop_policy, - "result", - orig_var, - is_bind_c, - funcdef, - owner_object, - ) - return self._bind_non_array_result(original, orig_var, is_bind_c, funcdef, owner_object) - - def _bind_data_buffer_result(self, subject, _policy, wrapped_var, is_bind_c, funcdef, owner_object=None): - """Convert an ordinary array result through the data-buffer ABI.""" - return self._bind_non_array_result(subject, wrapped_var, is_bind_c, funcdef, owner_object) - - def _bind_descriptor_result(self, subject, policy, wrapped_var, is_bind_c, funcdef, owner_object=None): - """Convert a native array handle result through descriptor ABI.""" - self._validate_descriptor_array_interop_policy(subject, policy) - return self._NATIVE_ARRAY_DESCRIPTOR_RESULT_DISPATCHER.dispatch( - self, - subject, - policy, - wrapped_var, - is_bind_c, - funcdef, - owner_object, - ) - - @staticmethod - def _bind_projected_native_array_handle_result( - _subject, - _decision, - _policy, - _wrapped_var, - _is_bind_c, - _funcdef, - _owner_object, - ): - """Project the caller's existing handle without a second native result.""" - return { - "c_results": [], - "py_result": Py_None, - "py_results": [], - "owned_py_results": [], - "body": [], - } - - def _bind_materialized_native_array_handle_result( - self, - subject, - _decision, - _policy, - wrapped_var, - is_bind_c, - funcdef, - owner_object, - ): - """Materialize an owned handle selected by completed result policy.""" - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - subject, - subject.native_array_handle_policy, - wrapped_var, - is_bind_c, - funcdef, - owner_object, - ) - - def _bind_non_array_result(self, original, orig_var, is_bind_c, funcdef, owner_object): - """Convert a native result without native descriptor-handle routing.""" - previous_owner = getattr(self, "_result_owner_object", None) - self._result_owner_object = owner_object - try: - return self._RESULT_POLICY_DISPATCHER.dispatch( - self, - original, - orig_var, - is_bind_c, - funcdef, - ) - finally: - self._result_owner_object = previous_owner - - def _convert_policy_custom_result(self, orig_var, decision, wrapped_var, is_bind_c, funcdef): - """Emit the completed custom-value result behavior.""" - return self._convert_custom_type_result(wrapped_var, is_bind_c, funcdef, decision) - - def _convert_policy_scalar_result(self, orig_var, decision, wrapped_var, is_bind_c, funcdef): - """Emit the completed scalar result behavior.""" - if decision.descriptor_boundary and decision.nullable: - return self._build_snapshot_copy_scalar_result(wrapped_var) - return self._convert_scalar_result(wrapped_var, is_bind_c, funcdef, decision) - - def _convert_snapshot_policy_scalar_result(self, orig_var, decision, wrapped_var, is_bind_c, funcdef): - """Emit a completed snapshot-copy scalar result.""" - return self._build_snapshot_copy_scalar_result(wrapped_var) - - def _convert_policy_array_result(self, orig_var, decision, wrapped_var, is_bind_c, funcdef): - """Emit the completed array result behavior for its concrete ABI representation.""" - if isinstance(getattr(wrapped_var, "class_type", None), BindCArrayType): - return self._convert_bind_c_array_result(wrapped_var, funcdef, decision=decision) - return self._convert_array_result(wrapped_var, is_bind_c, funcdef, decision) - - def _convert_policy_string_result(self, orig_var, decision, wrapped_var, is_bind_c, funcdef): - """Emit the completed string result behavior.""" - return self._convert_string_result(wrapped_var, is_bind_c, funcdef, decision) - - def _convert_custom_type_result(self, wrapped_var, is_bind_c, funcdef, decision): - """ - Get the code which translates a `Variable` containing a class instance to a PyObject. - - Get the code which translates a `Variable` containing a class instance to a PyObject. - - Parameters - ---------- - wrapped_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - is_bind_c : bool - True if the result comes from a C-binding from another language. False otherwise. - funcdef : FunctionDef - The function being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result. - """ - orig_var = getattr(wrapped_var, "original_var", wrapped_var) - name = orig_var.name - python_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - is_alias = decision.borrowed - setup = self._allocate_class_instance(python_res, python_res.cls_base.scope, is_alias) - if is_bind_c: - result_source = wrapped_var.new_var if isinstance(wrapped_var, BindCVariable) else wrapped_var - c_res = result_source.clone( - self.scope.get_new_name(orig_var.name), - is_argument=False, - memory_handling=result_source.memory_handling, - new_class=Variable, - ) - self.scope.insert_variable(c_res, orig_var.name) - scope = python_res.cls_base.scope - attribute = scope.find("instance", "variables", raise_if_missing=True) - attrib_var = attribute.clone(attribute.name, new_class=DottedVariable, lhs=python_res) - body = [AliasAssign(attrib_var, c_res)] - result = ObjectAddress(c_res) - else: - scope = python_res.cls_base.scope - attribute = scope.find("instance", "variables", raise_if_missing=True) - c_res = attribute.clone(attribute.name, new_class=DottedVariable, lhs=python_res) - setup.append(Allocate(c_res, shape=None, status="unallocated", like=orig_var)) - result = PointerCast(c_res, cast_type=orig_var) - body = [] - - if funcdef: - body.extend(self._connect_pointer_targets(orig_var, python_res, funcdef, is_bind_c)) - - return { - "c_results": [result], - "py_result": python_res, - "body": body, - "setup": setup, - } - - def _convert_scalar_result(self, orig_var, is_bind_c, funcdef, decision): - """ - Get the code which translates a `Variable` containing a scalar to a PyObject. - - Get the code which translates a `Variable` containing a scalar to a PyObject. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - is_bind_c : bool - True if the result comes from a C-binding from another language. False otherwise. - funcdef : FunctionDef - The function being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result. - """ - name = getattr(orig_var, "name", "tmp") - py_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - c_res = Variable( - orig_var.class_type, - self.scope.get_new_name(name), - ) - self.scope.insert_variable(c_res) - - body = [AliasAssign(py_res, FunctionCall(C_to_Python(c_res), [c_res]))] - return { - "c_results": [c_res], - "py_result": py_res, - "body": body, - "result_bindings": [ - { - "name": str(name), - "original": orig_var, - "c_result": c_res, - "py_result": py_res, - } - ], - } - - def _convert_array_result(self, orig_var, is_bind_c, funcdef, decision): - """ - Get the code which translates a `Variable` containing an array to a PyObject. - - Get the code which translates a `Variable` containing an array to a PyObject. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - is_bind_c : bool - True if the result comes from a C-binding from another language. False otherwise. - funcdef : FunctionDef - The function being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result. - """ - if is_bind_c: - return self._convert_bind_c_array_result(orig_var, funcdef, decision=decision) - name = self.scope.get_new_name(orig_var.name) - py_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - c_res = orig_var.clone(name, is_argument=False, memory_handling="alias") - data_var = DottedVariable(VoidType(), "data", memory_handling="alias", lhs=c_res) - shape_var = DottedVariable(NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), "shape", lhs=c_res) - release_memory = self._ARRAY_RELEASE_POLICY_DISPATCHER.dispatch(self, orig_var, decision) - body = [ - AliasAssign( - py_res, - self._array_to_python_call(orig_var, data_var, shape_var, None, release_memory), - ) - ] - self.scope.insert_variable(c_res) - c_result_vars = [c_res] - - if funcdef: - body.extend(self._connect_pointer_targets(orig_var, py_res, funcdef, False)) - - return {"c_results": c_result_vars, "py_result": py_res, "body": body} - - def _convert_result_tuple(self, tuple_var, is_bind_c, funcdef): - """Convert result tuple for the current wrapper.""" - c_results = [] - py_results = [] - owned_py_results = [] - result_bindings = [] - setup = [] - body = [] - assert funcdef is not None - for index in range(len(tuple_var.class_type)): - element = funcdef.scope.collect_tuple_element(IndexedElement(tuple_var, index)) - original_element = getattr(element, "original_var", element) - descriptor_result = bool( - original_element.array_interop_policy is not None - and original_element.array_interop_policy.is_descriptor - and original_element.native_array_handle_policy is not None - ) - if isinstance(getattr(element, "class_type", None), BindCArrayType) and not descriptor_result: - result = self._convert_bind_c_array_result( - element, - funcdef, - tuple_item=True, - decision=ownership_decision_for_codegen_variable(element.original_var), - ) - else: - result = self._convert_result(element, is_bind_c, funcdef) - item_c_results = result["c_results"] - if isinstance(item_c_results, PythonTuple): - c_results.extend(item_c_results.args) - else: - c_results.extend(item_c_results) - setup.extend(result.get("setup", ())) - body.extend(result["body"]) - py_results.extend(result.get("py_results", [result["py_result"]])) - owned_py_results.extend(result.get("owned_py_results", [True])) - result_bindings.extend(result.get("result_bindings", ())) - return { - "c_results": PythonTuple(*c_results), - "py_result": Py_None, - "py_results": py_results, - "owned_py_results": owned_py_results, - "body": body, - "setup": setup, - "result_bindings": result_bindings, - } - - def _convert_bind_c_array_result(self, wrapped_var, funcdef, *, tuple_item=False, decision): - """ - Get the code which translates a `Variable` containing an array to a PyObject. - - Get the code which translates a `Variable` containing a BindCArray, which describes an - array in Fortran, to a PyObject. - - Parameters - ---------- - wrapped_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - funcdef : FunctionDef - The function being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result. - """ - orig_var = wrapped_var.original_var - name = orig_var.name - py_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - # Result of calling the bind-c function - data_var = Variable(VoidType(), self.scope.get_new_name(name + "_data"), memory_handling="alias") - shape_var = Variable( - NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), - self.scope.get_new_name(name + "_shape"), - shape=(orig_var.rank,), - memory_handling="alias", - ) - # Save so we can find by iterating over func.results - self.scope.insert_variable(data_var) - self.scope.insert_variable(shape_var) - - release_memory = self._ARRAY_RELEASE_POLICY_DISPATCHER.dispatch(self, orig_var, decision) - itemsize_var = ( - Variable(NumpyInt64Type(), self.scope.get_new_name(name + "_itemsize")) - if self._is_character_array(orig_var) - else None - ) - if itemsize_var is not None: - self.scope.insert_variable(itemsize_var) - - array_to_python = AliasAssign( - py_res, - self._array_to_python_call(orig_var, data_var, shape_var, itemsize_var, release_memory), - ) - shape_vars = [IndexedElement(shape_var, i) for i in range(orig_var.rank)] - body = [array_to_python] - if decision.nullable: - if tuple_item: - body = [ - self._set_none_if_unallocated(data_var, py_res, shape_vars), - If(IfSection(IsNot(data_var, NIL), [array_to_python])), - ] - else: - body = [*self._return_none_if_unallocated(data_var, shape_vars), *body] - - c_results = [ObjectAddress(data_var)] - if itemsize_var is not None: - c_results.append(itemsize_var) - c_results.extend(shape_vars) - c_result_vars = PythonTuple(*c_results) - - if funcdef: - body.extend(self._connect_pointer_targets(orig_var, py_res, funcdef, True)) - - return { - "c_results": c_result_vars, - "py_result": py_res, - "py_results": [py_res], - "owned_py_results": [True], - "body": body, - } - - def _convert_string_result(self, wrapped_var, is_bind_c, funcdef, decision): - """Convert string result for the current wrapper.""" - orig_var = getattr(wrapped_var, "original_var", wrapped_var) - name = getattr(orig_var, "name", "tmp") - py_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - if is_bind_c: - c_res = Variable( - CharType(), - self.scope.get_new_name(name + "_data"), - memory_handling="alias", - ) - self.scope.insert_variable(c_res) - char_data = ObjectAddress(c_res) - result = [char_data] - else: - c_res = Variable(StringType(), self.scope.get_new_name(name), memory_handling="heap") - self.scope.insert_variable(c_res) - char_data = CStrStr(c_res) - result = [c_res] - - if is_bind_c: - if decision.nullable: - body = [ - If( - IfSection( - Is(c_res, NIL), - [AliasAssign(py_res, Py_None), Py_INCREF(Py_None)], - ), - IfSection( - convert_to_literal(True), - [AliasAssign(py_res, PyBuildValueNode([char_data])), Deallocate(c_res)], - ), - ) - ] - else: - body = [ - If( - IfSection( - Is(c_res, NIL), - [ - PyErr_SetString( - PyMemoryError, - CStrStr(convert_to_literal("Unable to allocate copy-return output string.")), - ), - Return(self._error_exit_code), - ], - ) - ), - AliasAssign(py_res, PyBuildValueNode([char_data])), - Deallocate(c_res), - ] - else: - body = [AliasAssign(py_res, PyBuildValueNode([char_data]))] - return { - "c_results": result, - "py_result": py_res, - "body": body, - "result_bindings": [ - { - "name": str(name), - "original": orig_var, - "c_result": c_res, - "py_result": py_res, - } - ], - } - - # ------------------------------------------------------------------ - # Node builders - # ------------------------------------------------------------------ - - def _build_module_init_function( - self, - expr, - imports, - module_def_name, - namespace_module_defs, - module_properties, - ): - """ - Build the function that will be called when the module is first imported. - - Build the function that will be called when the module is first imported. - This function must call any initialisation function of the underlying - module and must add any variables to the module variable. - - Parameters - ---------- - expr : Module - The module of interest. - - imports : list of Import - A list of any imports that will appear in the PyModule. - - module_def_name : str - The name of the structure which defined the module. - - Returns - ------- - PyModInitFunc - The initialisation function. - """ - mod_name = self.scope.get_python_name(getattr(expr, "original_module", expr).name) - # The name of the init function is compulsory for the wrapper to work - func_name = f"PyInit_{mod_name}" - # Initialise the scope - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - - for v in expr.variables: - func_scope.insert_symbol(v.name) - - n_classes = len(expr.classes) - - # Create necessary variables - module_var = self._new_python_object("mod") - API_var_name = self.scope.get_new_name(f"Py{mod_name}_API", object_type="wrapper") - API_var = Variable( - NumpyNDArrayType.get_new(BindCPointer(), 1, None, raw=True), - API_var_name, - shape=(n_classes,), - cls_base=StackArrayClass, - ) - self.scope.insert_variable(API_var) - capsule_obj = self._new_python_object(self.scope.get_new_name("c_api_object")) - - body = [ - AliasAssign(module_var, PyModule_Create(module_def_name)), - If(IfSection(Is(module_var, NIL), [Return(self._error_exit_code)])), - ] - - initialised = [module_var] - namespace_modules, namespace_body = self._create_namespace_modules( - namespace_module_defs, - module_var, - initialised, - ) - body.extend(namespace_body) - for namespace, descriptor in module_properties.items(): - target_module = module_var if not namespace else namespace_modules[namespace] - body.append( - If( - IfSection( - Lt( - PyModule_SetPropertyType(descriptor["setup_name"], target_module), - convert_to_literal(0), - ), - [Py_DECREF(item) for item in initialised] + [Return(self._error_exit_code)], - ) - ) - ) - - # Save classes to the module variable - for i, c in enumerate(expr.classes): - wrapped_class = self._python_object_map[c] - type_object = wrapped_class.type_object - - API_elem = IndexedElement(API_var, i) - body.append(Assign(API_elem, ObjectAddress(type_object))) - - ok_code = convert_to_literal(0) - - # Save Capsule describing types (needed for dependent modules) - body.append(AliasAssign(capsule_obj, PyCapsule_New(API_var, mod_name))) - body.extend(self._add_object_to_mod(module_var, capsule_obj, "_C_API", initialised)) - - body.append(import_array()) - import_funcs = [i.source_module.import_func for i in imports if isinstance(i.source_module, PyModule)] - for i_func in import_funcs: - body.append( - If( - IfSection( - Lt(i_func(), ok_code), - [Py_DECREF(i) for i in initialised] + [Return(self._error_exit_code)], - ) - ) - ) - - # Call the initialisation function - if expr.init_func: - body.append(expr.init_func()) - body.extend(self._initialise_module_variable_defaults(expr)) - - body.extend(self._add_classes_to_modules(expr, module_var, namespace_modules, initialised)) - body.extend(self._add_variables_to_modules(expr, module_var, namespace_modules, initialised)) - - body.append(Return(module_var)) - - self.exit_scope() - - return PyModInitFunc(func_name, body, [API_var], func_scope) - - def _initialise_module_variable_defaults(self, expr): - """Apply literal `.pyi` defaults to native module storage on import.""" - setters = self._module_variable_native_setters(expr) - body = [] - for variable in expr.original_module.variables: - if variable.is_private or variable.default_value is None or isinstance(variable.class_type, FinalType): - continue - setter = setters.get(str(variable.name)) - if setter is None: - raise ValueError(f"Module variable {variable.name!r} has a resolved initializer but no native setter") - body.append(setter(self._module_literal_value(variable))) - return body - - @staticmethod - def _module_variable_native_setters(expr): - """Return generated bind-C setters keyed by source module-variable name.""" - setters = {} - for function in expr.funcs: - decorators = getattr(getattr(function, "original_function", None), "decorators", {}) - if decorators.get(INTERNAL_MODULE_VARIABLE_ACCESS_METADATA) != "set": - continue - variable_name = decorators.get(INTERNAL_MODULE_VARIABLE_NAME_METADATA) - if isinstance(variable_name, str): - setters[variable_name] = function - return setters - - def _create_namespace_modules(self, namespace_module_defs, root_module, initialised): - """Create nested Python module objects and register them on parents.""" - namespace_modules = {} - body = [] - for namespace, child_def_name in namespace_module_defs.items(): - child_module = self._new_python_object("mod_" + "_".join(namespace)) - namespace_modules[namespace] = child_module - body.extend( - [ - AliasAssign(child_module, PyModule_Create(child_def_name)), - If( - IfSection( - Is(child_module, NIL), - [Py_DECREF(item) for item in initialised] + [Return(self._error_exit_code)], - ) - ), - ] - ) - parent_module = root_module if len(namespace) == 1 else namespace_modules[namespace[:-1]] - body.extend(self._add_object_to_mod(parent_module, child_module, namespace[-1], initialised)) - return namespace_modules, body - - def _add_classes_to_modules(self, expr, root_module, namespace_modules, initialised): - """Ready generated classes and add them to their export modules.""" - body = [] - for semantic_class in expr.classes: - type_object = self._python_object_map[semantic_class].type_object - body.append( - If( - IfSection( - Lt(PyType_Ready(type_object), convert_to_literal(0)), - [Py_DECREF(item) for item in initialised] + [Return(self._error_exit_code)], - ) - ) - ) - for namespace, class_name in expr.get_python_exports(semantic_class): - target_module = root_module if not namespace else namespace_modules[namespace] - body.extend(self._add_object_to_mod(target_module, type_object, class_name, initialised)) - return body - - def _add_variables_to_modules(self, expr, root_module, namespace_modules, initialised): - """Install generated module-variable descriptors on export modules.""" - body = [] - for variable in expr.variables: - decision = ownership_decision_for_codegen_variable(variable) - if variable.is_private or ( - isinstance(variable, BindCArrayVariable) and decision.storage_mode is StorageMode.HEAP - ): - continue - previous_owner = getattr(self, "_native_array_handle_owner_module", None) - self._native_array_handle_owner_module = root_module - try: - body.extend(self._visit(variable)) - finally: - self._native_array_handle_owner_module = previous_owner - wrapped_variable = self._python_object_map[variable] - for namespace, variable_name in expr.get_python_exports(variable): - target_module = root_module if not namespace else namespace_modules[namespace] - body.extend(self._add_object_to_mod(target_module, wrapped_variable, variable_name, initialised)) - return body - - def _build_module_import_function(self, expr): - """ - Build the function that will be called in order to use the module from another module. - - Build the function that will be called when the module is first imported. - This function must import the capsule created in the module initialisation. - In order for this to work from any folder the `sys.path` list is modified to include - the folder where the file is located (currently this is done by temporarily modifying - an element of the list as the stable C-Python API doesn't contain any functions for - reducing the size of lists). - See - for more details. - - Parameters - ---------- - expr : Module - The module of interest. - - Returns - ------- - API_var : Variable - The variable which contains the data extracted from the capsule. - - import_func : FunctionDef - The import function. - """ - mod_name = self.scope.get_python_name(getattr(expr, "original_module", expr).name) - # Initialise the scope - func_name = self.scope.get_new_name("import") - - API_var_name = self.scope.insert_symbol(f"Py{mod_name}_API", "wrapper") - API_var = Variable( - NumpyNDArrayType.get_new(BindCPointer(), 1, None, raw=True), - API_var_name, - shape=(None,), - cls_base=StackArrayClass, - memory_handling="alias", - ) - self.scope.insert_variable(API_var) - - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - - ok_code = convert_to_literal(0, dtype=CNativeInt()) - error_code = convert_to_literal(-1, dtype=CNativeInt()) - self._error_exit_code = error_code - - # Create variables to temporarily modify the Python path so the file will be discovered - current_path = func_scope.get_temporary_variable(PythonObjectType(), "current_path", memory_handling="alias") - stash_path = func_scope.get_temporary_variable(PythonObjectType(), "stash_path", memory_handling="alias") - - body = [ - AliasAssign(current_path, PySys_GetObject(CStrStr(convert_to_literal("path")))), - AliasAssign( - stash_path, - PyList_GetItem(current_path, convert_to_literal(0, dtype=CNativeInt())), - ), - Py_INCREF(stash_path), - If( - IfSection( - Eq( - PyList_SetItem( - current_path, - convert_to_literal(0, dtype=CNativeInt()), - PyUnicode_FromString(CStrStr(convert_to_literal(self._sharedlib_dirpath))), - ), - convert_to_literal(-1), - ), - [Return(self._error_exit_code)], - ) - ), - AliasAssign(API_var, PyCapsule_Import(mod_name)), - If( - IfSection( - Eq( - PyList_SetItem( - current_path, - convert_to_literal(0, dtype=CNativeInt()), - stash_path, - ), - convert_to_literal(-1), - ), - [Return(self._error_exit_code)], - ) - ), - Return(IfTernaryOperator(IsNot(API_var, NIL), ok_code, error_code)), - ] - - result = func_scope.get_temporary_variable(CNativeInt()) - self.exit_scope() - self._error_exit_code = NIL - import_func = FunctionDef( - func_name, - (), - body, - FunctionDefResult(result), - is_static=True, - scope=func_scope, - ) - - return API_var, import_func - - def _build_snapshot_copy_scalar_result(self, wrapped_var): - """Build snapshot copy scalar result nodes.""" - orig_var = getattr(wrapped_var, "original_var", wrapped_var) - name = getattr(orig_var, "name", "tmp") - py_res = self._new_python_object(f"{name}_obj", orig_var.dtype) - data_var = Variable(VoidType(), self.scope.get_new_name(f"{name}_data"), memory_handling="alias") - value_var = orig_var.clone( - self.scope.get_new_name(f"{name}_value"), - new_class=Variable, - is_argument=False, - memory_handling="stack", - ) - pointer_type = orig_var.clone( - self.scope.get_new_name(f"{name}_pointer_type"), - new_class=Variable, - is_argument=False, - memory_handling="alias", - ) - self.scope.insert_variable(data_var) - self.scope.insert_variable(value_var) - copy_value = Assign(value_var, PointerCast(data_var, pointer_type)) - convert_value = AliasAssign(py_res, FunctionCall(C_to_Python(value_var), [value_var])) - body = [ - If( - IfSection(Is(data_var, NIL), [AliasAssign(py_res, Py_None), Py_INCREF(Py_None)]), - IfSection(convert_to_literal(True), [copy_value, convert_value, Deallocate(data_var)]), - ) - ] - return {"c_results": [data_var], "py_result": py_res, "body": body} - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _function_docstring(self, name, func, original_func=None): - """Handle function docstring for the current generation context.""" - original_func = original_func or func - visible_args = [arg for arg in func.arguments if not arg.bound_argument] - result_vars = self._doc_python_result_vars(func, original_func) - signature = f"{name}({', '.join(self._doc_argument_name(arg) for arg in visible_args)})" - signature += f" -> {self._doc_result_summary(result_vars)}" if result_vars else " -> None" - - sections = [signature] - user_doc = self._existing_docstring_text(getattr(original_func, "docstring", None)) - if user_doc: - sections.extend(["", user_doc]) - - if visible_args: - sections.extend(["", "Parameters", "----------"]) - for arg in visible_args: - sections.extend(self._argument_doc_lines(arg)) - - sections.extend(["", "Returns", "-------"]) - if result_vars: - for result in result_vars: - sections.extend(self._variable_doc_lines(self._doc_original_var(result), result_name=True)) - else: - sections.append("None") - - notes = self._result_notes(result_vars) - if notes: - sections.extend(["", "Notes", "-----", *notes]) - - sections.extend( - [ - "", - "Raises", - "------", - "TypeError", - " If an argument has incompatible dtype, rank, shape, layout, or wrapped class.", - ] - ) - if isinstance(getattr(original_func, "decorators", {}).get(RUNTIME_STATUS_ERROR_METADATA), dict): - sections.extend( - [ - "RuntimeError", - " If the annotated native status output is not the declared success value.", - ] - ) - return CommentBlock("\n".join(sections)) - - @staticmethod - def _existing_docstring_text(docstring): - """Handle existing docstring text for the current generation context.""" - if not docstring: - return "" - return "\n".join(str(line) for line in docstring.comments if str(line).strip()) - - def _argument_doc_lines(self, arg): - """Handle argument doc lines for the current generation context.""" - var = self._doc_original_var(arg.var) - if isinstance(var, FunctionAddress): - argument_types = ", ".join(self._type_doc(item.var) for item in var.arguments) - result_type = "None" if var.results.var is NIL else self._type_doc(var.results.var) - return [ - f"{self._doc_argument_name(arg)} : Callable[[{argument_types}], {result_type}]", - " Immediate-call callback retained only for the duration of this call.", - " Callback exceptions print their traceback and abort the host process.", - ] - can_be_none = ( - getattr(arg.var, "is_optional", False) - or getattr(var, "is_optional", False) - or self._is_nullable_replacement_argument(var) - ) - header = f"{self._doc_argument_name(arg)} : {self._type_doc(var, include_none=can_be_none)}" - details = self._argument_detail_lines(var) - if can_be_none: - if self._is_optional_scalar_descriptor_var(var): - details.append(" Omit to make the native optional dummy absent.") - details.append(" Pass None for a present unallocated or unassociated descriptor.") - elif self._is_nullable_replacement_argument(var): - details.append(" May be passed as None for initially unallocated storage.") - else: - details.append(" May be omitted or passed as None.") - if arg.has_default and not self._is_optional_scalar_descriptor_var(var): - details.append(f" Default is {arg.value}.") - return [header, *details] - - def _variable_doc_lines(self, var, *, result_name=False): - """Handle variable doc lines for the current generation context.""" - name = str(var.name) if result_name else "result" - header = f"{name} : {self._type_doc(var, include_none=self._may_return_none(var))}" - return [header, *self._result_detail_lines(var)] - - def _argument_detail_lines(self, var): - """Handle argument detail lines for the current generation context.""" - lines = self._value_detail_lines(var) - lines.extend(self._ARGUMENT_DETAIL_DISPATCHER.dispatch(self, var)) - return lines - - @staticmethod - def _direct_argument_detail_lines(_var, _decision): - """Return documentation details for a direct scalar value.""" - return [] - - @staticmethod - def _call_local_argument_detail_lines(_var, decision): - """Describe whether call-local native mutation is discarded.""" - if decision.mutates_native: - return [" Mutates: no; native mutation is discarded"] - return [] - - @staticmethod - def _in_place_argument_detail_lines(_var, _decision): - """Describe an input/output argument that mutates caller storage.""" - return [" Mutates: yes"] - - @staticmethod - def _identity_output_detail_lines(var, _decision): - """Describe an output that fills and returns caller storage.""" - lines = [" Mutates: fills in-place"] - if var.rank: - lines.append(" Initial contents are ignored.") - return lines - - @staticmethod - def _discarded_identity_output_detail_lines(_var, _decision): - """Describe an immutable output whose call-local mutation is discarded.""" - return [" Mutates: no; native mutation is discarded"] - - @staticmethod - def _replacement_value_detail_lines(_var, _decision): - """Describe immutable scalar or string replacement semantics.""" - return [" Mutates: no; returns a replacement value"] - - @staticmethod - def _replacement_array_detail_lines(_var, decision): - """Describe immutable array replacement and nullability semantics.""" - suffix = " or None" if decision.nullable else "" - return [f" Mutates: no; returns a replacement array{suffix}"] - - def _result_detail_lines(self, var): - """Handle result detail lines for the current generation context.""" - lines = self._value_detail_lines(var) - lines.extend(self._RESULT_DETAIL_DISPATCHER.dispatch(self, var)) - return lines - - def _borrowed_detail_lines(self, var): - """Handle borrowed detail lines for the current generation context.""" - lines = self._value_detail_lines(var) - lines.extend(self._RESULT_DETAIL_DISPATCHER.dispatch(self, var)) - return lines - - def _result_notes(self, result_vars): - """Handle result notes for the current generation context.""" - notes = [] - seen_note_groups = set() - for result in result_vars: - var = self._doc_original_var(result) - action_notes = self._RESULT_NOTE_DISPATCHER.dispatch(self, var) - note_key = tuple(action_notes) - if not action_notes or note_key in seen_note_groups: - continue - seen_note_groups.add(note_key) - if notes and action_notes: - notes.append("") - notes.extend(action_notes) - return notes - - def _default_result_detail_lines(self, var, decision): - """Handle default result detail lines for the current generation context.""" - if not var.rank: - return [] - lines = [f" Ownership: {decision.owner_label}"] - handle_policy = getattr(var, "native_array_handle_policy", None) - if handle_policy is not None: - lines.append(f" Descriptor ownership: {handle_policy.descriptor_ownership}") - if handle_policy.descriptor_kind == "allocatable": - lines.append(" Unallocated state remains inside the returned handle.") - else: - lines.append(" Unassociated state remains inside the returned handle.") - return lines - if decision.nullable: - lines.append(" Returns None when unallocated.") - return lines - - def _snapshot_copy_result_detail_lines(self, var, decision): - """Handle snapshot copy result detail lines for the current generation context.""" - lines = [f" Ownership: {decision.owner_label}"] - if decision.nullable: - state = "unallocated" if decision.storage_mode is StorageMode.HEAP else "unassociated" - lines.append(f" Returns None when {state}.") - return lines - - def _copy_return_result_notes(self, var, decision): - """Handle copy return result notes for the current generation context.""" - if not self._is_allocatable_copy_return_result(var): - return [] - return [ - "Allocatable array outputs and replacements are copied into Python-owned NumPy arrays.", - "This copy adds overhead proportional to the returned array size.", - ] - - def _snapshot_copy_result_notes(self, var, decision): - """Handle snapshot copy result notes for the current generation context.""" - if decision.storage_mode is StorageMode.HEAP: - if not var.rank: - return [ - "Allocatable scalar snapshots are copied into detached Python values.", - "Unallocated allocatable scalar snapshots return None.", - ] - return [ - "The explicit array result is copied into Python-owned NumPy storage.", - "The returned array is independent from later native changes.", - ] - if not var.rank: - return [ - "Pointer scalar snapshots are copied into detached Python values.", - "Unassociated pointer scalar snapshots return None.", - ] - return [ - "Pointer array results are copied into Python-owned NumPy arrays.", - "Unassociated pointer results return None.", - ] - - def _borrowed_view_result_notes(self, var, decision): - """Handle borrowed view result notes for the current generation context.""" - if not var.rank: - return [] - return self._borrowed_view_notes() - - def _empty_result_notes(self, var, decision): - """Handle empty result notes for the current generation context.""" - return [] - - @staticmethod - def _borrowed_view_notes(): - """Handle borrowed view notes for the current generation context.""" - return [ - "The returned NumPy array is a zero-copy view of native Fortran memory.", - "", - "If the corresponding allocatable variable is deallocated or", - "reallocated on the native side, previously obtained views may", - "become invalid.", - "", - "Use ``x.copy()`` to obtain an independent NumPy array.", - ] - - def _value_detail_lines(self, var): - """Handle value detail lines for the current generation context.""" - lines = [] - decision = getattr(var, "ownership_decision", None) - if decision is not None and decision.python_barrier_action is PythonBarrierAction.STRING_STORAGE: - itemsize = self._fixed_character_itemsize(var) - if itemsize is not None: - lines.append(f" Dtype: S{itemsize}") - lines.append(" Rank: 0") - return lines - if var.rank: - shape_doc = self._shape_doc(var) - if shape_doc: - lines.append(f" Shape: {shape_doc}") - if self._is_assumed_rank_array(var): - lines.append(f" Rank: 1..{_MAX_SUPPORTED_ASSUMED_RANK}") - else: - lines.append(f" Rank: {var.rank}") - layout_doc = self._layout_doc(var) - if layout_doc: - lines.append(f" Layout: {layout_doc}") - return lines - - @staticmethod - def _type_doc(var, *, include_none=False, signature=False): - """Handle type doc for the current generation context.""" - decision = getattr(var, "ownership_decision", None) - handle_policy = getattr(var, "native_array_handle_policy", None) - if handle_policy is not None: - handle_name = "AllocatableArray" if handle_policy.descriptor_kind == "allocatable" else "PointerArray" - doc_type = f"{handle_name}[{CPythonBindingGenerator._dtype_doc(var)}]" - elif decision is not None and decision.python_barrier_action is PythonBarrierAction.STRING_STORAGE: - doc_type = "ndarray[bytes]" - elif getattr(var, "is_ndarray", False): - doc_type = f"ndarray[{CPythonBindingGenerator._dtype_doc(var)}]" - else: - doc_type = str(var.class_type).removeprefix("numpy.") - if not include_none: - return doc_type - return f"{doc_type} | None" if signature else f"{doc_type} or None" - - @staticmethod - def _dtype_doc(var): - """Handle dtype doc for the current generation context.""" - return str(var.dtype).removeprefix("numpy.") - - @staticmethod - def _may_return_none(var): - """Handle may return none for the current generation context.""" - if getattr(var, "native_array_handle_policy", None) is not None: - return False - decision = ownership_decision_for_codegen_variable(var) - return decision.nullable - - @staticmethod - def _is_nullable_replacement_argument(var): - """Return whether a completed replacement accepts initially absent storage.""" - decision = ownership_decision_for_codegen_variable(var) - return bool(decision.codegen_action is CodegenAction.COPY_IN_OUT and decision.nullable) - - @staticmethod - def _is_allocatable_copy_return_result(var): - """Return whether a policy-selected copy return uses heap storage.""" - decision = ownership_decision_for_codegen_variable(var) - return decision.storage_mode is StorageMode.HEAP - - @staticmethod - def _shape_doc(var): - """Handle shape doc for the current generation context.""" - shape = getattr(var, "alloc_shape", None) - if not shape or all(dim is None for dim in shape): - return None - shape_parts = ["any" if dim is None else str(dim) for dim in shape] - trailing_comma = "," if len(shape_parts) == 1 else "" - return f"({', '.join(shape_parts)}{trailing_comma})" - - @staticmethod - def _layout_doc(var): - """Handle layout doc for the current generation context.""" - if getattr(var, "rank", 0) <= 1: - return None - order = getattr(var, "order", None) - if order == "F": - return "F-contiguous" - if order == "C": - return "C-contiguous" - return "C-contiguous" - - @staticmethod - def _doc_original_var(var): - """Handle doc original var for the current generation context.""" - return getattr(var, "original_var", var) - - def _doc_argument_name(self, arg): - """Handle doc argument name for the current generation context.""" - return str(self._doc_original_var(arg.var).name) - - @staticmethod - def _doc_result_vars(func): - """Handle doc result vars for the current generation context.""" - if func.results.var is NIL: - return [] - return [ - var - for var in func.scope.collect_all_tuple_elements(func.results.var) - if isinstance(var, Variable) and var is not NIL - ] - - def _doc_python_result_vars(self, func, original_func): - """Handle doc python result vars for the current generation context.""" - result_vars = [] - if original_func.results.var is not NIL: - result_vars.extend(self._doc_result_vars(original_func)) - result_vars.extend( - arg.var - for arg in original_func.arguments - if not arg.bound_argument - and not isinstance(arg.var, FunctionAddress) - and ownership_decision_for_codegen_variable(arg.var).projects_result - ) - if not result_vars: - result_vars = self._doc_result_vars(func) - excluded = self._status_error_output_names(original_func) - return [var for var in result_vars if str(self._doc_original_var(var).name) not in excluded] - - def _doc_result_summary(self, result_vars): - """Handle doc result summary for the current generation context.""" - parts = [ - self._type_doc( - self._doc_original_var(var), - include_none=self._may_return_none(self._doc_original_var(var)), - signature=True, - ) - for var in result_vars - ] - if len(parts) == 1: - result_var = self._doc_original_var(result_vars[0]) - return self._type_doc(result_var, include_none=self._may_return_none(result_var), signature=True) - return f"tuple[{', '.join(parts)}]" - - def _class_docstring(self, cls): - """Handle class docstring for the current generation context.""" - lines = [str(cls.name), "", "Fields", "------"] - if cls.attributes: - for attribute in cls.attributes: - attr_name, var = self._class_attribute_doc_target(attribute) - lines.append(f"{attr_name} : {self._type_doc(var, include_none=self._may_return_none(var))}") - lines.extend(self._borrowed_detail_lines(var)) - else: - lines.append("None") - lines.extend(["", "Methods", "-------"]) - public_methods = [] - for method in cls.methods: - if not method.is_semantic or method.is_private: - continue - original = getattr(method, "original_function", method) - py_name = str(original.scope.get_python_name(original.name)) - if py_name == "__del__": - continue - public_methods.append(py_name) - if public_methods: - lines.extend(public_methods) - else: - lines.append("None") - return CommentBlock("\n".join(lines)) - - def _class_attribute_doc_target(self, attribute): - """Handle class attribute doc target for the current generation context.""" - if isinstance(attribute, BindCNativeArrayHandleProperty): - return attribute.python_name, self._doc_original_var(attribute.original_variable) - if isinstance(attribute, BindCClassProperty): - original = attribute.getter.original_function - if isinstance(original, DottedVariable): - return attribute.python_name, self._doc_original_var(original) - return attribute.python_name, self._doc_original_var(original.results.var) - return str(attribute.name), self._doc_original_var(attribute) - - def _attribute_docstring(self, name, var, getter_policy, setter_policy): - """Handle attribute docstring for the current generation context.""" - var = self._doc_original_var(var) - lines = [ - f"{name} : {self._type_doc(var, include_none=self._may_return_none(var))}", - *self._borrowed_detail_lines(var), - ] - if setter_policy is not None and setter_policy.setter_action is SetterAction.WRITE_THROUGH: - lines.append(" Assigning writes through the generated setter when available.") - if getter_policy is not None and getter_policy.borrowed: - lines.extend(["", "Notes", "-----", *self._borrowed_view_notes()]) - return "\n".join(lines) - - def _module_array_getter_docstring(self, name, var): - """Handle module array getter docstring for the current generation context.""" - var = self._doc_original_var(var) - notes = self._RESULT_NOTE_DISPATCHER.dispatch(self, var) - lines = [ - f"{name}() -> {self._type_doc(var, include_none=True, signature=True)}", - "", - "Returns", - "-------", - f"{var.name} : {self._type_doc(var, include_none=True)}", - *self._result_detail_lines(var), - ] - if notes: - lines.extend(["", "Notes", "-----", *notes]) - return CommentBlock("\n".join(lines)) - - def _new_python_object(self, name, dtype=None, is_temp=False): - """ - Create new `PythonObjectType` `Variable` with the desired name. - - Create a new `Variable` with the datatype `PythonObjectType` and the desired name. - A `PythonObjectType` datatype means that this variable can be accessed and - manipulated from Python. - - Parameters - ---------- - name : str - The desired name. - - dtype : DataType, optional - The datatype of the object which will be represented by this PyObject. - This is not necessary unless a variable sis required which will describe - a class. - - is_temp : bool, default=False - Indicates if the Variable is temporary. A temporary variable may be ignored - by the printer. - - Returns - ------- - Variable - The new variable. - """ - if isinstance(dtype, CustomDataType): - var = Variable( - self._python_object_map[dtype], - self.scope.get_new_name(name), - memory_handling="alias", - cls_base=self.scope.find(dtype.name, "classes", raise_if_missing=True), - is_temp=is_temp, - ) - else: - var = Variable( - PythonObjectType(), - self.scope.get_new_name(name), - memory_handling="alias", - is_temp=is_temp, - ) - self.scope.insert_variable(var) - return var - - def _get_python_argument_variables(self, args): - """ - Get a new set of `PythonObjectType` `Variable`s representing each of the arguments. - - Create a new `PythonObjectType` variable for each argument returned in Python. - The results are saved to the `self._python_object_map` dictionary so they can be - discovered later. - - Parameters - ---------- - args : iterable of FunctionDefArguments - The arguments of the function. - - Returns - ------- - list of Variable - Variables which will hold the arguments in Python. - """ - orig_args = [getattr(a.var, "original_var", a.var) for a in args] - is_bound = [getattr(a, "wrapping_bound_argument", a.bound_argument) for a in args] - collect_args = [ - self._new_python_object(o_a.name + "_obj", dtype=o_a.dtype if b else None) - for a, b, o_a in zip(args, is_bound, orig_args, strict=False) - ] - self._python_object_map.update(dict(zip(args, collect_args, strict=False))) - return collect_args - - def _unpack_python_args(self, args, class_base=None, *, python_arg_names=None): - """ - Unpack the arguments received from Python into the expected Python variables. - - Create the wrapper arguments of the current `FunctionDef` (`self`, `args`, `kwargs`). - Get a new set of `PythonObjectType` `Variable`s representing each of the expected - arguments. Add the code which unpacks the `args` and `kwargs` into individual - `PythonObjectType`s for each of the expected arguments. - - Parameters - ---------- - args : iterable of FunctionDefArguments - The expected arguments of the function. - - class_base : DataType, optional - The DataType of the class which the method belongs to. In the case of a method - defined in a module this value is None. - - Returns - ------- - func_args : list of Variable - The arguments of the FunctionDef. - - body : list of codegen model object - The code which unpacks the arguments. - - Examples - -------- - >>> arg = Variable('int', 'x') - >>> func_args = (FunctionDefArgument(arg),) - >>> wrapper_args, body = self._unpack_python_args(func_args) - >>> wrapper_args - [Variable('self', dtype=PythonObjectType()), Variable('args', dtype=PythonObjectType()), Variable('kwargs', dtype=PythonObjectType())] - >>> body - [, ] - >>> CPythonCodePrinter('wrapper_file.c').doprint(expr) - static char *kwlist[] = { - "x", - NULL - }; - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O", kwlist, &x_obj)) - { - return NULL; - } - """ - has_bound_arg = class_base is not None - bound_arg = args[0] if has_bound_arg else None - args = args[int(has_bound_arg) :] - if python_arg_names is not None: - python_arg_names = python_arg_names[int(has_bound_arg) :] - # Create necessary variables - func_args = [self._new_python_object("self", class_base)] + [ - self._new_python_object(n) for n in ("args", "kwargs") - ] - arg_vars = self._get_python_argument_variables(args) - keyword_list_name = self.scope.get_new_name("kwlist") - - if has_bound_arg: - self._python_object_map[bound_arg] = func_args[0] - - # Create the list of argument names - if python_arg_names is None: - arg_names = ["" if a.is_posonly else getattr(a.var, "original_var", a.var).name for a in args] - else: - arg_names = ["" if a.is_posonly else name for a, name in zip(args, python_arg_names, strict=False)] - keyword_list = PyArgKeywords(keyword_list_name, arg_names) - - # Parse arguments - parse_node = PyArg_ParseTupleNode(*func_args[1:], args, arg_vars, keyword_list) - - # Initialise optionals - body = [ - AliasAssign(py_arg, self._python_optional_argument_default(func_def_arg)) - for func_def_arg, py_arg in zip(args, arg_vars, strict=False) - if func_def_arg.has_default - ] - - body.append(keyword_list) - body.append(If(IfSection(Not(parse_node), [Return(self._error_exit_code)]))) - - return func_args, body - - def _python_optional_argument_default(self, function_arg): - """Return the Python-object sentinel used before argument parsing.""" - if self._uses_optional_scalar_descriptor_python_presence(function_arg): - return NIL - return Py_None - - @staticmethod - def _uses_optional_scalar_descriptor_python_presence(function_arg): - """Return whether a parsed Python argument must preserve omitted vs None.""" - source_var = getattr(function_arg.var, "original_var", function_arg.var) - decision = getattr(source_var, "ownership_decision", None) - return bool( - function_arg.has_default - and CPythonBindingGenerator._is_optional_scalar_descriptor_var(source_var, decision) - ) - - @staticmethod - def _function_argument_python_name(original_func, function_arg): - """Handle function argument python name for the current generation context.""" - source_var = getattr(function_arg.var, "original_var", function_arg.var) - try: - return original_func.scope.get_python_name(source_var.name) - except RuntimeError: - return str(source_var.name) - - @staticmethod - def _scalar_storage_type_check_condition(py_obj, arg, raise_error): - """Build the rank-0 NumPy storage type check for a visible scalar output.""" - try: - type_ref = numpy_dtype_registry[arg.dtype] - except KeyError: - raise TypeError(f"Can't check the type of scalar storage {arg.dtype}") from None - allow_empty = convert_to_literal(False) - if raise_error: - return pyarray_check( - CStrStr(convert_to_literal(arg.name)), - py_obj, - type_ref, - convert_to_literal(0), - no_order_check, - allow_empty, - ) - return is_numpy_array(py_obj, type_ref, convert_to_literal(0), no_order_check, allow_empty) - - def _get_type_check_condition( - self, - py_obj, - arg, - raise_error, - body, - allow_empty_arrays, - *, - native_scalar_check=None, - ): - """ - Get the condition which checks if an argument has the expected type. - - Using the C-compatible description of a function argument, determine whether the Python - object (with datatype `PythonObjectType`) holds data which is compatible with the expected - type. The check is returned along with any errors that may be raised depending upon the - result and the value of `raise_error`. - - Parameters - ---------- - py_obj : Variable - The variable with datatype `PythonObjectType` where the arguments is stored in Python. - - arg : Variable - The C-compatible variable which holds all the details about the expected type. - - raise_error : bool - True if an error should be raised in case of an unexpected type, False otherwise. - - body : list - A list describing code where the type check will occur. This allows any necessary code - to be inserted into the code block. E.g. code which should be run before the condition - can be checked. - - allow_empty_arrays : bool - A boolean indicating whether empty arrays are authorised. This is necessary as STC - does not handle empty arrays. - - Returns - ------- - type_check_condition : FunctionCall | Variable - The function call which checks if the argument has the expected type or the variable - indicating if the argument has the expected type. - - error_code : tuple of codegen model object - The code which raises any necessary errors. - """ - rank = arg.rank - error_code = () - dtype = arg.dtype - uses_raw_address = self._uses_python_raw_address(arg) - if uses_raw_address: - type_check_condition = Ne(PyLong_Check(py_obj), convert_to_literal(0)) - elif self._uses_python_scalar_storage(arg): - type_check_condition = self._scalar_storage_type_check_condition(py_obj, arg, raise_error) - elif isinstance(dtype, CustomDataType): - python_cls_base = self.scope.find(dtype.name, "classes", raise_if_missing=True) - type_check_condition = PyObject_TypeCheck(py_obj, python_cls_base.type_object) - elif isinstance(dtype, StringType): - type_check_condition = Ne(PyUnicode_Check(py_obj), convert_to_literal(0)) - elif rank == 0: - try: - cast_function = check_type_registry[dtype] - except KeyError: - raise TypeError(f"Can't check the type of {dtype}") from None - func = FunctionDef( - name=cast_function, - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyBoolType(), name="v")), - ) - - type_check_condition = func(py_obj) - if native_scalar_check is not None: - native_func = FunctionDef( - name=native_scalar_check, - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyBoolType(), name="v")), - ) - type_check_condition = Or(type_check_condition, native_func(py_obj)) - elif isinstance(arg.class_type, NumpyNDArrayType): - type_ref = self._numpy_array_type_ref(arg) - if self._is_assumed_rank_array(arg): - type_check_condition = self._assumed_rank_type_check_condition(py_obj, arg, type_ref) - if raise_error: - error_code = ( - PyArgumentError( - PyTypeError, - f"Expected a NumPy array of type {arg.dtype} with rank 1 through " - f"{_MAX_SUPPORTED_ASSUMED_RANK} for argument {arg.name}. " - "Received {type(arg)}", - arg=py_obj, - ), - ) - return type_check_condition, error_code - - # order/contiguity flag - if not arg.class_type.allows_strides: - if rank == 1: - flag = require_any_contiguous - elif arg.order == "F": - flag = require_f_contiguous - else: - flag = require_c_contiguous - elif rank == 1: - flag = no_order_check - elif arg.order == "F": - flag = numpy_flag_f_contig - else: - flag = numpy_flag_c_contig - - allow_empty = convert_to_literal(allow_empty_arrays) - - if raise_error: - type_check_condition = pyarray_check( - CStrStr(convert_to_literal(arg.name)), - py_obj, - type_ref, - convert_to_literal(rank), - flag, - allow_empty, - ) - else: - type_check_condition = is_numpy_array(py_obj, type_ref, convert_to_literal(rank), flag, allow_empty) - - else: - raise TypeError(f"Can't check the type of an array of {arg.class_type}") - - if raise_error and (uses_raw_address or not isinstance(arg.class_type, NumpyNDArrayType)): - # No error code required for arrays as the error is raised inside pyarray_check - python_error = PyArgumentError( - PyTypeError, - f"Expected an argument of type {arg.class_type} for argument {arg.name}. Received {{type(arg)}}", - arg=py_obj, - ) - error_code = (python_error,) - - return type_check_condition, error_code - - @staticmethod - def _is_assumed_rank_array(arg): - """Return whether is assumed rank array.""" - return bool(getattr(arg, "assumed_rank", False) and isinstance(arg.class_type, NumpyNDArrayType)) - - @staticmethod - def _is_character_array(arg): - """Return whether ``arg`` is a fixed-width Fortran character array.""" - variable = getattr(arg, "original_var", arg) - return isinstance(variable.class_type, NumpyNDArrayType) and isinstance(variable.dtype, CharType) - - @staticmethod - def _fixed_character_itemsize(arg): - """Return the compile-time character itemsize, when fixed and numeric.""" - variable = getattr(arg, "original_var", arg) - length = getattr(variable, "fortran_character_length", None) - if length in (None, ":"): - return None - value = getattr(length, "python_value", length) - if isinstance(value, int): - return value - if isinstance(value, str) and value.isdigit(): - return int(value) - return None - - def _numpy_array_type_ref(self, arg): - """Return the NumPy typenum variable for an array contract.""" - variable = getattr(arg, "original_var", arg) - if self._is_character_array(variable): - return numpy_string_type - try: - return numpy_dtype_registry[variable.dtype] - except KeyError: - raise TypeError(f"Can't check the type of an array of {variable.dtype}") from None - - def _array_to_python_call(self, orig_var, data_var, shape_var, itemsize_var, release_memory): - """Build the helper call that converts native array storage to Python.""" - if self._is_character_array(orig_var): - if itemsize_var is None: - raise TypeError(f"Character array result {orig_var.name} is missing itemsize metadata") - return to_numpy_bytes_array( - convert_to_literal(orig_var.rank), - data_var, - shape_var, - itemsize_var, - convert_to_literal(orig_var.order != "F"), - release_memory, - ) - return to_pyarray( - convert_to_literal(orig_var.rank), - self._numpy_array_type_ref(orig_var), - data_var, - shape_var, - convert_to_literal(orig_var.order != "F"), - release_memory, - ) - - @staticmethod - def _array_descriptor_rank(arg): - """Handle array descriptor rank for the current generation context.""" - return _MAX_SUPPORTED_ASSUMED_RANK if CPythonBindingGenerator._is_assumed_rank_array(arg) else arg.rank - - def _assumed_rank_type_check_condition(self, py_obj, arg, type_ref): - """Handle assumed rank type check condition for the current generation context.""" - pyarray = PointerCast(py_obj, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - pyarray_address = ObjectAddress(pyarray) - runtime_rank = PyArray_NDIM(pyarray_address) - return And( - PyArray_Check(py_obj), - Eq(PyArray_TYPE(pyarray_address), type_ref), - Ge(runtime_rank, convert_to_literal(1, dtype=CNativeInt())), - Le(runtime_rank, convert_to_literal(_MAX_SUPPORTED_ASSUMED_RANK, dtype=CNativeInt())), - Or( - Eq(runtime_rank, convert_to_literal(1, dtype=CNativeInt())), - PyArray_CHKFLAGS(pyarray_address, numpy_flag_f_contig), - ), - ) - - def _get_type_check_function(self, name, args, funcs, *, allow_native_scalars=False): - """ - Determine the flags which allow correct function to be identified from the interface. - - Each function must be identifiable by a different integer value. This value is known - as a flag. Different parts of the flag indicate the types of different arguments. - Take for example the following function: - ```python - @types('int', 'int') - @types('float', 'float') - def f(a, b): - pass - ``` - The values 0 (int) and 1 (float) would indicate the type of the argument a. In order - to preserve this information the values which indicate the type of the argument b - must only change the part of the flag which does not contain this information. In other - words `flag % n_types_a = flag_a`. Therefore the values 0 (int) and 2(float) indicate - the type of the argument b. - We then finally have the following four options: - 1. 0 = 0 + 0 => (int,int) - 2. 1 = 1 + 0 => (float,int) - 3. 2 = 0 + 2 => (int, float) - 4. 3 = 1 + 2 => (float, float) - - of which only the first and last flags indicate acceptable arguments. - - The function returns a dictionary whose keys are the functions and whose values are - a list of the flags which would indicate the correct types. - In the above example we would return `{func_0 : [0,0], func_1 : [1,2]}`. - It also returns a FunctionDef which determines the index of the chosen function. - - Parameters - ---------- - name : str - The name of the function to be generated. - - args : iterable of Variable - A list containing the variables of datatype `PythonObjectType` describing the - arguments that were passed to the function from Python. - - funcs : list of FunctionDefs - The functions in the FunctionOverloadSet. - - Returns - ------- - func : FunctionDef - The function which determines the key identifying the relevant function. - - argument_type_flags : dict - A dictionary whose keys are the functions and whose values are the integer keys - which indicate that the function should be chosen. - """ - args = [a.clone(a.name, is_argument=True) for a in args] - func_scope = self.scope.new_child_scope(name, "function") - self.scope = func_scope - orig_funcs = [getattr(func, "original_function", func) for func in funcs] - type_indicator = Variable(NumpyInt64Type(), self.scope.get_new_name("type_indicator")) - is_bind_c = isinstance(funcs[0], BindCFunctionDef) - - # Initialise the argument_type_flags - argument_type_flags = dict.fromkeys(funcs, 0) - - # Initialise type_indicator - body = [Assign(type_indicator, convert_to_literal(0))] - - step = 1 - for i, py_arg in enumerate(args): - # Get the relevant typed arguments from the original functions - interface_args = [func.arguments[i].var for func in orig_funcs] - # Get a dictionary mapping each unique type key to an example argument - type_to_example_arg = {a.class_type: a for a in interface_args} - # Get a list of unique keys - possible_types = list(type_to_example_arg.keys()) - native_scalar_checks = {} - if allow_native_scalars: - family_counts = {} - for possible_type in possible_types: - if not isinstance(possible_type, FixedSizeNumericType): - continue - primitive_type = possible_type.primitive_type - family_counts[type(primitive_type)] = family_counts.get(type(primitive_type), 0) + 1 - native_check_names = { - PrimitiveIntegerType: "PyIs_NativeInt", - PrimitiveFloatingPointType: "PyIs_NativeFloat", - PrimitiveComplexType: "PyIs_NativeComplex", - } - for possible_type in possible_types: - if not isinstance(possible_type, FixedSizeNumericType): - continue - primitive_cls = type(possible_type.primitive_type) - if family_counts[primitive_cls] == 1 and primitive_cls in native_check_names: - native_scalar_checks[possible_type] = native_check_names[primitive_cls] - - n_possible_types = len(possible_types) - if orig_funcs[0].arguments[i].has_default: - # The default must have a type that can be deduced so this can be checked - # in the wrapper of the implementation - pass - elif n_possible_types != 1: - # Update argument_type_flags with the index of the type key - for func, a in zip(funcs, interface_args, strict=False): - index = next(i for i, p_t in enumerate(possible_types) if p_t is a.class_type) * step - argument_type_flags[func] += index - - # Create the type checks and incrementation of the type_indicator - if_blocks = [] - for index, t in enumerate(possible_types): - check_func_call, _ = self._get_type_check_condition( - py_arg, - type_to_example_arg[t], - False, - body, - allow_empty_arrays=is_bind_c, - native_scalar_check=native_scalar_checks.get(t), - ) - if_blocks.append( - IfSection( - check_func_call, - [AugAssign(type_indicator, "+", convert_to_literal(index * step))], - ) - ) - body.append( - If( - *if_blocks, - IfSection( - convert_to_literal(True), - [ - PyArgumentError( - PyTypeError, - f"Unexpected type for argument {interface_args[0].name}. Received {{type(arg)}}", - arg=py_arg, - ), - Return(convert_to_literal(-1)), - ], - ), - ) - ) - else: - check_func_call, err_body = self._get_type_check_condition( - py_arg, - type_to_example_arg.popitem()[1], - True, - body, - allow_empty_arrays=is_bind_c, - native_scalar_check=next(iter(native_scalar_checks.values()), None), - ) - err_body = (*err_body, Return(convert_to_literal(-1))) - if_sec = IfSection(Not(check_func_call), err_body) - body.append(If(if_sec)) - - # Update the step to ensure unique indices for each argument - step *= n_possible_types - - body.append(Return(type_indicator)) - - self.exit_scope() - - docstring = CommentBlock( - "Assess the types. Raise an error for unexpected types and calculate an integer\n" - + "which indicates which function should be called." - ) - - # Build the function - func = FunctionDef( - name, - [FunctionDefArgument(a) for a in args], - body, - FunctionDefResult(type_indicator), - docstring=docstring, - scope=func_scope, - ) - - return func, argument_type_flags - - def _save_referenced_objects(self, func, func_args): - """ - Save any arguments passed to the wrapper which are then stored in pointers. - - If arguments are saved into pointers (e.g. inside classes) then their reference - counter must be incremented. This prevents them being deallocated if they go - out of scope in Python. The class must then take care to decrement their - reference counter when it is itself deallocated to prevent a memory leak. - The attribute `FunctionDefArgument.persistent_target` indicates whether an - argument is a target inside the function. When it is true then additional code - is added to the wrapper body. This code increments the reference counter for - the argument and adds the object to a list of objects whose reference counter - must be decremented in the class destructor. - - Parameters - ---------- - func : FunctionDef - The function being wrapped. - func_args : list[FunctionDefArgument] | list[Variable] - The arguments passed by Python to the function (self, args, kwargs). - - Returns - ------- - list - A list of any expressions which should be added to the wrapper body to - add references to the arguments. - """ - body = [] - class_arg_var = func_args[0] - if isinstance(class_arg_var, FunctionDefArgument): - class_arg_var = class_arg_var.var - class_scope = class_arg_var.cls_base.scope - for a in func.arguments: - if a.persistent_target: - ref_attribute = class_scope.find("referenced_objects", "variables", raise_if_missing=True) - ref_list = ref_attribute.clone(ref_attribute.name, new_class=DottedVariable, lhs=class_arg_var) - python_arg = self._python_object_map[a] - if not isinstance(python_arg.dtype, PythonObjectType): - python_arg = ObjectAddress(PointerCast(python_arg, PyList_Append.arguments[1].var)) - append_call = PyList_Append(ref_list, python_arg) - body.extend( - [ - If( - IfSection( - Eq(append_call, convert_to_literal(-1)), - [Return(self._error_exit_code)], - ) - ) - ] - ) - return body - - def _incref_borrowed_array_getter(self, _orig_var, _decision, ref_obj, return_var): - """Retain the owner of an array returned by a borrowed getter.""" - save_ref_call = PyArray_SetBaseObject( - ObjectAddress(PointerCast(return_var, PyArray_SetBaseObject.arguments[0].var)), - ObjectAddress(PointerCast(ref_obj, PyArray_SetBaseObject.arguments[1].var)), - ) - return [ - Py_INCREF(ref_obj), - If( - IfSection( - Lt(save_ref_call, convert_to_literal(0, dtype=CNativeInt())), - [Return(self._error_exit_code)], - ) - ), - ] - - def _incref_borrowed_custom_getter(self, _orig_var, _decision, ref_obj, return_var): - """Retain the owner of a derived value returned by a borrowed getter.""" - ref_attribute = return_var.cls_base.scope.find("referenced_objects", "variables", raise_if_missing=True) - ref_list = ref_attribute.clone(ref_attribute.name, new_class=DottedVariable, lhs=return_var) - save_ref_call = PyList_Append(ref_list, ObjectAddress(PointerCast(ref_obj, ref_list))) - return [ - If( - IfSection( - Lt(save_ref_call, convert_to_literal(0, dtype=CNativeInt())), - [Return(self._error_exit_code)], - ) - ) - ] - - def _add_object_to_mod(self, module_var, obj, name, initialised): - """ - Get code for adding an object to the module. - - This function creates the AST nodes necessary to add an object to - the module. This includes the creation of the success check and - the dereferencing of any objects used. - - Parameters - ---------- - module_var : Variable - The variable containing the PyObject* which describes the module. - - obj : Variable - The variable containing the PyObject* which should be added to the module. - - name : str - The name by which the object will be known in X2py. - - initialised : list[Variable] - A list of the variables which have had their reference counter incremented - and must therefore decrement their counter if an error is raised. - - Returns - ------- - list[model object] - The code which adds the object to the module. - """ - add_expr = PyModule_AddObject(module_var, CStrStr(convert_to_literal(name)), obj) - if_expr = If( - IfSection( - Lt(add_expr, convert_to_literal(0)), - [Py_DECREF(i) for i in initialised] + [Return(self._error_exit_code)], - ) - ) - initialised.append(obj) - return [if_expr, Py_INCREF(obj)] - - def _allocate_class_instance(self, class_var, scope, is_alias): - """ - Get all expressions necessary to allocate a new class description. - - Get all expressions necessary to allocate a new class description, this includes allocating - the object itself, creating the list of referenced_objects and saving the alias status. - - Parameters - ---------- - class_var : Variable - The variable where the class instance is stored. - - scope : Scope - The scope of the class (containing the class attributes). - - is_alias : bool - A boolean indicating if an alias is being stored. - - Returns - ------- - list[model object] - A list of expressions necessary to allocate a new class description. - """ - # Get the list of referenced objects - ref_attribute = scope.find("referenced_objects", "variables", raise_if_missing=True) - ref_list = ref_attribute.clone(ref_attribute.name, new_class=DottedVariable, lhs=class_var) - - # Get alias attribute - attribute = scope.find("is_alias", "variables", raise_if_missing=True) - alias_bool = attribute.clone(attribute.name, new_class=DottedVariable, lhs=class_var) - - alias_val = convert_to_literal(True) if is_alias else convert_to_literal(False) - - return [ - Allocate(class_var, shape=None, status="unallocated"), - AliasAssign(ref_list, PyList_New()), - Assign(alias_bool, alias_val), - ] - - def _get_class_allocator(self, class_dtype, func=None): - """ - Create the allocator for the class. - - Create a function which will allocate the memory for the class instance. This - is equivalent to the `__new__` function. - - Parameters - ---------- - class_dtype : DataType - The datatype of the class being translated. - - func : FunctionDef, optional - The function which provides a new instance of the class. - - Returns - ------- - PyFunctionDef - A function that can be called to create the class instance. - """ - if func: - func_name = self.scope.get_new_name(f"{func.name}__wrapper", object_type="wrapper") - else: - func_name = self.scope.get_new_name(f"{class_dtype.name}__new__wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - - self_var = Variable( - PythonTypeObjectType(), - name=self.scope.get_new_name("self"), - memory_handling="alias", - ) - self.scope.insert_variable(self_var, "self") - func_args = [self_var] + [self._new_python_object(n) for n in ("args", "kwargs")] - func_args = [FunctionDefArgument(a) for a in func_args] - - func_results = FunctionDefResult(self._new_python_object("result", is_temp=True)) - - # Get the results of the PyFunctionDef - python_result_var = self._new_python_object("result_obj", class_dtype) - scope = python_result_var.cls_base.scope - attribute = scope.find("instance", "variables", raise_if_missing=True) - c_res = attribute.clone(attribute.name, new_class=DottedVariable, lhs=python_result_var) - - body = self._allocate_class_instance(python_result_var, scope, False) - - if func: - body.append(AliasAssign(c_res, func())) - else: - result_name = self.scope.get_new_name("result") - result = Variable(class_dtype, result_name) - body.append(Allocate(c_res, shape=None, status="unallocated", like=result)) - - body.append(Return(PointerCast(python_result_var, func_results.var))) - - self.exit_scope() - - return PyFunctionDef( - func_name, - func_args, - body, - func_results, - scope=func_scope, - original_function=None, - ) - - def _get_class_initialiser(self, init_function, cls_dtype): - """ - Create the constructor for the class. - - Create a function which will initialise the class. This function creates - the `__new__` function to allocate the memory which stores the class - instance and calls the `__init__` function. - - Parameters - ---------- - init_function : FunctionDef - The `__init__` function in the translated class. - - cls_dtype : DataType - The datatype of the class being translated. - - Returns - ------- - new_function : PyFunctionDef - A function that can be called to create the class instance. - - init_function : PyFunctionDef - A function that can be called to create the class instance. - """ - original_func = getattr(init_function, "original_function", init_function) - func_name = self.scope.get_new_name(f"{cls_dtype.name}__init__wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - self._error_exit_code = convert_to_literal(-1, dtype=CNativeInt()) - - isinstance(init_function, BindCFunctionDef) - - # Add the variables to the expected symbols in the scope - for a in init_function.arguments: - a_var = a.var - func_scope.insert_symbol(getattr(a_var, "original_var", a_var).name) - - # Get variables describing the arguments and results that are seen from Python - python_args = init_function.arguments - - # Get the arguments of the PyFunctionDef - func_args, body = self._unpack_python_args(python_args, cls_dtype) - func_args = [FunctionDefArgument(a) for a in func_args] - - # Get the results of the PyFunctionDef - python_result_variable = Variable(CNativeInt(), self.scope.get_new_name(), is_temp=True) - - # Get the code required to extract the C-compatible arguments from the Python arguments - wrapped_args = [self._visit(a) for a in python_args] - body += [line for arg in wrapped_args for line in arg["body"]] - callback_setup = [line for arg in wrapped_args for line in arg.get("callback_setup", ())] - callback_cleanup = [line for arg in reversed(wrapped_args) for line in arg.get("callback_cleanup", ())] - - # Get the arguments and results which should be used to call the c-compatible function - func_call_args = [ca for a in wrapped_args for ca in a["args"]] - - body.extend(self._save_referenced_objects(init_function, func_args)) - - # Call the C-compatible function - body.extend(callback_setup) - body.extend( - self._native_call_nodes( - init_function, - original_func, - func_call_args, - [], - wrapped_args, - force_hold=True, - ) - ) - body.extend(callback_cleanup) - - # Pack the Python compatible results of the function into one argument. - func_results = FunctionDefResult(python_result_variable) - body.append(Return(convert_to_literal(0, dtype=CNativeInt()))) - - self.exit_scope() - for a in python_args: - if not a.bound_argument: - self._python_object_map.pop(a) - - function = PyFunctionDef( - func_name, - func_args, - body, - func_results, - scope=func_scope, - docstring=init_function.docstring, - original_function=original_func, - ) - - self.scope.insert_function(function, func_scope.get_python_name(func_name)) - self._python_object_map[init_function] = function - self._error_exit_code = NIL - - return function - - @staticmethod - def _default_constructor_property(prop): - """Handle default constructor property for the current generation context.""" - setter = prop.setter - if setter is None: - return None - source_property = getattr(setter, "original_function", None) - if not isinstance(source_property, BindCClassProperty): - return None - if source_property.setter_policy.setter_action is not SetterAction.WRITE_THROUGH: - return None - return prop - - def _get_default_class_initialiser(self, wrapped_class, cls_dtype): - """Create the generated keyword-only component initializer.""" - init_name = wrapped_class.original_class.scope.get_new_name("__init__", object_type="function") - original_function = FunctionDef( - init_name, - [], - [], - FunctionDefResult(NIL), - scope=wrapped_class.original_class.scope, - ) - properties = [ - prop - for prop in (self._default_constructor_property(item) for item in wrapped_class.properties) - if prop is not None - ] - - func_name = self.scope.get_new_name(f"{cls_dtype.name}__default_init_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - self._error_exit_code = convert_to_literal(-1, dtype=CNativeInt()) - - bound_arg = FunctionDefArgument( - Variable(cls_dtype, self.scope.get_new_name("self"), cls_base=wrapped_class.original_class), - bound_argument=True, - ) - field_args = [ - FunctionDefArgument( - Variable(PythonObjectType(), prop.python_name, memory_handling="alias"), - value=Py_None, - kwonly=True, - ) - for prop in properties - ] - unpack_args = [bound_arg, *field_args] - func_args, body = self._unpack_python_args(unpack_args, cls_dtype) - self_obj = func_args[0] - - for prop, field_arg in zip(properties, field_args, strict=True): - field_obj = self._python_object_map[field_arg] - body.append( - If( - IfSection( - IsNot(field_obj, Py_None), - [ - If( - IfSection( - Lt( - prop.setter(self_obj, field_obj, NIL), convert_to_literal(0, dtype=CNativeInt()) - ), - [Return(self._error_exit_code)], - ) - ) - ], - ) - ) - ) - body.append(Return(convert_to_literal(0, dtype=CNativeInt()))) - result = FunctionDefResult(self.scope.get_temporary_variable(CNativeInt())) - self.exit_scope() - - for arg in unpack_args: - self._python_object_map.pop(arg, None) - - function = PyFunctionDef( - func_name, - [FunctionDefArgument(arg) for arg in func_args], - body, - result, - scope=func_scope, - original_function=original_function, - ) - self.scope.insert_function(function, func_scope.get_python_name(func_name)) - self._error_exit_code = NIL - return function - - def _get_blocked_class_initialiser(self, wrapped_class, cls_dtype): - """Reject public construction when an edited contract removed ``__init__``.""" - init_name = wrapped_class.original_class.scope.get_new_name("__init__", object_type="function") - original_function = FunctionDef( - init_name, - [], - [], - FunctionDefResult(NIL), - scope=wrapped_class.original_class.scope, - ) - func_name = self.scope.get_new_name(f"{cls_dtype.name}__blocked_init_wrapper", object_type="wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - self._error_exit_code = convert_to_literal(-1, dtype=CNativeInt()) - - bound_arg = FunctionDefArgument( - Variable(cls_dtype, self.scope.get_new_name("self"), cls_base=wrapped_class.original_class), - bound_argument=True, - ) - func_args, body = self._unpack_python_args([bound_arg], cls_dtype) - body.extend( - ( - PyErr_SetString( - PyTypeError, - CStrStr( - convert_to_literal( - f"{wrapped_class.name} has no public constructor in the edited .pyi contract" - ) - ), - ), - Return(self._error_exit_code), - ) - ) - result = FunctionDefResult(self.scope.get_temporary_variable(CNativeInt())) - self.exit_scope() - self._python_object_map.pop(bound_arg, None) - - function = PyFunctionDef( - func_name, - [FunctionDefArgument(arg) for arg in func_args], - body, - result, - scope=func_scope, - original_function=original_function, - ) - self.scope.insert_function(function, func_scope.get_python_name(func_name)) - self._error_exit_code = NIL - return function - - @staticmethod - def _suppresses_default_class_initialiser(cls): - """Return whether suppresses default class initialiser.""" - current = cls - while current is not None: - decorators = getattr(current, "decorators", {}) - if hasattr(decorators, "get") and decorators.get(SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA): - return True - next_class = getattr(current, "original_class", None) - if next_class is current: - return False - current = next_class - return False - - def _get_class_destructor(self, del_function, cls_dtype, wrapper_scope): - """ - Create the destructor for the class. - - Create a function which will act as a destructor for the class. This - function calls the `__del__` function and frees the memory allocated - to store the class instance. - - Parameters - ---------- - del_function : FunctionDef - The `__del__` function in the translated class. - - cls_dtype : DataType - The datatype of the class being translated. - - wrapper_scope : Scope - The scope for the wrapped version of the class. - - Returns - ------- - PyFunctionDef - A function that can be called to destroy the class instance. - """ - original_func = getattr(del_function, "original_function", del_function) - func_name = self.scope.get_new_name(f"{cls_dtype.name}__del__wrapper") - func_scope = self.scope.new_child_scope(func_name, "function") - self.scope = func_scope - - # Add the variables to the expected symbols in the scope - for a in del_function.arguments: - func_scope.insert_symbol(a.var.name) - func_arg = self._new_python_object("self", cls_dtype) - - attribute = wrapper_scope.find("instance", "variables") - c_obj = attribute.clone(attribute.name, new_class=DottedVariable, lhs=func_arg) - - attribute = wrapper_scope.find("is_alias", "variables") - is_alias = attribute.clone(attribute.name, new_class=DottedVariable, lhs=func_arg) - - if isinstance(del_function, BindCFunctionDef): - body = [del_function(c_obj)] - else: - body = [del_function(c_obj), Deallocate(c_obj)] - body.append(AliasAssign(c_obj, NIL)) - body = [If(IfSection(Not(is_alias), body))] - - # Get the list of referenced objects - ref_attribute = wrapper_scope.find("referenced_objects", "variables", raise_if_missing=True) - ref_list = ref_attribute.clone(ref_attribute.name, new_class=DottedVariable, lhs=func_arg) - - body.extend([Py_DECREF(ref_list), Deallocate(func_arg)]) - - self.exit_scope() - - function = PyFunctionDef( - func_name, - [FunctionDefArgument(func_arg)], - body, - scope=func_scope, - original_function=original_func, - ) - - self.scope.insert_function(function, func_scope.get_python_name(func_name)) - self._python_object_map[del_function] = function - - return function - - def _get_array_parts(self, orig_var, collect_arg): - """ - Get AST nodes describing the extraction of the data pointer, shape, and strides from a Python array object. - - Get AST nodes describing the extraction of the data pointer, shape, and strides from a Python array object. - These nodes as well as the new objects can then be packed into a structure or passed directly to a function - depending on the target language. - - Parameters - ---------- - orig_var : Variable | IndexedElement - An object representing the variable or an element of the variable from the - FunctionDefArgument being wrapped. - - collect_arg : Variable - A variable with type PythonObject* holding the Python argument from which the - C-compatible argument should be collected. - - Returns - ------- - dict[str, Any] - A dictionary with the keys: - - body : a list containing the AST nodes which extract the data pointer, shape, and strides. - - data : a Variable describing a pointer in which the data is stored. - - shape : a Variable describing a stack array in which the shape information is stored. - - strides : a Variable describing a stack array in which the strides are stored. - """ - pyarray_collect_arg = PointerCast(collect_arg, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - data_var = Variable( - VoidType(), - self.scope.get_new_name(orig_var.name + "_data"), - memory_handling="alias", - ) - itemsize_var = ( - self.scope.get_temporary_variable(NumpyInt64Type(), name=f"{orig_var.name}_itemsize") - if self._is_character_array(orig_var) - else None - ) - descriptor_rank = self._array_descriptor_rank(orig_var) - actual_rank_var = ( - self.scope.get_temporary_variable(NumpyInt64Type(), name=f"{orig_var.name}_rank") - if self._is_assumed_rank_array(orig_var) - else None - ) - base_shape_var = Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - self.scope.get_new_name(orig_var.name + "_base_shape"), - shape=(descriptor_rank,), - ) - ubound_var = Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - self.scope.get_new_name(orig_var.name + "_ubound"), - shape=(descriptor_rank,), - ) - stride_var = Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - self.scope.get_new_name(orig_var.name + "_strides"), - shape=(descriptor_rank,), - ) - self.scope.insert_variable(data_var) - self.scope.insert_variable(base_shape_var) - self.scope.insert_variable(ubound_var) - self.scope.insert_variable(stride_var) - - get_data = AliasAssign(data_var, PyArray_DATA(ObjectAddress(pyarray_collect_arg))) - get_strides_and_shape = get_strides_and_shape_from_numpy_array( - ObjectAddress(collect_arg), - base_shape_var, - ubound_var, - stride_var, - convert_to_literal(False if self._is_assumed_rank_array(orig_var) else orig_var.order != "F"), - ) - - body = [get_data] - if actual_rank_var is not None: - body.append( - Assign( - actual_rank_var, - cast_to(PyArray_NDIM(ObjectAddress(pyarray_collect_arg)), NumpyInt64Type()), - ) - ) - if itemsize_var is not None: - body.append( - Assign( - itemsize_var, - cast_to(PyArray_ITEMSIZE(ObjectAddress(pyarray_collect_arg)), NumpyInt64Type()), - ) - ) - body.append(get_strides_and_shape) - - return { - "body": body, - "data": data_var, - "itemsize": itemsize_var, - "rank": actual_rank_var, - "shape": base_shape_var, - "ubounds": ubound_var, - "strides": stride_var, - } - - def _call_wrapped_function(self, func, args, results): - """ - Call the wrapped function. - - Call the wrapped function. The call is either a FunctionCall, an Assign or - an AliasAssign depending on the number of results and the return type. - - Parameters - ---------- - func : FunctionDef - The function being wrapped. - args : iterable[model object] - The arguments passed to the wrapped function. - results : iterable[model object] - The results returned from the wrapped function. - - Returns - ------- - FunctionCall | Assign | AliasAssign - An AST node describing the function call. - """ - n_results = len(results) - if n_results == 0: - return func(*args) - if isinstance(results, PythonTuple): - return Assign(results, func(*args)) - if n_results == 1: - res = results[0] - func_call = func(*args) - if func_call.is_alias and self._returns_address_projected_value(func): - return Assign(res, func_call) - if func_call.is_alias: - if isinstance(res, PointerCast): - res = res.obj - if isinstance(res, ObjectAddress): - res = res.obj - return AliasAssign(res, func_call) - return Assign(res, func_call) - return Assign(results, func(*args)) - - @staticmethod - def _returns_address_projected_value(func) -> bool: - """Return whether an alias-marked native result is emitted as a value result.""" - result = getattr(func.results, "var", NIL) - original = getattr(result, "original_var", result) - decision = getattr(original, "ownership_decision", None) - return bool( - decision is not None - and decision.kind is ObjectKind.SCALAR - and decision.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS - and not decision.descriptor_boundary - ) - - @staticmethod - def _native_call_holds_gil(original_func, wrapped_args, *, force_hold=False): - """Handle native call holds gil for the current generation context.""" - decorators = getattr(original_func, "decorators", {}) - return bool( - force_hold - or decorators.get(RUNTIME_HOLD_GIL_METADATA) - or "property" in decorators - or any(arg.get("callback_setup") for arg in wrapped_args) - ) - - def _native_call_nodes(self, func, original_func, args, results, wrapped_args, *, force_hold=False): - """Handle native call nodes for the current generation context.""" - call = self._call_wrapped_function(func, args, results) - if self._native_call_holds_gil(original_func, wrapped_args, force_hold=force_hold): - return [call] - return [PyAllowThreadsBegin(), call, PyAllowThreadsEnd()] - - @staticmethod - def _status_error_output_names(original_func): - """Handle status error output names for the current generation context.""" - policy = getattr(original_func, "decorators", {}).get(RUNTIME_STATUS_ERROR_METADATA) - if not isinstance(policy, NativeStatusErrorPolicy): - return set() - names = {policy.status.name} - if policy.message is not None: - names.add(policy.message.name) - return names - - @staticmethod - def _result_bindings_by_name(wrapped_results): - """Handle result bindings by name for the current generation context.""" - bindings = {} - for binding in wrapped_results.get("result_bindings", ()): - name = binding.get("name") - if isinstance(name, str): - bindings[name] = binding - return bindings - - @staticmethod - def _status_error_bindings(policy, bindings): - """Resolve already-completed status outputs in the lowered binding graph.""" - status = bindings.get(policy.status.name) - if status is None: - raise ValueError(f"completed raises status target {policy.status.name!r} is missing after lowering") - message = None - if policy.message is not None: - message = bindings.get(policy.message.name) - if message is None: - raise ValueError(f"completed raises message target {policy.message.name!r} is missing after lowering") - return status, message - - @staticmethod - def _status_error_exception(policy): - """Lower one completed Python exception kind without datatype inference.""" - if policy.exception_kind is PythonExceptionKind.RUNTIME_ERROR: - return PyRuntimeError - raise ValueError(f"Unsupported completed Python exception kind: {policy.exception_kind!r}") - - def _status_error_check( - self, - original_func, - wrapped_results, - native_py_results, - native_owned_results, - cleanup, - ): - """Handle status error check for the current generation context.""" - policy = getattr(original_func, "decorators", {}).get(RUNTIME_STATUS_ERROR_METADATA) - if not isinstance(policy, NativeStatusErrorPolicy): - return [] - - bindings = self._result_bindings_by_name(wrapped_results) - status, message = self._status_error_bindings(policy, bindings) - status_var = status["c_result"] - success = policy.success - exception = self._status_error_exception(policy) - if message is not None: - set_error = PyErr_SetObject(exception, message["py_result"]) - else: - set_error = PyErr_SetString( - exception, - CStrStr(convert_to_literal(f"native call failed with status {status['name']} != {success}")), - ) - error_body = [ - set_error, - *(Py_DECREF(item) for item, owned in zip(native_py_results, native_owned_results, strict=False) if owned), - *cleanup, - Return(self._error_exit_code), - ] - return [ - If( - IfSection( - Ne(status_var, convert_to_literal(success, dtype=status_var.dtype)), - error_body, - ) - ) - ] - - def _project_python_return( - self, - func, - original_func, - native_py_results, - native_owned_results, - *, - excluded_output_names=(), - ): - """Handle project python return for the current generation context.""" - output_items = [] - output_owned = [] - discarded_owned_items = [] - excluded = set(excluded_output_names) - native_index = self._project_native_function_result( - original_func, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ) - - projected_argument_objects = self._projected_argument_objects(func) - for argument in original_func.arguments: - native_index = self._project_argument_return( - argument, - native_index, - native_py_results, - native_owned_results, - excluded, - projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ) - return self._pack_projected_python_return(output_items, output_owned, discarded_owned_items) - - @staticmethod - def _append_projected_native_result( - index, - name, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ) -> None: - """Append or discard one native result according to exclusions.""" - if name not in excluded: - output_items.append(native_py_results[index]) - output_owned.append(native_owned_results[index]) - elif native_owned_results[index]: - discarded_owned_items.append(native_py_results[index]) - - def _project_native_function_result( - self, - original_func, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ): - """Project the explicit native function result when one exists.""" - result_var = original_func.results.var - if result_var is NIL: - return 0 - result_name = getattr(result_var, "name", None) - if isinstance(getattr(result_var, "class_type", None), BindCResultTupleType): - for index in range(len(native_py_results)): - self._append_projected_native_result( - index, - result_name, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ) - return len(native_py_results) - self._append_projected_native_result( - 0, - result_name, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ) - return 1 - - def _project_argument_return( - self, - argument, - native_index, - native_py_results, - native_owned_results, - excluded, - projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ): - """Project one output argument into the Python return sequence.""" - orig_var = argument.var - if isinstance(orig_var, FunctionAddress) or argument.bound_argument: - return native_index - return self._ARGUMENT_RETURN_PROJECTION_DISPATCHER.dispatch( - self, - orig_var, - native_index, - native_py_results, - native_owned_results, - excluded, - projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ) - - def _skip_argument_return_projection( - self, - _orig_var, - _decision, - native_index, - _native_py_results, - _native_owned_results, - _excluded, - _projected_argument_objects, - _output_items, - _output_owned, - _discarded_owned_items, - ): - """Leave one non-projected argument out of the Python return sequence.""" - return native_index - - def _project_native_argument_return( - self, - orig_var, - _decision, - native_index, - native_py_results, - native_owned_results, - excluded, - _projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ): - """Project one native output object produced for an argument.""" - output_name = getattr(orig_var, "name", None) - self._append_projected_native_result( - native_index, - output_name, - native_py_results, - native_owned_results, - excluded, - output_items, - output_owned, - discarded_owned_items, - ) - return native_index + 1 - - def _project_visible_argument_return( - self, - orig_var, - _decision, - native_index, - native_py_results, - native_owned_results, - excluded, - projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ): - """Project a caller-supplied Python object that native code mutated.""" - output_name = getattr(orig_var, "name", None) - visible_object = projected_argument_objects.get(orig_var) or projected_argument_objects.get(output_name) - if visible_object is not None: - if output_name not in excluded: - output_items.append(visible_object) - output_owned.append(False) - return native_index - return self._project_native_argument_return( - orig_var, - _decision, - native_index, - native_py_results, - native_owned_results, - excluded, - projected_argument_objects, - output_items, - output_owned, - discarded_owned_items, - ) - - def _pack_projected_python_return(self, output_items, output_owned, discarded_owned_items): - """Pack projected Python outputs and apply ownership cleanup.""" - decrefs = [Py_DECREF(item) for item in discarded_owned_items] - if not output_items: - return {"body": [*decrefs, Py_INCREF(Py_None)], "result": Py_None, "owned_result": False} - if len(output_items) == 1: - if not output_owned[0]: - return { - "body": [*decrefs, Py_INCREF(output_items[0])], - "result": output_items[0], - "owned_result": False, - } - return {"body": decrefs, "result": output_items[0], "owned_result": True} - tuple_result = self._new_python_object("result_obj") - body = [*decrefs, AliasAssign(tuple_result, PyTuple_Pack(*(ObjectAddress(item) for item in output_items)))] - body.append(If(IfSection(Is(tuple_result, NIL), [Return(self._error_exit_code)]))) - body.extend(Py_DECREF(item) for item, owned in zip(output_items, output_owned, strict=False) if owned) - return {"body": body, "result": tuple_result, "owned_result": True} - - def _projected_argument_objects(self, func): - """Return Python argument objects that are also projected as results.""" - outputs = {} - for argument in func.arguments: - var = argument.var - orig_var = getattr(var, "original_var", var) - if isinstance(orig_var, FunctionAddress): - continue - self._PROJECTED_ARGUMENT_OBJECT_DISPATCHER.dispatch(self, orig_var, argument, outputs) - return outputs - - def _skip_projected_argument_object(self, _orig_var, _decision, _argument, _outputs): - """Leave one argument out of the projected-object lookup.""" - - def _record_projected_argument_object(self, orig_var, _decision, argument, outputs): - """Record the Python argument object selected as a projected result.""" - outputs[orig_var] = self._python_object_map[argument] - outputs[getattr(orig_var, "name", None)] = self._python_object_map[argument] - - def _connect_pointer_targets(self, orig_var, python_res, funcdef, is_bind_c): - """ - Get the code to connect pointers to their targets. - - Get the code to connect pointers to their targets. The connection is done via reference - counting to ensure that the target is not cleaned by the garbage collector before the - pointer. - - Parameters - ---------- - orig_var : Variable - The result of the function being wrapped. - python_res : Variable - The Python accessible result of the function being wrapped. - funcdef : FunctionDef - The function being wrapped. - is_bind_c : bool - True if the code is translated from a C-compatible language. False if the - translated code is in C. - - Returns - ------- - list - Any nodes which must be printed to increase reference counts. - """ - python_args = funcdef.arguments - arg_targets = funcdef.result_pointer_map.get(orig_var, ()) - n_targets = len(arg_targets) - if n_targets == 1: - collect_arg = self._python_object_map[python_args[arg_targets[0]]] - return self._incref_return_pointer(collect_arg, python_res, orig_var) - if n_targets > 1: - if isinstance(orig_var.class_type, NumpyNDArrayType): - raise RuntimeError( - f"Can't determine the pointer target for the return object {orig_var}. " - "Please avoid calling this function to prevent accidental creation of dangling pointers." - ) - body = [] - for t in arg_targets: - collect_arg = self._python_object_map[python_args[t]] - body.extend(self._incref_return_pointer(collect_arg, python_res, orig_var)) - return body - return [] - - # -------------------------------------------------------------------------------------------------------------------------------------------- - - @staticmethod - def _module_literal_value(expr): - """Convert a semantic `.pyi` literal into a typed codegen literal.""" - value = expr.default_value - if value is None: - raise ValueError(f"Module value {expr.name} needs a literal value before wrapper generation") - dtype = expr.class_type.underlying_type if isinstance(expr.class_type, FinalType) else expr.class_type - text = str(value).strip() - if isinstance(dtype, NumpyBoolType): - return convert_to_literal(text.lower() in {".true.", "true", "1"}, dtype=dtype) - if isinstance(dtype, StringType): - return convert_to_literal(str(ast.literal_eval(text)), dtype=dtype) - if isinstance(dtype.primitive_type, PrimitiveIntegerType): - return convert_to_literal(int(ast.literal_eval(text)), dtype=dtype) - if isinstance(dtype.primitive_type, PrimitiveFloatingPointType): - return convert_to_literal(float(text.replace("d", "e").replace("D", "E")), dtype=dtype) - if isinstance(dtype.primitive_type, PrimitiveComplexType): - parts = ast.literal_eval(text.replace("d", "e").replace("D", "E")) - return convert_to_literal(complex(parts[0], parts[1]), dtype=dtype) - raise TypeError(f"No Python constant conversion registered for {expr.class_type}") - - _module_constant_literal = _module_literal_value - - def _get_allocatable_module_array_getter(self, expr): - """Return allocatable module array getter.""" - python_name = f"get_{self.scope.get_python_name(expr.name)}" - wrapper_name = self.scope.get_new_name(f"{python_name}_wrapper", object_type="wrapper") - original_name = self.scope.get_new_public_name( - python_name, - object_type="function", - owner=f"module array getter {python_name}", - ) - original = FunctionDef( - original_name, - (), - (), - FunctionDefResult(expr), - scope=self.scope, - decorators={ - INTERNAL_MODULE_VARIABLE_NAME_METADATA: expr.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: "get", - }, - ) - func_scope = self.scope.new_child_scope(wrapper_name, "function") - self.scope = func_scope - - func_args, body = self._unpack_python_args(()) - body.extend(self._visit_BindCArrayVariable(expr)) - py_result = self._python_object_map.pop(expr) - body.append(Return(py_result)) - self.exit_scope() - - return PyFunctionDef( - wrapper_name, - [FunctionDefArgument(arg) for arg in func_args], - body, - FunctionDefResult(py_result), - scope=func_scope, - docstring=self._module_array_getter_docstring(python_name, expr), - original_function=original, - ) - - def _return_none_if_unallocated(self, data_ptr, shape_vars=()): - """Handle return none if unallocated for the current generation context.""" - return [ - If( - IfSection( - Is(data_ptr, NIL), - [ - *self._raise_memory_error_if_shape_is_nonzero(shape_vars), - Py_INCREF(Py_None), - Return(Py_None), - ], - ) - ) - ] - - def _set_none_if_unallocated(self, data_ptr, py_res, shape_vars): - """Set none if unallocated.""" - return If( - IfSection( - Is(data_ptr, NIL), - [ - *self._raise_memory_error_if_shape_is_nonzero(shape_vars), - Py_INCREF(Py_None), - AliasAssign(py_res, Py_None), - ], - ) - ) - - def _raise_memory_error_if_shape_is_nonzero(self, shape_vars): - """Raise the required error when memory error if shape is nonzero.""" - condition = None - for shape_var in shape_vars: - axis_has_extent = Ne(shape_var, convert_to_literal(0)) - condition = axis_has_extent if condition is None else Or(condition, axis_has_extent) - if condition is None: - return [] - return [ - If( - IfSection( - condition, - [ - PyErr_SetString( - PyMemoryError, - CStrStr(convert_to_literal("Unable to allocate copy-return output array.")), - ), - Return(self._error_exit_code), - ], - ) - ) - ] - - def _array_shape_validation(self, orig_var, shape_elems): - """Handle array shape validation for the current generation context.""" - checks = [] - for axis, (actual, expected) in enumerate(zip(shape_elems, orig_var.alloc_shape or (), strict=False)): - if expected is None: - continue - checks.append( - If( - IfSection( - Ne(actual, expected), - [ - PyErr_SetString( - PyTypeError, - CStrStr( - convert_to_literal( - f"Argument {orig_var.name} has incompatible shape at axis {axis}" - ) - ), - ), - Return(self._error_exit_code), - ], - ) - ) - ) - return checks - - def _array_itemsize_validation(self, orig_var, itemsize, _collect_arg): - """Validate fixed-width bytes dtype itemsize for character arrays.""" - expected = self._fixed_character_itemsize(orig_var) - if expected is None or itemsize is None: - return [] - return [ - If( - IfSection( - Ne(itemsize, convert_to_literal(expected, dtype=NumpyInt64Type())), - [ - PyErr_SetString( - PyTypeError, - CStrStr( - convert_to_literal( - f"Argument {orig_var.name} must have NumPy bytes dtype itemsize {expected}" - ) - ), - ), - Return(self._error_exit_code), - ], - ) - ) - ] - - def _array_access_validation(self, orig_var, decision, collect_arg): - """Handle array access validation for the current generation context.""" - return self._ARRAY_ACCESS_VALIDATION_DISPATCHER.dispatch_decision(self, orig_var, decision, collect_arg) - - def _readable_array_access_validation(self, orig_var, _decision, collect_arg): - """Validate a NumPy argument whose selected policy only reads storage.""" - pyarray = PointerCast(collect_arg, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - return [ - self._array_native_byte_order_validation( - pyarray, - f"Argument {orig_var.name} must use native byte order", - ), - self._array_flag_validation( - pyarray, - numpy_flag_aligned, - f"Argument {orig_var.name} must be aligned", - ), - ] - - def _writable_array_access_validation(self, orig_var, decision, collect_arg): - """Validate a NumPy argument whose selected policy mutates storage.""" - pyarray = PointerCast(collect_arg, Variable(NumpyArrayObjectType(), "_", memory_handling="alias")) - return [ - *self._readable_array_access_validation(orig_var, decision, collect_arg), - self._array_flag_validation( - pyarray, - numpy_flag_writeable, - f"Argument {orig_var.name} must be writeable", - ), - ] - - def _array_flag_validation(self, pyarray, flag, message): - """Handle array flag validation for the current generation context.""" - return If( - IfSection( - Not(PyArray_CHKFLAGS(ObjectAddress(pyarray), flag)), - [ - PyErr_SetString(PyTypeError, CStrStr(convert_to_literal(message))), - Return(self._error_exit_code), - ], - ) - ) - - def _array_native_byte_order_validation(self, pyarray, message): - """Handle array native byte order validation for the current generation context.""" - return If( - IfSection( - Not(PyArray_ISNOTSWAPPED(ObjectAddress(pyarray))), - [ - PyErr_SetString(PyTypeError, CStrStr(convert_to_literal(message))), - Return(self._error_exit_code), - ], - ) - ) - - def _bind_c_string_arg_parts(self, orig_var, *, writable): - """Handle bind c string arg parts for the current generation context.""" - class_type = NumpyNDArrayType.get_new(CharType(), 1, None, raw=True) - if not writable: - class_type = FinalType.get_new(class_type) - data_var = Variable( - class_type, - self.scope.get_expected_name(orig_var.name), - shape=(None,), - memory_handling="alias", - ) - size_var = Variable(NumpyInt64Type(), self.scope.get_new_name(f"{data_var.name}_size")) - arg_var = Variable( - BindCArrayType.get_new(1, False), - self.scope.get_new_name(orig_var.name), - shape=(convert_to_literal(2),), - ) - self.scope.insert_variable(data_var, orig_var.name) - self.scope.insert_variable(size_var) - data_element = IndexedElement(arg_var, convert_to_literal(0)) - size_element = IndexedElement(arg_var, convert_to_literal(1)) - self.scope.insert_symbolic_alias(data_element, ObjectAddress(data_var)) - self.scope.insert_symbolic_alias(size_element, size_var) - return data_var, size_var, arg_var - - def _string_utf8_source(self, orig_var, collect_arg): - """Handle string utf8 source for the current generation context.""" - source_var = Variable( - FinalType.get_new(CharType()), - self.scope.get_new_name(f"{orig_var.name}_utf8"), - memory_handling="alias", - ) - source_size = Variable(NumpyInt64Type(), self.scope.get_new_name(f"{orig_var.name}_utf8_size")) - self.scope.insert_variable(source_var) - self.scope.insert_variable(source_size) - body = [ - AliasAssign(source_var, PyUnicode_AsUTF8AndSize(collect_arg, ObjectAddress(source_size))), - If(IfSection(Is(source_var, NIL), [Return(self._error_exit_code)])), - If( - IfSection( - Ne(cast_to(c_strlen(source_var), NumpyInt64Type()), source_size), - [ - PyErr_SetString( - PyTypeError, - CStrStr(convert_to_literal(f"Argument {orig_var.name} cannot contain embedded NUL")), - ), - Return(self._error_exit_code), - ], - ) - ), - *self._fixed_string_length_validation(orig_var, source_size), - ] - return source_var, source_size, body - - def _fixed_string_length_validation(self, orig_var, source_size): - """Validate exact Python string length for a fixed-length character contract.""" - expected = self._fixed_character_itemsize(orig_var) - if expected is None: - return [] - return [ - If( - IfSection( - Ne(source_size, convert_to_literal(expected, dtype=NumpyInt64Type())), - [ - PyErr_SetString( - PyTypeError, - CStrStr( - convert_to_literal(f"Argument {orig_var.name} must encode to exactly {expected} bytes") - ), - ), - Return(self._error_exit_code), - ], - ) - ) - ] - - def _string_replacement_payload_size(self, orig_var, source_size): - """Handle string replacement payload size for the current generation context.""" - fixed_len = orig_var.alloc_shape[0] - return source_size if fixed_len is None else fixed_len - - @staticmethod - def _string_replacement_copy_body(data_var, source_var, source_size, payload_size, *, fixed_length): - """Handle string replacement copy body for the current generation context.""" - if not fixed_length: - return [c_memcpy(data_var, source_var, payload_size)] - return [ - c_memset(data_var, convert_to_literal(ord(" ")), payload_size), - If( - IfSection( - Lt(source_size, payload_size), - [c_memcpy(data_var, source_var, source_size)], - ), - IfSection(convert_to_literal(True), [c_memcpy(data_var, source_var, payload_size)]), - ), - ] diff --git a/x2py/codegen/bindings/cpython_api.py b/x2py/codegen/bindings/cpython_api.py deleted file mode 100644 index 8312098dd..000000000 --- a/x2py/codegen/bindings/cpython_api.py +++ /dev/null @@ -1,1777 +0,0 @@ -""" -Module representing objects (functions/variables etc) required for the interface -between Python code and C code (using Python/C Api and x2py_runtime/python_runtime.c). -This file contains classes but also many FunctionDef/Variable instances representing -objects defined in Python.h. -""" - -import re - -from ..bind_c import BindCPointer -from .c_concepts import CNativeInt, ObjectAddress -from ..models.core import ( - ClassDef, - Declare, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - FunctionOverloadSet, - Module, -) -from ..models.datatypes import ( - CharType, - CustomDataType, - FixedSizeType, - init_model_object, - PrimitiveBooleanType, - PrimitiveComplexType, - PrimitiveFloatingPointType, - PrimitiveIntegerType, - NumpyBoolType, - NumpyComplex64Type, - NumpyComplex128Type, - NumpyFloat32Type, - NumpyFloat64Type, - NumpyInt8Type, - NumpyInt16Type, - NumpyInt32Type, - NumpyInt64Type, - attach_model_child, - detach_model_child, - register_model_class, - StringType, - VoidType, - NIL, - convert_to_literal, -) -from ..models.core import Function -from ..models.core import Variable - -__all__ = ( - # --------- CLASSES ----------- - "PyAllowThreadsBegin", - "PyAllowThreadsEnd", - "PyArgKeywords", - "PyArg_ParseTupleNode", - "PyArgumentError", - # --------- CONSTANTS ---------- - "PyAttributeError", - "PyBuildValueNode", - "PyCallbackContextPop", - "PyCallbackContextPush", - "PyCallbackValidate", - "PyCapsule_Import", - "PyCapsule_New", - "PyClassDef", - # ----- C / PYTHON FUNCTIONS --- - "PyDict_New", - "PyDict_SetItem", - "PyErr_Occurred", - "PyErr_SetObject", - "PyErr_SetString", - "PyErr_WarnEx", - "PyFunctionDef", - "PyFunctionOverloadSet", - "PyGetSetDefElement", - "PyImport_ImportModule", - "PyList_Append", - "PyList_GetItem", - "PyList_New", - "PyList_SetItem", - "PyLong_AsLongLong", - "PyLong_AsVoidPtr", - "PyLong_Check", - "PyLong_FromLong", - "PyLong_FromLongLong", - "PyLong_FromVoidPtr", - "PyMemoryError", - "PyModInitFunc", - "PyModule", - "PyModule_AddObject", - "PyModule_Create", - "PyModule_GetDict", - "PyModule_SetPropertyType", - "PyNotImplementedError", - "PyObject_CallObject", - "PyObject_GetAttrString", - "PyObject_TypeCheck", - "PyRun_String", - "PyRuntimeError", - "PyRuntimeWarning", - "PySys_GetObject", - "PyTuple_GetItem", - "PyTuple_Pack", - "PyTypeError", - "PyType_Ready", - "PyUnicode_AsUTF8", - "PyUnicode_AsUTF8AndSize", - "PyUnicode_Check", - "PyUnicode_FromString", - "Py_DECREF", - "Py_False", - "Py_INCREF", - "Py_None", - "Py_True", - # --------- DATATYPES ----------- - "Py_ssize_t", - "PythonClassType", - "PythonObjectType", - "PythonTypeObjectType", - "WrapperCustomDataType", - "c_memcpy", - "c_memset", - "c_strlen", - "x2py_malloc", -) - - -# ------------------------------------------------------------------- -# Python DataTypes -# ------------------------------------------------------------------- -class PythonObjectType(FixedSizeType): - """ - Datatype representing a `PyObject`. - - Datatype representing a `PyObject` which is the - class used to hold Python objects in `Python.h`. - """ - - __slots__ = () - _name = "pyobject" - - -class PythonClassType(FixedSizeType): - """ - Datatype representing a subclass of `PyObject`. - - Datatype representing a subclass of `PyObject`. This is the - datatype of a class which is compatible with Python. - """ - - __slots__ = () - _name = "pyclasstype" - - -class PythonTypeObjectType(FixedSizeType): - """ - Datatype representing a `PyTypeObject`. - - Datatype representing a `PyTypeObject` which is the - class used to hold Python class objects in `Python.h`. - """ - - __slots__ = () - _name = "pytypeobject" - - -class PyCallbackValidate: - """Validate that a Python argument is callable before entering native code.""" - - __slots__ = ("callback", "error_exit", "python_object") - _attribute_nodes = ("callback", "error_exit", "python_object") - - def __init__(self, callback, python_object, error_exit): - """Initialize one ``PyCallbackValidate`` model instance.""" - self.callback = callback - self.python_object = python_object - self.error_exit = error_exit - init_model_object(self) - - -class PyCallbackContextPush: - """Install one call-scoped callback context immediately before a native call.""" - - __slots__ = ("callback", "python_object") - _attribute_nodes = ("callback", "python_object") - - def __init__(self, callback, python_object): - """Initialize one ``PyCallbackContextPush`` model instance.""" - self.callback = callback - self.python_object = python_object - init_model_object(self) - - -class PyCallbackContextPop: - """Restore the prior callback context after a native call returns.""" - - __slots__ = ("callback",) - _attribute_nodes = ("callback",) - - def __init__(self, callback): - """Initialize one ``PyCallbackContextPop`` model instance.""" - self.callback = callback - init_model_object(self) - - -class PyAllowThreadsBegin: - """Release the CPython GIL before entering a callback-free native call.""" - - __slots__ = () - - def __init__(self): - """Initialize one ``PyAllowThreadsBegin`` model instance.""" - init_model_object(self) - - -class PyAllowThreadsEnd: - """Reacquire the CPython GIL after a callback-free native call returns.""" - - __slots__ = () - - def __init__(self): - """Initialize one ``PyAllowThreadsEnd`` model instance.""" - init_model_object(self) - - -class WrapperCustomDataType(CustomDataType): - """ - Datatype representing a subclass of `PyObject`. - - Datatype representing a subclass of `PyObject`. This is the - datatype of a class which is compatible with Python. - """ - - __slots__ = () - _name = "pycustomclasstype" - - -class Py_ssize_t(FixedSizeType): - """ - Class representing Python's Py_ssize_t type. - - Class representing Python's Py_ssize_t type. - """ - - __slots__ = () - _name = "int" - _primitive_type = PrimitiveIntegerType() - - -# ------------------------------------------------------------------- -# Parsing and Building Classes -# ------------------------------------------------------------------- - - -# TODO: Is there an equivalent to static so this can be a static list of strings? -class PyArgKeywords: - """ - Represents the list containing the names of all arguments to a function. - This information allows the function to be called by keyword - - Parameters - ---------- - name : str - The name of the variable in which the list is stored - arg_names : list of str - A list of the names of the function arguments - """ - - __slots__ = ("_arg_names", "_name") - _attribute_nodes = () - - def __init__(self, name, arg_names): - """Initialize one ``PyArgKeywords`` model instance.""" - self._name = name - self._arg_names = arg_names - init_model_object(self) - - @property - def name(self): - """The name of the variable in which the list of - all arguments to the function is stored - """ - return self._name - - @property - def arg_names(self): - """The names of the arguments to the function which are - contained in the PyArgKeywords list - """ - return self._arg_names - - -# ------------------------------------------------------------------- -class PyArg_ParseTupleNode: - """ - Represents a call to the function `PyArg_ParseTupleNode`. - - Represents a call to the function `PyArg_ParseTupleNode` from `Python.h`. - This function collects the expected arguments from `self`, `args`, `kwargs` - and packs them into variables with datatype `PythonObjectType`. - - Parameters - ---------- - python_func_args : Variable - Args provided to the function in Python. - python_func_kwargs : Variable - Kwargs provided to the function in Python. - c_func_args : list of Variable - List of expected arguments. This helps determine the expected output types. - parse_args : list of Variable - List of arguments into which the result will be collected. - arg_names : list of str - A list of the names of the function arguments. - """ - - __slots__ = ("_arg_names", "_flags", "_parse_args", "_pyarg", "_pykwarg") - _attribute_nodes = ("_pyarg", "_pykwarg", "_parse_args", "_arg_names") - - def __init__(self, python_func_args, python_func_kwargs, c_func_args, parse_args, arg_names): - """Initialize one ``PyArg_ParseTupleNode`` model instance.""" - if not isinstance(python_func_args, Variable): - raise TypeError("Python func args should be a Variable") - if not isinstance(python_func_kwargs, Variable): - raise TypeError("Python func kwargs should be a Variable") - if not isinstance(parse_args, list) and any(not isinstance(c, Variable) for c in parse_args): - raise TypeError("Parse args should be a list of Variables") - if not isinstance(arg_names, PyArgKeywords): - raise TypeError("Parse args should be a list of Variables") - - self._flags = "" - has_default = False - has_keyword = False - for a in c_func_args: - if a.has_default and not has_default: - self._flags += "|" - has_default = True - if a.is_kwonly and not has_keyword: - self._flags += "$" - has_keyword = True - self._flags += "O" - - if any(a.is_vararg or a.is_kwarg for a in c_func_args): - raise NotImplementedError( - "Variadic arguments (*args, **kwargs) are not yet supported in the wrapper.", - ) - - self._pyarg = python_func_args - self._pykwarg = python_func_kwargs - self._parse_args = parse_args - self._arg_names = arg_names - init_model_object(self) - - @property - def pyarg(self): - """The variable containing all positional arguments - passed to the function - """ - return self._pyarg - - @property - def pykwarg(self): - """The variable containing all keyword arguments - passed to the function - """ - return self._pykwarg - - @property - def flags(self): - """ - The flags indicating the types of the objects. - - The flags indicating the types of the objects to be collected from - the Python arguments passed to the function. - """ - return self._flags - - @property - def args(self): - """The arguments into which the python args and kwargs - are collected - """ - return self._parse_args - - @property - def arg_names(self): - """The PyArgKeywords object which contains all the - names of the function's arguments - """ - return self._arg_names - - -# ------------------------------------------------------------------- -class PyBuildValueNode(Function): - """ - Represents a call to the function PyBuildValueNode. - - The function PyBuildValueNode can be found in Python.h. - It describes the creation of a new Python object based - on a format string. More details can be found in Python's - docs. - - Parameters - ---------- - result_args : list of Variable - List of arguments which the result will be built from. - """ - - __slots__ = ("_flags", "_result_args") - _attribute_nodes = ("_result_args",) - _shape = None - _class_type = PythonObjectType() - - def __init__(self, result_args=()): - """Initialize one ``PyBuildValueNode`` model instance.""" - self._flags = "" - self._result_args = result_args - for i in result_args: - if isinstance(i.dtype, WrapperCustomDataType): - self._flags += "O" - else: - self._flags += pytype_parse_registry[i.dtype] - super().__init__() - - @property - def flags(self): - """Handle flags on ``PyBuildValueNode``.""" - return self._flags - - @property - def args(self): - """Handle args on ``PyBuildValueNode``.""" - return self._result_args - - -# ------------------------------------------------------------------- -class PyModule_AddObject(Function): - """ - Represents a call to the PyModule_AddObject function. - - The PyModule_AddObject function can be found in Python.h. - It adds a PythonObject to a module. More information about - this function can be found in Python's documentation. - - Parameters - ---------- - mod_name : str - The name of the variable containing the module. - name : str - The name of the variable being added to the module. - variable : Variable - The variable containing the PythonObject. - """ - - __slots__ = ("_mod_name", "_name", "_var") - _attribute_nodes = ("_name", "_var") - _shape = None - _class_type = NumpyInt64Type() - - def __init__(self, mod_name, name, variable): - """Initialize one ``PyModule_AddObject`` model instance.""" - assert isinstance(name.dtype, CharType) - if not isinstance(variable, Variable) or variable.dtype not in ( - PythonObjectType(), - PythonClassType(), - ): - raise TypeError("Variable must be a PyObject Variable") - self._mod_name = mod_name - self._name = name - self._var = ObjectAddress(variable) - super().__init__() - - @property - def mod_name(self): - """The name of the variable containing the module""" - return self._mod_name - - @property - def name(self): - """The name of the variable being added to the module""" - return self._name - - @property - def variable(self): - """The variable containing the PythonObject""" - return self._var - - -# ------------------------------------------------------------------- -class PyModule_Create(Function): - """ - Represents a call to the PyModule_Create function. - - The PyModule_Create function can be found in Python.h. - It acts as a constructor for a module. More information about - this function can be found in Python's documentation. - See https://docs.python.org/3/c-api/module.html#c.PyModule_Create . - - Parameters - ---------- - module_def_name : str - The name of the structure which defined the module. - """ - - __slots__ = ("_module_def_name",) - _attribute_nodes = () - _shape = None - _class_type = PythonObjectType() - - def __init__(self, module_def_name): - """Initialize one ``PyModule_Create`` model instance.""" - self._module_def_name = module_def_name - super().__init__() - - @property - def module_def_name(self): - """ - Get the name of the structure which defined the module. - - Get the name of the structure which defined the module. - """ - return self._module_def_name - - -class PyModule_SetPropertyType(Function): - """Call a generated helper that installs module-variable attribute hooks.""" - - __slots__ = ("_module", "_setup_name") - _attribute_nodes = ("_module",) - _shape = None - _class_type = CNativeInt() - - def __init__(self, setup_name, module): - """Store the setup helper name and target module expression.""" - self._setup_name = setup_name - self._module = module - super().__init__(module) - - @property - def setup_name(self): - """Return the generated setup helper name.""" - return self._setup_name - - @property - def module(self): - """Return the module object receiving the custom type.""" - return self._module - - -# ------------------------------------------------------------------- -class PyCapsule_New(Function): - """ - Represents a call to the function PyCapsule_New. - - The function PyCapsule_New can be found in Python.h. It describes - the creation of a capsule. A capsule contains all information - from a module which should be exposed to other modules that import - this module. - See https://docs.python.org/3/extending/extending.html#using-capsules - for a tutorial involving capsules. - See https://docs.python.org/3/c-api/capsule.html#c.PyCapsule_New - for the API docstrings for this method. - - Parameters - ---------- - API_var : Variable - The variable which contains all elements of the API which should be exposed. - - module_name : str - The name of the module being exposed. - """ - - __slots__ = ("_API_var", "_capsule_name") - _attribute_nodes = ("_API_var",) - _shape = None - _class_type = PythonObjectType() - - def __init__(self, API_var, module_name): - """Initialize one ``PyCapsule_New`` model instance.""" - self._capsule_name = f"{module_name}._C_API" - self._API_var = API_var - super().__init__() - - @property - def capsule_name(self): - """ - Get the name of the capsule being created. - - Get the name of the capsule being created. - """ - return self._capsule_name - - @property - def API_var(self): - """ - Get the variable describing the API. - - Get the variable which contains all elements of the API which - should be exposed. - """ - return self._API_var - - -# ------------------------------------------------------------------- -class PyCapsule_Import(Function): - """ - Represents a call to the function PyCapsule_Import. - - The function PyCapsule_Import can be found in Python.h. It describes - the initialisation of a capsule by importing the information from - another module. A capsule contains all information from a module - which should be exposed to other modules that import this module. - See https://docs.python.org/3/extending/extending.html#using-capsules - for a tutorial involving capsules. - See https://docs.python.org/3/c-api/capsule.html#c.PyCapsule_Import - for the API docstrings for this method. - - Parameters - ---------- - module_name : str - The name of the module being retrieved. - """ - - __slots__ = ("_capsule_name",) - _attribute_nodes = () - _shape = None - _class_type = BindCPointer() - - def __init__(self, module_name): - """Initialize one ``PyCapsule_Import`` model instance.""" - self._capsule_name = f"{module_name}._C_API" - super().__init__() - - @property - def capsule_name(self): - """ - Get the name of the capsule being retrieved. - - Get the name of the capsule being retrieved. - """ - return self._capsule_name - - -# ------------------------------------------------------------------- -class PyModule(Module): - """ - Class to hold a module which is accessible from Python. - - Class to hold a module which is accessible from Python. This class - adds external functions and external declarations to the basic - Module. However its main utility is in order to differentiate - itself such that a different `_print` function can be implemented - to handle it. - - Parameters - ---------- - name : str - Name of the module. - - *args : tuple - See Module. - - external_funcs : iterable of FunctionDef - A list of external functions. - - declarations : iterable - Any declarations of (external) variables which should be made in the module. - - init_func : FunctionDef, optional - The function which is executed when a module is initialised. - See: https://docs.python.org/3/c-api/module.html#multi-phase-initialization . - - import_func : FunctionDef, optional - The function which allows types from this module to be imported in other - modules. - See: https://docs.python.org/3/extending/extending.html . - - module_def_name : str - The name of the structure which defined the module. - - **kwargs : dict - See Module. - - See Also - -------- - Module : The super class from which the class inherits. - """ - - __slots__ = ( - "_declarations", - "_external_funcs", - "_import_func", - "_module_def_name", - "_module_properties", - "_namespace_module_defs", - ) - _attribute_nodes = (*Module._attribute_nodes, "_external_funcs", "_declarations", "_import_func") - - def __init__( - self, - name, - *args, - external_funcs=(), - declarations=(), - init_func=None, - import_func, - module_def_name, - module_properties=None, - namespace_module_defs=None, - **kwargs, - ): - """Initialize one ``PyModule`` model instance.""" - self._external_funcs = external_funcs - self._declarations = declarations - self._module_def_name = module_def_name - self._module_properties = dict(module_properties or {}) - self._namespace_module_defs = dict(namespace_module_defs or {}) - self._import_func = import_func - super().__init__(name, *args, init_func=init_func, **kwargs) - - @property - def external_funcs(self): - """ - A list of external functions. - - The external functions which should be declared at the start of the module. - This is useful for declaring the existence of Fortran functions whose - definition and declaration is inaccessible from C. - """ - return self._external_funcs - - @property - def namespace_module_defs(self): - """Return child namespace paths and their generated module definitions.""" - return self._namespace_module_defs - - @property - def module_properties(self): - """Return generated module-variable property descriptors by namespace.""" - return self._module_properties - - @external_funcs.setter - def external_funcs(self, funcs): - """Handle external funcs on ``PyModule``.""" - for f in self._external_funcs: - detach_model_child(self, f) - self._external_funcs = funcs - for f in funcs: - attach_model_child(self, f) - - @property - def declarations(self): - """ - All declarations that need printing in the module. - - All declarations that need printing in the module. This usually includes - any variables coming from a non-C language for which compatibility with C - exists. - """ - return self._declarations - - @declarations.setter - def declarations(self, decs): - """Handle declarations on ``PyModule``.""" - for d in self._declarations: - detach_model_child(self, d) - self._declarations = decs - for d in decs: - attach_model_child(self, d) - - @property - def import_func(self): - """ - The function which allows types from this module to be imported in other modules. - - The function which allows types from this module to be imported in other modules. - See https://docs.python.org/3/extending/extending.html to understand how this - is done. - """ - return self._import_func - - @property - def module_def_name(self): - """ - The name of the PyModuleDef object describing the module. - - The name of the PyModuleDef object describing the module and - its contents for Python. - """ - return self._module_def_name - - -# ------------------------------------------------------------------- -class PyFunctionDef(FunctionDef): - """ - Class to hold a FunctionDef which is accessible from Python. - - Contains the Python-compatible version of the function which is - used for the wrapper. - As compared to a normal FunctionDef, this version contains - arguments for the shape of arrays. It should be generated by - calling `codegen.wrapper.CToPythonWrapper.wrap`. - - Parameters - ---------- - *args : list - See FunctionDef. - - original_function : FunctionDef - The function from which the Python-compatible version was created. - - **kwargs : dict - See FunctionDef. - - See Also - -------- - x2py.ast.core.FunctionDef - The class from which BindCFunctionDef inherits which contains all - details about the args and kwargs. - """ - - __slots__ = ("_original_function",) - _attribute_nodes = (*FunctionDef._attribute_nodes, "_original_function") - - def __init__(self, *args, original_function, **kwargs): - """Initialize one ``PyFunctionDef`` model instance.""" - self._original_function = original_function - super().__init__(*args, **kwargs, is_static=True) - - @property - def original_function(self): - """ - The function which is wrapped by this PyFunctionDef. - - The original function which would be printed in pure C which is not - compatible with Python. - """ - return self._original_function - - -# ------------------------------------------------------------------- -class PyFunctionOverloadSet(FunctionOverloadSet): - """ - Class to hold an FunctionOverloadSet which is accessible from Python. - - A class which holds the Python-compatible FunctionOverloadSet. It contains functions for - determining the type of the arguments passed to the FunctionOverloadSet and the functions - called through the interface. - - Parameters - ---------- - name : str - The name of the interface. See FunctionOverloadSet. - - functions : iterable of FunctionDef - The functions of the interface. See FunctionOverloadSet. - - dispatcher_func : FunctionDef - The function which Python will call to access the interface. - - type_check_func : FunctionDef - The helper function which will determine the types of the arguments passed. - - original_overload_set : FunctionOverloadSet - The interface being wrapped. - - **kwargs : dict - See FunctionOverloadSet. - - See Also - -------- - FunctionOverloadSet : The super class. - """ - - __slots__ = ("_dispatcher_func", "_original_overload_set", "_type_check_func") - _attribute_nodes = ( - *FunctionOverloadSet._attribute_nodes, - "_dispatcher_func", - "_type_check_func", - "_original_overload_set", - ) - - def __init__( - self, - name, - functions, - dispatcher_func, - type_check_func, - original_overload_set, - **kwargs, - ): - """Initialize one ``PyFunctionOverloadSet`` model instance.""" - self._dispatcher_func = dispatcher_func - self._type_check_func = type_check_func - self._original_overload_set = original_overload_set - for f in functions: - if not isinstance(f, PyFunctionDef): - raise TypeError("PyFunctionOverloadSet functions should be instances of the class PyFunctionDef.") - super().__init__(name, functions, False, **kwargs) - - @property - def dispatcher_func(self): - """ - The function which is exposed to Python. - - The function which receives the Python arguments `self`, `args`, and `kwargs` and calls - the appropriate function. - """ - return self._dispatcher_func - - @property - def type_check_func(self): - """ - The function which determines the types which were passed to the FunctionOverloadSet. - - The function which takes the arguments passed to the function and returns an integer - indicating which function was called. - """ - return self._type_check_func - - @property - def original_function(self): - """ - The FunctionOverloadSet which is wrapped by this PyFunctionOverloadSet. - - The original interface which would be printed in C. - """ - return self._original_overload_set - - -# ------------------------------------------------------------------- -class PyClassDef(ClassDef): - """ - Class to hold a class definition which is accessible from Python. - - Class to hold a class definition which is accessible from Python. - - Parameters - ---------- - original_class : ClassDef - The original class being wrapped. - - struct_name : str - The name of the structure which will hold the Python-compatible - class definition. - - type_name : str - The name of the instance of the Python-compatible class definition - structure. This object is necessary to add the class to the module. - - scope : Scope - The scope for the class contents. - - **kwargs : dict - See ClassDef. - - See Also - -------- - ClassDef - The class from which PyClassDef inherits. This is also the object being - wrapped. - """ - - __slots__ = ( - "_magic_methods", - "_new_func", - "_original_class", - "_properties", - "_struct_name", - "_type_name", - "_type_object", - ) - _attribute_nodes = (*ClassDef._attribute_nodes, "_magic_methods") - - def __init__(self, original_class, struct_name, type_name, scope, **kwargs): - """Initialize one ``PyClassDef`` model instance.""" - assert isinstance(original_class, ClassDef) - self._original_class = original_class - self._struct_name = struct_name - self._type_name = type_name - self._type_object = Variable(PythonClassType(), type_name) - self._new_func = None - self._properties = () - self._magic_methods = () - variables = [ - Variable(VoidType(), scope.get_new_name("instance"), memory_handling="alias"), - Variable( - PythonObjectType(), - scope.get_new_name("referenced_objects"), - memory_handling="alias", - ), - Variable(NumpyBoolType(), scope.get_new_name("is_alias")), - ] - scope.insert_variable(variables[0]) - scope.insert_variable(variables[1]) - scope.insert_variable(variables[2]) - super().__init__(original_class.name, variables, scope=scope, **kwargs) - - @property - def struct_name(self): - """ - The name of the structure which will hold the Python-compatible class definition. - - The name of the structure which will hold the Python-compatible class definition. - """ - return self._struct_name - - @property - def type_name(self): - """ - The name of the Python-compatible class definition instance. - - The name of the instance of the Python-compatible class definition - structure. This object is necessary to add the class to the module. - """ - return self._type_name - - @property - def type_object(self): - """ - The Python-compatible class definition instance. - - The Variable describing the instance of the Python-compatible class definition - structure. This object is necessary to add the class to the module. - """ - return self._type_object - - @property - def original_class(self): - """ - The class which is wrapped by this PyClassDef. - - The original class which would be printed in pure C which is not - compatible with Python. - """ - return self._original_class - - def add_alloc_method(self, f): - """ - Add the wrapper for `__new__` to the class definition. - - Add the wrapper for `__new__` which allocates the memory for the class instance. - - Parameters - ---------- - f : PyFunctionDef - The wrapper for the `__new__` function. - """ - self._new_func = f - - @property - def new_func(self): - """ - Get the wrapper for `__new__`. - - Get the wrapper for `__new__` which allocates the memory for the class instance. - """ - return self._new_func - - def add_property(self, p): - """ - Add a class property which has been wrapped. - - Add a class property which has been wrapped. - - Parameters - ---------- - p : model object - The new wrapped property which is added to the class. - """ - attach_model_child(self, p) - self._properties += (p,) - - @property - def properties(self): - """ - Get all wrapped class properties. - - Get all wrapped class properties. - """ - return self._properties - - def add_new_magic_method(self, method): - """ - Add a new magic method to the current class. - - Add a new magic method to the current ClassDef. - - Parameters - ---------- - method : FunctionDef - The Method that will be added. - """ - - if not isinstance(method, PyFunctionDef | PyFunctionOverloadSet): - raise TypeError("Method must be PyFunctionDef or PyFunctionOverloadSet") - attach_model_child(self, method) - self._magic_methods += (method,) - - @property - def magic_methods(self): - """ - Get the magic methods describing methods. - - Get the magic methods describing methods such as __add__. - """ - return self._magic_methods - - -# ------------------------------------------------------------------- - - -class PyGetSetDefElement: - """ - A class representing a PyGetSetDef object. - - A class representing an element of the list of PyGetSetDef objects - which are used to add attributes/properties to classes. - See https://docs.python.org/3/c-api/structures.html#c.PyGetSetDef . - - Parameters - ---------- - python_name : str - The name of the attribute/property in the original Python code. - getter : FunctionDef - The function which collects the value of the class attribute. - setter : FunctionDef - The function which modifies the value of the class attribute. - docstring : Literal - The docstring of the property. - """ - - _attribute_nodes = ("_getter", "_setter", "_docstring") - __slots__ = ("_docstring", "_getter", "_python_name", "_setter") - - def __init__(self, python_name, getter, setter, docstring): - """Initialize one ``PyGetSetDefElement`` model instance.""" - assert isinstance(getter, PyFunctionDef) - assert isinstance(setter, PyFunctionDef) or setter is None - self._python_name = python_name - self._getter = getter - self._setter = setter - self._docstring = docstring - init_model_object(self) - - @property - def python_name(self): - """ - The name of the attribute/property in the original Python code. - - The name of the attribute/property in the original Python code. - """ - return self._python_name - - @property - def getter(self): - """ - The BindCFunctionDef describing the getter function. - - The BindCFunctionDef describing the function which allows the user to collect - the value of the property. - """ - return self._getter - - @property - def setter(self): - """ - The BindCFunctionDef describing the setter function. - - The BindCFunctionDef describing the function which allows the user to modify - the value of the property. - """ - return self._setter - - @property - def docstring(self): - """ - The docstring of the property being wrapped. - - The docstring of the property being wrapped. - """ - return self._docstring - - -# ------------------------------------------------------------------- -class PyModInitFunc(FunctionDef): - """ - A class representing the PyModInitFunc function def. - - A class representing the PyModInitFunc function def. This function returns the - macro PyModInitFunc, takes no arguments and initialises a module. - - Parameters - ---------- - name : str - The name of the function. - - body : list[model object] - The code executed in the function. - - static_vars : list[Variable] - A list of variables which should be declared as static objects. - - scope : Scope - The scope of the function. - """ - - __slots__ = ("_static_vars",) - - def __init__(self, name, body, static_vars, scope): - """Initialize one ``PyModInitFunc`` model instance.""" - self._static_vars = static_vars - super().__init__(name, (), body, scope=scope) - - @property - def declarations(self): - """ - Returns the declarations of the variables. - - Returns the declarations of the variables. - """ - return [ - Declare( - v, - static=(v in self._static_vars), - value=(NIL if isinstance(v.class_type, VoidType | BindCPointer) else None), - ) - for v in self.scope.variables.values() - ] - - -class PyTuple_Pack(Function): - """ - A class representing a call to Python's PyTuple_Pack function. - - A class representing a call to Python's PyTuple_Pack function. A class - is used instead of a FunctionDef as the number of arguments is variable. - A PyTuple_Pack is described here: - https://docs.python.org/3/c-api/tuple.html#c.PyTuple_Pack - - Parameters - ---------- - *args : model object - The arguments that should be packed into the tuple. - """ - - __slots__ = () - _class_type = PythonObjectType() - _shape = None - - -class PyArgumentError: - """ - Class to display errors related to arguments. - - Class to display errors related to arguments. This class helps - format the arguments to display the type of the received argument. - - Parameters - ---------- - error_type : Variable - A Variable containing the error type to be raised. E.g. PyTypeError. - error_msg : str - The message to be displayed containing f-string style type indicators. - **kwargs : dict[str, Variable] - The arguments whose types will be printed. - """ - - __slots__ = ("_args", "_error_msg", "_error_type") - _attribute_nodes = ("_args",) - - def __init__(self, error_type, error_msg: str, **kwargs): - """Initialize one ``PyArgumentError`` model instance.""" - assert isinstance(error_type, Variable) - assert isinstance(error_msg, str) - args = [] - # Find all expressions of the style '{type(var_name)}' in the error message - type_indicators = re.findall(r"{type\([a-zA-Z0-9_]+\)}", error_msg) - # Save the error message, replacing type indicators with the format string - self._error_msg = re.sub(r"{type\([a-zA-Z0-9_]+\)}", "%V", error_msg) - # Find the relevant arguments for each type indicator - for t in type_indicators: - var_name = t.removeprefix("{type(").removesuffix(")}") - args.append(ObjectAddress(kwargs[var_name])) - - self._args = tuple(args) - self._error_type = error_type - init_model_object(self) - - @property - def error_type(self): - """ - The error type that should be raised. - - The error type that should be raised. - """ - return self._error_type - - @property - def error_msg(self): - """ - The error message that should be formatted. - - The error message that should be formatted. - """ - return self._error_msg - - @property - def args(self): - """ - The arguments whose types are printed in the error message. - - The arguments whose types are printed in the error message. - These arguments are displayed in the order they appear in - the error message. - """ - return self._args - - -# ------------------------------------------------------------------- -# Python.h Constants -# ------------------------------------------------------------------- - -# Python.h object representing Booleans True and False -Py_True = Variable(PythonObjectType(), "Py_True", memory_handling="alias") -Py_False = Variable(PythonObjectType(), "Py_False", memory_handling="alias") - -# Python.h object representing None -Py_None = Variable(PythonObjectType(), "Py_None", memory_handling="alias") - -# https://docs.python.org/3/c-api/refcounting.html#c.Py_INCREF -Py_INCREF = FunctionDef( - name="Py_INCREF", - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], -) - -# https://docs.python.org/3/c-api/refcounting.html#c.Py_DECREF -Py_DECREF = FunctionDef( - name="Py_DECREF", - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], -) - -# https://docs.python.org/3/c-api/type.html#c.PyType_Ready -PyType_Ready = FunctionDef( - name="PyType_Ready", - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyInt64Type(), "_")), -) - -# https://docs.python.org/3/c-api/sys.html#PySys_GetObject -PySys_GetObject = FunctionDef( - name="PySys_GetObject", - body=[], - arguments=[FunctionDefArgument(Variable(StringType(), name="_"))], - results=FunctionDefResult(Variable(PythonObjectType(), name="o", memory_handling="alias")), -) - -# https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_FromString -PyUnicode_FromString = FunctionDef( - name="PyUnicode_FromString", - body=[], - arguments=[FunctionDefArgument(Variable(StringType(), name="_"))], - results=FunctionDefResult(Variable(PythonObjectType(), name="o", memory_handling="alias")), -) - -# ------------------------------------------------------------------- - -# using the documentation of PyArg_ParseTuple() and Py_BuildValue https://docs.python.org/3/c-api/arg.html -pytype_parse_registry = { - NumpyFloat64Type(): "d", - NumpyComplex128Type(): "O", - NumpyBoolType(): "p", - StringType(): "s", - CharType(): "s", - PythonObjectType(): "O", -} - -# ------------------------------------------------------------------- -# python_runtime.h functions -# ------------------------------------------------------------------- - -# Functions definitions are defined in x2py/stdlib/x2py_runtime/python_runtime.c -py_to_c_registry = { - (PrimitiveBooleanType(), -1): "PyBool_to_Bool", - (PrimitiveIntegerType(), 1): "PyInt8_to_Int8", - (PrimitiveIntegerType(), 2): "PyInt16_to_Int16", - (PrimitiveIntegerType(), 4): "PyInt32_to_Int32", - (PrimitiveIntegerType(), 8): "PyInt64_to_Int64", - (PrimitiveFloatingPointType(), 4): "PyFloat_to_Float", - (PrimitiveFloatingPointType(), 8): "PyDouble_to_Double", - (PrimitiveComplexType(), 4): "PyComplex_to_Complex64", - (PrimitiveComplexType(), 8): "PyComplex_to_Complex128", -} - - -def C_to_Python(c_object): - """ - Create a FunctionDef responsible for casting scalar C results to Python. - - Creates a FunctionDef node which contains all the code necessary - for casting a C object, whose characteristics match that of the object - passed as an argument, to a PythonObject which can be used in Python code. - - Parameters - ---------- - c_object : Variable - The variable needed for the generation of the cast_function. - - Returns - ------- - FunctionDef - The function which casts the C object to Python. - """ - assert c_object.rank == 0 - try: - cast_function = c_to_py_registry[c_object.dtype] - except KeyError: - raise TypeError(f"No C-to-Python cast registered for {c_object.dtype}") from None - memory_handling = "alias" - - return FunctionDef( - name=cast_function, - body=[], - arguments=[ - FunctionDefArgument( - c_object.clone( - "v", - is_argument=True, - memory_handling=memory_handling, - new_class=Variable, - ) - ) - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="o", memory_handling="alias")), - ) - - -# Functions definitions are defined in x2py/stdlib/x2py_runtime/python_runtime.c -c_to_py_registry = { - NumpyBoolType(): "Bool_to_PyBool", - NumpyInt8Type(): "Int8_to_NumpyLong", - NumpyInt16Type(): "Int16_to_NumpyLong", - NumpyInt32Type(): "Int32_to_PyLong", - NumpyInt64Type(): "Int" + str(NumpyInt64Type().precision * 8) + "_to_PyLong", - NumpyFloat32Type(): "Float_to_NumpyDouble", - NumpyFloat64Type(): "Double_to_PyDouble", - NumpyComplex64Type(): "Complex64_to_NumpyComplex", - NumpyComplex128Type(): "Complex128_to_PyComplex", -} - - -# ------------------------------------------------------------------- -# errors and check functions -# ------------------------------------------------------------------- - -# https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Occurred -PyErr_Occurred = FunctionDef( - name="PyErr_Occurred", - arguments=[], - results=FunctionDefResult(Variable(PythonObjectType(), name="r", memory_handling="alias")), - body=[], -) - -PyErr_SetString = FunctionDef( - name="PyErr_SetString", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="o")), - FunctionDefArgument(Variable(CharType(), name="s", memory_handling="alias")), - ], -) - -PyErr_SetObject = FunctionDef( - name="PyErr_SetObject", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="o")), - FunctionDefArgument(Variable(PythonObjectType(), name="value", memory_handling="alias")), - ], -) - -PyErr_WarnEx = FunctionDef( - name="PyErr_WarnEx", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="category")), - FunctionDefArgument(Variable(CharType(), name="message", memory_handling="alias")), - FunctionDefArgument(Variable(Py_ssize_t(), name="stack_level")), - ], - results=FunctionDefResult(Variable(CNativeInt(), name="status")), -) - -PyImport_ImportModule = FunctionDef( - name="PyImport_ImportModule", - body=[], - arguments=[FunctionDefArgument(Variable(CharType(), name="name", memory_handling="alias"))], - results=FunctionDefResult(Variable(PythonObjectType(), name="module", memory_handling="alias")), -) - -PyNotImplementedError = Variable(PythonObjectType(), name="PyExc_NotImplementedError") -PyMemoryError = Variable(PythonObjectType(), name="PyExc_MemoryError") -PyTypeError = Variable(PythonObjectType(), name="PyExc_TypeError") -PyAttributeError = Variable(PythonObjectType(), name="PyExc_AttributeError") -PyRuntimeError = Variable(PythonObjectType(), name="PyExc_RuntimeError") -PyRuntimeWarning = Variable(PythonObjectType(), name="PyExc_RuntimeWarning") - -PyObject_TypeCheck = FunctionDef( - name="PyObject_TypeCheck", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "o", memory_handling="alias")), - FunctionDefArgument(Variable(PythonClassType(), "c_type", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(NumpyBoolType(), "r")), - body=[], -) - -# ------------------------------------------------------------------- -# List functions -# ------------------------------------------------------------------- - -# https://docs.python.org/3/c-api/list.html#c.PyList_New -PyList_New = FunctionDef( - name="PyList_New", - arguments=[FunctionDefArgument(Variable(NumpyInt64Type(), "size"), value=convert_to_literal(0))], - results=FunctionDefResult(Variable(PythonObjectType(), "r", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/list.html#c.PyList_Append -PyList_Append = FunctionDef( - name="PyList_Append", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "list", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), "item", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(CNativeInt(), "i")), - body=[], -) - -# https://docs.python.org/3/c-api/list.html#c.PyList_GetItem -PyList_GetItem = FunctionDef( - name="PyList_GetItem", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "list", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyInt64Type(), "i")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), "item", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/list.html#c.PyList_SetItem -PyList_SetItem = FunctionDef( - name="PyList_SetItem", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="l", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyInt64Type(), name="i")), - FunctionDefArgument(Variable(PythonObjectType(), name="new_item", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(CNativeInt(), "i")), -) - - -# ------------------------------------------------------------------- -# Tuple functions -# ------------------------------------------------------------------- - -PyTuple_GetItem = FunctionDef( - name="PyTuple_GetItem", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "tuple", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyInt64Type(), "i")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), "item", memory_handling="alias")), - body=[], -) - - -# ------------------------------------------------------------------- -# Dict functions -# ------------------------------------------------------------------- - - -# https://docs.python.org/3/c-api/dict.html#c.PyDict_New -PyDict_New = FunctionDef( - name="PyDict_New", - arguments=[], - results=FunctionDefResult(Variable(PythonObjectType(), "dict", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/dict.html#c.PyDict_SetItem -PyDict_SetItem = FunctionDef( - name="PyDict_SetItem", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "dict", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), "key", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), "val", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(NumpyInt64Type(), "i")), - body=[], -) - - -PyModule_GetDict = FunctionDef( - name="PyModule_GetDict", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "module", memory_handling="alias"))], - results=FunctionDefResult(Variable(PythonObjectType(), "dict", memory_handling="alias")), - body=[], -) - - -PyRun_String = FunctionDef( - name="PyRun_String", - body=[], - arguments=[ - FunctionDefArgument(Variable(CharType(), name="code", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), name="start")), - FunctionDefArgument(Variable(PythonObjectType(), name="globals", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), name="locals", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="result", memory_handling="alias")), -) - - -PyObject_GetAttrString = FunctionDef( - name="PyObject_GetAttrString", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="object", memory_handling="alias")), - FunctionDefArgument(Variable(CharType(), name="attr", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="result", memory_handling="alias")), -) - - -PyObject_CallObject = FunctionDef( - name="PyObject_CallObject", - body=[], - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), name="callable", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), name="args", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="result", memory_handling="alias")), -) - - -# ------------------------------------------------------------------- -# String functions -# ------------------------------------------------------------------- -# https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUTF8 -PyUnicode_AsUTF8 = FunctionDef( - name="PyUnicode_AsUTF8", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "unicode", memory_handling="alias"))], - results=FunctionDefResult(Variable(CharType(), "str", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUTF8AndSize -PyUnicode_AsUTF8AndSize = FunctionDef( - name="PyUnicode_AsUTF8AndSize", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "unicode", memory_handling="alias")), - FunctionDefArgument(Variable(Py_ssize_t(), "size", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(CharType(), "str", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check -PyUnicode_Check = FunctionDef( - name="PyUnicode_Check", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "str", memory_handling="alias"))], - results=FunctionDefResult(Variable(CNativeInt(), "out")), - body=[], -) - -# https://docs.python.org/3/c-api/long.html#c.PyLong_AsVoidPtr -PyLong_AsVoidPtr = FunctionDef( - name="PyLong_AsVoidPtr", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "o", memory_handling="alias"))], - results=FunctionDefResult(Variable(BindCPointer(), "ptr", memory_handling="alias")), - body=[], -) - -PyLong_AsLongLong = FunctionDef( - name="PyLong_AsLongLong", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyInt64Type(), "value")), - body=[], -) - -PyLong_FromVoidPtr = FunctionDef( - name="PyLong_FromVoidPtr", - arguments=[FunctionDefArgument(Variable(BindCPointer(), "p", memory_handling="alias"))], - results=FunctionDefResult(Variable(PythonObjectType(), "o", memory_handling="alias")), - body=[], -) - -PyLong_FromLong = FunctionDef( - name="PyLong_FromLong", - arguments=[FunctionDefArgument(Variable(CNativeInt(), "value"))], - results=FunctionDefResult(Variable(PythonObjectType(), "o", memory_handling="alias")), - body=[], -) - -PyLong_FromLongLong = FunctionDef( - name="PyLong_FromLongLong", - arguments=[FunctionDefArgument(Variable(NumpyInt64Type(), "value"))], - results=FunctionDefResult(Variable(PythonObjectType(), "o", memory_handling="alias")), - body=[], -) - -# https://docs.python.org/3/c-api/long.html#c.PyLong_Check -PyLong_Check = FunctionDef( - name="PyLong_Check", - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "o", memory_handling="alias"))], - results=FunctionDefResult(Variable(CNativeInt(), "out")), - body=[], -) - -c_memcpy = FunctionDef( - name="memcpy", - arguments=[ - FunctionDefArgument(Variable(VoidType(), "dest", memory_handling="alias")), - FunctionDefArgument(Variable(VoidType(), "src", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyInt64Type(), "n")), - ], - results=FunctionDefResult(NIL), - body=[], -) - -c_memset = FunctionDef( - name="memset", - arguments=[ - FunctionDefArgument(Variable(VoidType(), "s", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), "c")), - FunctionDefArgument(Variable(NumpyInt64Type(), "n")), - ], - results=FunctionDefResult(NIL), - body=[], -) - -c_strlen = FunctionDef( - name="strlen", - arguments=[FunctionDefArgument(Variable(CharType(), "s", memory_handling="alias"))], - results=FunctionDefResult(Variable(CNativeInt(), "n")), - body=[], -) - -x2py_malloc = FunctionDef( - name="x2py_malloc", - arguments=[FunctionDefArgument(Variable(NumpyInt64Type(), "size"))], - results=FunctionDefResult(Variable(VoidType(), "ptr", memory_handling="alias")), - body=[], -) - -# Functions definitions are defined in x2py/stdlib/x2py_runtime/python_runtime.c -check_type_registry = { - NumpyBoolType(): "PyIs_Bool", - NumpyInt64Type(): "PyIs_NativeInt", - NumpyFloat64Type(): "PyIs_NativeFloat", - NumpyComplex128Type(): "PyIs_NativeComplex", -} - - -for _model_cls in ( - PyArgKeywords, - PyArg_ParseTupleNode, - PyGetSetDefElement, - PyArgumentError, -): - register_model_class(_model_cls) - -del _model_cls diff --git a/x2py/codegen/bindings/numpy_cpython_api.py b/x2py/codegen/bindings/numpy_cpython_api.py deleted file mode 100644 index eb3cc5d52..000000000 --- a/x2py/codegen/bindings/numpy_cpython_api.py +++ /dev/null @@ -1,385 +0,0 @@ -""" -Handling the transitions between Python code and C code using (Numpy/C Api). -""" - -import numpy as np - -from .c_concepts import CNativeInt -from ..models.core import FunctionDef, FunctionDefArgument, FunctionDefResult -from .cpython_api import ( - PythonObjectType, - c_to_py_registry, - check_type_registry, - pytype_parse_registry, -) -from ..models.datatypes import CharType, FixedSizeType, GenericType, NumpyBoolType, VoidType -from ..models.datatypes import ( - NumpyComplex64Type, - NumpyComplex128Type, - NumpyComplex256Type, - NumpyFloat32Type, - NumpyFloat64Type, - NumpyFloat128Type, - NumpyInt8Type, - NumpyInt16Type, - NumpyInt32Type, - NumpyInt64Type, - NumpyNDArrayType, -) -from ..models.core import Variable - -__all__ = ( - # --------- DATATYPES --------- - "NumpyArrayObjectType", - # -------HELPERS ------ - "PyArray_SetBaseObject", - # -------OTHERS-------- - "get_numpy_max_acceptable_version_file", - # ------- CAST FUNCTIONS ------ - "pyarray_to_ndarray", -) - - -class NumpyArrayObjectType(FixedSizeType): - """ - Datatype representing a `PyArrayObject`. - - Datatype representing a `PyArrayObject` which is the - class used to hold NumPy array objects in Python. - """ - - __slots__ = () - _name = "PyArrayObject" - - -# ------------------------------------------------------------------- -# Numpy functions -# ------------------------------------------------------------------- - - -def get_numpy_max_acceptable_version_file(): - """ - Get the macro specifying the most recent acceptable NumPy version. - - Get the macro specifying the most recent acceptable NumPy version. - If NumPy is more recent than this then deprecation warnings are shown. - - The most recent acceptable NumPy version is 1.19. If the current version is older - than this then the last acceptable NumPy version is the current version. - - Returns - ------- - str - A string containing the code which defines the macro. - """ - numpy_max_acceptable_version = [1, 19] - numpy_current_version = [int(v) for v in np.version.version.split(".")[:2]] - numpy_api_acceptable_version = min(numpy_max_acceptable_version, numpy_current_version) - major, minor = numpy_api_acceptable_version - numpy_api_macro = f"# define NPY_NO_DEPRECATED_API NPY_{major}_{minor}_API_VERSION\n" - version_file = "#ifndef NPY_NO_DEPRECATED_API\n" + numpy_api_macro + "#endif\n" - if numpy_current_version[0] >= 2: - version_file += "#ifndef NPY_TARGET_VERSION\n# define NPY_TARGET_VERSION NPY_2_0_API_VERSION\n#endif\n" - return version_file - - -PyArray_Check = FunctionDef( - name="PyArray_Check", - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyBoolType(), name="b")), -) - -PyArray_DATA = FunctionDef( - name="PyArray_DATA", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(VoidType(), name="b", memory_handling="alias")), -) - -PyArray_NDIM = FunctionDef( - name="PyArray_NDIM", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(CNativeInt(), name="nd")), -) - -PyArray_TYPE = FunctionDef( - name="PyArray_TYPE", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(CNativeInt(), name="typenum")), -) - -PyArray_BASE = FunctionDef( - name="PyArray_BASE", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias")), -) - -PyArray_SHAPE = FunctionDef( - name="PyArray_SHAPE", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult( - Variable(NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), name="s", memory_handling="alias") - ), -) - -PyArray_STRIDES = FunctionDef( - name="PyArray_STRIDES", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult( - Variable(NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), name="s", memory_handling="alias") - ), -) - -PyArray_ITEMSIZE = FunctionDef( - name="PyArray_ITEMSIZE", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyInt32Type(), name="s")), -) - -# NumPy array to c ndarray : function definition in x2py/stdlib/x2py_runtime/python_runtime.c -pyarray_to_ndarray = FunctionDef( - name="pyarray_to_ndarray", - body=[], - arguments=[FunctionDefArgument(Variable(PythonObjectType(), "a", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyNDArrayType.get_new(GenericType(), 1, None), "array")), -) - -numpy_to_stc_strides = FunctionDef( - name="numpy_to_stc_strides", - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="o", memory_handling="alias"))], - body=[], - results=FunctionDefResult(Variable(NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), "strides")), -) - -# NumPy array check elements : function definition in x2py/stdlib/x2py_runtime/python_runtime.c -pyarray_check = FunctionDef( - name="pyarray_check", - arguments=[ - FunctionDefArgument(Variable(CharType(), "name", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), "a", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), "dtype")), - FunctionDefArgument(Variable(CNativeInt(), "rank")), - FunctionDefArgument(Variable(CNativeInt(), "flag")), - FunctionDefArgument(Variable(NumpyBoolType(), "allow_empty")), - ], - body=[], - results=FunctionDefResult(Variable(NumpyBoolType(), "b")), -) - -is_numpy_array = FunctionDef( - name="is_numpy_array", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "a", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), "dtype")), - FunctionDefArgument(Variable(CNativeInt(), "rank")), - FunctionDefArgument(Variable(CNativeInt(), "flag")), - FunctionDefArgument(Variable(NumpyBoolType(), "allow_empty")), - ], - body=[], - results=FunctionDefResult(Variable(NumpyBoolType(), "b")), -) - -get_strides_and_shape_from_numpy_array = FunctionDef( - name="get_strides_and_shape_from_numpy_array", - arguments=[ - FunctionDefArgument(Variable(PythonObjectType(), "arr", memory_handling="alias")), - FunctionDefArgument( - Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - "base_shape", - memory_handling="alias", - ) - ), - FunctionDefArgument( - Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - "ubounds", - memory_handling="alias", - ) - ), - FunctionDefArgument( - Variable( - NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), - "strides", - memory_handling="alias", - ) - ), - FunctionDefArgument(Variable(NumpyBoolType(), "c_order")), - ], - body=[], -) - -PyArray_DATA = FunctionDef( - name="PyArray_DATA", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), "arr", memory_handling="alias"))], - results=FunctionDefResult(Variable(VoidType(), "data", memory_handling="alias")), -) - -PyArray_SetBaseObject = FunctionDef( - name="PyArray_SetBaseObject", - body=[], - arguments=[ - FunctionDefArgument(Variable(NumpyArrayObjectType(), name="arr", memory_handling="alias")), - FunctionDefArgument(Variable(PythonObjectType(), name="obj", memory_handling="alias")), - ], - results=FunctionDefResult(Variable(CNativeInt(), name="d")), -) - -PyArray_CHKFLAGS = FunctionDef( - name="PyArray_CHKFLAGS", - body=[], - arguments=[ - FunctionDefArgument(Variable(NumpyArrayObjectType(), name="arr", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), name="flags")), - ], - results=FunctionDefResult(Variable(NumpyBoolType(), name="b")), -) - -PyArray_CLEARFLAGS = FunctionDef( - name="PyArray_CLEARFLAGS", - body=[], - arguments=[ - FunctionDefArgument(Variable(NumpyArrayObjectType(), name="arr", memory_handling="alias")), - FunctionDefArgument(Variable(CNativeInt(), name="flags")), - ], -) - -PyArray_ISNOTSWAPPED = FunctionDef( - name="PyArray_ISNOTSWAPPED", - body=[], - arguments=[FunctionDefArgument(Variable(NumpyArrayObjectType(), name="arr", memory_handling="alias"))], - results=FunctionDefResult(Variable(NumpyBoolType(), name="b")), -) - -to_pyarray = FunctionDef( - name="to_pyarray", - body=[], - arguments=[ - FunctionDefArgument(Variable(CNativeInt(), name="nd")), - FunctionDefArgument(Variable(CNativeInt(), name="typenum")), - FunctionDefArgument(Variable(VoidType(), name="data", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyNDArrayType.get_new(NumpyInt32Type(), 1, None, raw=True), "shape")), - FunctionDefArgument(Variable(NumpyBoolType(), "c_order")), - FunctionDefArgument(Variable(NumpyBoolType(), "release_memory")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="arr", memory_handling="alias")), -) - -to_numpy_bytes_array = FunctionDef( - name="x2py_to_numpy_bytes_array", - body=[], - arguments=[ - FunctionDefArgument(Variable(CNativeInt(), name="nd")), - FunctionDefArgument(Variable(VoidType(), name="data", memory_handling="alias")), - FunctionDefArgument(Variable(NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None, raw=True), "shape")), - FunctionDefArgument(Variable(NumpyInt64Type(), name="itemsize")), - FunctionDefArgument(Variable(NumpyBoolType(), "c_order")), - FunctionDefArgument(Variable(NumpyBoolType(), "release_memory")), - ], - results=FunctionDefResult(Variable(PythonObjectType(), name="arr", memory_handling="alias")), -) - - -import_array = FunctionDef("import_array", (), ()) - -# Basic Array Flags -# https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_OWNDATA -numpy_flag_own_data = Variable(CNativeInt(), name="NPY_ARRAY_OWNDATA") -# https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_WRITEABLE -numpy_flag_writeable = Variable(CNativeInt(), name="NPY_ARRAY_WRITEABLE") -# https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_ALIGNED -numpy_flag_aligned = Variable(CNativeInt(), name="NPY_ARRAY_ALIGNED") -# https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_C_CONTIGUOUS -numpy_flag_c_contig = Variable(CNativeInt(), name="NPY_ARRAY_C_CONTIGUOUS") -# https://numpy.org/doc/stable/reference/c-api/array.html#c.NPY_ARRAY_F_CONTIGUOUS -numpy_flag_f_contig = Variable(CNativeInt(), name="NPY_ARRAY_F_CONTIGUOUS") - -# Custom Array Flags defined in x2py/stdlib/x2py_runtime/python_runtime.h -no_type_check = Variable(CNativeInt(), name="NO_TYPE_CHECK") -no_order_check = Variable(CNativeInt(), name="NO_ORDER_CHECK") -require_c_contiguous = Variable(CNativeInt(), name="REQUIRE_C_CONTIGUOUS") -require_f_contiguous = Variable(CNativeInt(), name="REQUIRE_F_CONTIGUOUS") -require_any_contiguous = Variable(CNativeInt(), name="REQUIRE_ANY_CONTIGUOUS") - -# https://numpy.org/doc/stable/reference/c-api/dtype.html -numpy_bool_type = Variable(CNativeInt(), name="NPY_BOOL") -numpy_byte_type = Variable(CNativeInt(), name="NPY_BYTE") -numpy_string_type = Variable(CNativeInt(), name="NPY_STRING") -numpy_ubyte_type = Variable(CNativeInt(), name="NPY_UBYTE") -numpy_short_type = Variable(CNativeInt(), name="NPY_SHORT") -numpy_ushort_type = Variable(CNativeInt(), name="NPY_USHORT") -numpy_int32_type = Variable(CNativeInt(), name="NPY_INT32") -numpy_uint_type = Variable(CNativeInt(), name="NPY_UINT") -numpy_long_type = Variable(CNativeInt(), name="NPY_LONG") -numpy_ulong_type = Variable(CNativeInt(), name="NPY_ULONG") -numpy_int64_type = Variable(CNativeInt(), name="NPY_INT64") -numpy_ulonglong_type = Variable(CNativeInt(), name="NPY_ULONGLONG") -numpy_float_type = Variable(CNativeInt(), name="NPY_FLOAT") -numpy_double_type = Variable(CNativeInt(), name="NPY_DOUBLE") -numpy_longdouble_type = Variable(CNativeInt(), name="NPY_LONGDOUBLE") -numpy_cfloat_type = Variable(CNativeInt(), name="NPY_CFLOAT") -numpy_cdouble_type = Variable(CNativeInt(), name="NPY_CDOUBLE") -numpy_clongdouble_type = Variable(CNativeInt(), name="NPY_CLONGDOUBLE") - -numpy_dtype_registry = { - CharType(): numpy_byte_type, - NumpyBoolType(): numpy_bool_type, - NumpyInt8Type(): numpy_byte_type, - NumpyInt16Type(): numpy_short_type, - NumpyInt32Type(): numpy_int32_type, - NumpyInt64Type(): numpy_int64_type, - NumpyFloat32Type(): numpy_float_type, - NumpyFloat64Type(): numpy_double_type, - NumpyFloat128Type(): numpy_longdouble_type, - NumpyComplex64Type(): numpy_cfloat_type, - NumpyComplex128Type(): numpy_cdouble_type, - NumpyComplex256Type(): numpy_clongdouble_type, -} - -# Needed to check for NumPy arguments type -check_type_registry.update( - { - NumpyInt8Type(): "PyIs_Int8", - NumpyInt16Type(): "PyIs_Int16", - NumpyInt32Type(): "PyIs_Int32", - NumpyInt64Type(): "PyIs_Int64", - NumpyFloat32Type(): "PyIs_Float", - NumpyFloat64Type(): "PyIs_Double", - NumpyComplex64Type(): "PyIs_Complex64", - NumpyComplex128Type(): "PyIs_Complex128", - } -) - -c_to_py_registry.update( - { - NumpyInt8Type(): "Int8_to_NumpyLong", - NumpyInt16Type(): "Int16_to_NumpyLong", - NumpyInt32Type(): "Int32_to_NumpyLong", - NumpyInt64Type(): "Int64_to_NumpyLong", - NumpyFloat32Type(): "Float_to_NumpyDouble", - NumpyFloat64Type(): "Double_to_NumpyDouble", - NumpyComplex64Type(): "Complex64_to_NumpyComplex", - NumpyComplex128Type(): "Complex128_to_NumpyComplex", - } -) - -pytype_parse_registry.update( - { - NumpyInt8Type(): "b", - NumpyInt16Type(): "h", - NumpyInt32Type(): "i", - NumpyInt64Type(): "l", - NumpyFloat32Type(): "f", - NumpyFloat64Type(): "d", - NumpyComplex64Type(): "O", - NumpyComplex128Type(): "O", - } -) diff --git a/x2py/codegen/bridges/fortran_to_c.py b/x2py/codegen/bridges/fortran_to_c.py deleted file mode 100644 index 261061d77..000000000 --- a/x2py/codegen/bridges/fortran_to_c.py +++ /dev/null @@ -1,4939 +0,0 @@ -""" -Module describing the code-wrapping class : FortranToCWrapper -which creates an interface exposing Fortran code to C. -THIS CREATES BIND(C) FORTRAN FILE -""" - -import re -from dataclasses import replace -from functools import reduce -from typing import ClassVar - -from x2py.semantics.ownership import ( - AssignmentMode, - CodegenAction, - DestructionPolicy, - NativeBarrierAction, - NativeBarrierDispatcher, - ObjectKind, - OwnershipDecision, - OwnershipOwner, - PolicyActionDispatcher, - PythonBarrierAction, - SetterAction, - SetterActionDispatcher, - StorageMode, - TransferMode, - ownership_decision_for_codegen_variable, -) -from x2py.semantics.models import ( - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA, - INTERNAL_MODULE_VARIABLE_NAME_METADATA, - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA, - INTERNAL_NATIVE_ARRAY_HANDLE_OWNER_CLASS_METADATA, - RESOLVED_CLASS_INSTANCE_POLICY_METADATA, - RESOLVED_CLASS_SELF_POLICY_METADATA, - RUNTIME_HOLD_GIL_METADATA, - RUNTIME_RETAIN_RESULT_OWNER_METADATA, -) -from x2py.semantics.native_array_handles import ( - ArrayInteropPolicy, - ArrayInteropPolicyDispatcher, - NativeArrayHandlePolicyDispatcher, -) - -from ..bind_c import ( - C_NULL_CHAR, - BindCArrayType, - BindCArrayVariable, - BindCClassDef, - BindCClassProperty, - BindCFunctionDef, - BindCModule, - BindCModuleConstant, - BindCNativeArrayDescriptorType, - BindCNativeArrayHandleProperty, - BindCNativeArrayHandleVariable, - BindCPointer, - BindCResultTupleType, - BindCScalarDescriptorType, - BindCAccessorModuleVariable, - BindCSizeOf, - BindCVariable, - C_F_Pointer, - CLocFunc, - DeallocatePointer, - FortranTransfer, - c_malloc, - native_array_descriptor_argument_type, -) -from ..models.core import ( - AliasAssign, - Allocate, - ArrayAllocated, - ArrayAssociated, - ArrayContiguous, - ArrayLowerBound, - ArrayShapeElement, - ArraySize, - AsName, - Assign, - CaseSection, - Deallocate, - EmptyNode, - FortranCharacterLength, - FunctionAddress, - FunctionCallArgument, - FunctionDef, - FunctionDefArgument, - FunctionDefResult, - get_direct_overload_set, - get_enclosing_module, - If, - IfSection, - Import, - FunctionOverloadSet, - Nullify, - Pass, - Return, - SelectCase, -) -from ..models.datatypes import ( - CharType, - CustomDataType, - FinalType, - FixedSizeNumericType, - NumpyBoolType, - NumpyInt64Type, - TupleType, - NIL, - cast_to, - convert_to_literal, -) -from ..models.core import Slice -from ..models.datatypes import NumpyInt32Type, NumpyNDArrayType, numpy_precision_map -from ..models.core import Add, IsNot, Mul -from ..models.core import DottedVariable, IndexedElement, Variable -from ..scope import Scope - -from ..generator import BridgeGenerator - -_MAX_SUPPORTED_ASSUMED_RANK = 15 - - -class FortranToCBridgeGenerator(BridgeGenerator): - """Create a C-compatible bridge AST for a Fortran module. - - The class follows the same reading order as ``FortranParser``: - - - public generation entrypoint inherited from ``BridgeGenerator``; - - module, function, variable, and class visitors; - - argument conversion helpers; - - result conversion helpers; - - shared predicates and low-level builders. - - Contract-value conversion dispatches only from the completed post-IR object - kind and codegen action. Model-node dispatch remains exclusively owned by - ``_visit``. - - Parameters - ---------- - sharedlib_dirpath : str - The folder where the generated .so file will be located. - verbose : int - The level of verbosity. - """ - - target_language = "C" - start_language = "Fortran" - _NATIVE_BARRIER_DISPATCHER = NativeBarrierDispatcher( - { - NativeBarrierAction.PASS_VALUE: "_convert_native_value_argument", - NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS: "_convert_native_call_local_address_argument", - NativeBarrierAction.PASS_STORAGE_ADDRESS: "_convert_native_storage_address_argument", - NativeBarrierAction.PASS_RAW_ADDRESS: "_convert_native_raw_address_argument", - NativeBarrierAction.PASS_ARRAY_BUFFER: "_convert_native_array_buffer_argument", - NativeBarrierAction.PASS_WRAPPER_ADDRESS: "_convert_native_wrapper_address_argument", - } - ) - _RESULT_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_scalar_result", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_scalar_result", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_scalar_result", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_scalar_result", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_convert_snapshot_scalar_result", - (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_convert_scalar_result", - (ObjectKind.STRING, CodegenAction.COPY_OUT): "_convert_string_result", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_string_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_convert_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_convert_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_convert_array_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_convert_owned_custom_type_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_convert_borrowed_custom_type_result", - } - ) - _FUNCTION_ARGUMENT_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_visible_function_argument", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_convert_replacement_function_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT): "_convert_visible_function_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_convert_visible_function_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_convert_visible_function_argument", - } - ) - _REPLACEMENT_RESULT_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.COPY_IN_OUT): "_build_scalar_replacement_result", - (ObjectKind.STRING, CodegenAction.COPY_IN_OUT): "_build_string_replacement_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_build_array_replacement_result", - } - ) - _NDARRAY_RESULT_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_build_snapshot_copy_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_build_borrowed_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_build_copy_return_array_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_build_copy_return_array_result", - } - ) - _ALLOCATABLE_RESULT_HELPER_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_uses_heap_allocatable_result_helper", - (ObjectKind.NUMPY_ARRAY, CodegenAction.WRAPPER_INSTANCE): "_uses_heap_allocatable_result_helper", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_IN_OUT): "_skips_allocatable_result_helper", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_skips_allocatable_result_helper", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_skips_allocatable_result_helper", - } - ) - _FIELD_ASSIGNMENT_BY_POLICY: ClassVar[dict[AssignmentMode, type]] = { - AssignmentMode.VALUE_COPY: Assign, - AssignmentMode.ALIAS: AliasAssign, - } - _FIELD_SETTER_POLICY_DISPATCHER = SetterActionDispatcher( - { - SetterAction.WRITE_THROUGH: "_build_field_setter", - SetterAction.REJECT_REPLACEMENT: "_skip_field_setter", - SetterAction.OMIT: "_skip_field_setter", - } - ) - _FIELD_GETTER_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_append_value_field_getter", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_append_nullable_scalar_field_getter", - (ObjectKind.STRING, CodegenAction.COPY_OUT): "_append_value_field_getter", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_append_borrowed_array_field_getter", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_append_alias_field_getter", - } - ) - _MODULE_VARIABLE_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_literal_module_constant", - (ObjectKind.SCALAR, CodegenAction.BORROWED_VIEW): "_scalar_module_variable", - (ObjectKind.SCALAR, CodegenAction.SNAPSHOT_COPY): "_scalar_module_variable", - (ObjectKind.STRING, CodegenAction.DIRECT_VALUE): "_literal_module_constant", - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_array_module_variable", - (ObjectKind.NUMPY_ARRAY, CodegenAction.SNAPSHOT_COPY): "_array_module_variable", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_copied_derived_module_constant", - (ObjectKind.DERIVED_TYPE, CodegenAction.BORROWED_VIEW): "_derived_module_variable", - } - ) - _MODULE_ARRAY_GETTER_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.NUMPY_ARRAY, CodegenAction.BORROWED_VIEW): "_borrowed_module_array_getter_result", - } - ) - _NATIVE_ARRAY_HANDLE_DISPATCHER = NativeArrayHandlePolicyDispatcher( - { - ("allocatable", "argument_descriptor"): "_bridge_allocatable_descriptor_argument", - ("allocatable", "borrowed_field_descriptor"): "_bridge_borrowed_native_array_field_handle", - ("allocatable", "borrowed_module_descriptor"): "_bridge_borrowed_native_array_module_handle", - ("allocatable", "optional_absent_handle"): "_bridge_optional_native_array_handle", - ("allocatable", "owned_result_descriptor"): "_bridge_owned_allocatable_result_handle", - ("pointer", "argument_descriptor"): "_bridge_pointer_descriptor_argument", - ("pointer", "borrowed_field_descriptor"): "_bridge_borrowed_native_array_field_handle", - ("pointer", "borrowed_module_descriptor"): "_bridge_borrowed_native_array_module_handle", - ("pointer", "optional_absent_handle"): "_bridge_optional_native_array_handle", - } - ) - _ARRAY_INTEROP_POLICY_DISPATCHER = ArrayInteropPolicyDispatcher( - { - ("argument", "data_buffer"): "_bridge_data_buffer_argument", - ("argument", "descriptor"): "_bridge_descriptor_argument", - ("module_variable", "data_buffer"): "_bridge_data_buffer_module_variable", - ("module_variable", "descriptor"): "_bridge_descriptor_module_variable", - ("result", "data_buffer"): "_bridge_data_buffer_result", - ("result", "descriptor"): "_bridge_descriptor_result", - } - ) - _COPY_RETURN_ARRAY_BY_STORAGE: ClassVar[dict[StorageMode, str]] = { - StorageMode.STACK: "_build_stack_copy_return_array_result", - StorageMode.HEAP: "_build_heap_copy_return_array_result", - } - _CALLBACK_ARGUMENT_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_callback_scalar_argument", - (ObjectKind.SCALAR, CodegenAction.CALL_LOCAL_INPUT): "_convert_callback_scalar_argument", - (ObjectKind.SCALAR, CodegenAction.IN_PLACE_ARGUMENT): "_convert_callback_scalar_storage_writable_argument", - (ObjectKind.SCALAR, CodegenAction.IDENTITY_OUTPUT): "_convert_callback_scalar_storage_output_argument", - (ObjectKind.STRING, CodegenAction.CALL_LOCAL_INPUT): "_convert_callback_string_input_argument", - (ObjectKind.STRING, CodegenAction.IN_PLACE_ARGUMENT): "_convert_callback_string_storage_writable_argument", - (ObjectKind.STRING, CodegenAction.IDENTITY_OUTPUT): "_convert_callback_string_storage_output_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.CALL_LOCAL_INPUT): "_convert_callback_array_input_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IN_PLACE_ARGUMENT): "_convert_callback_array_writable_argument", - (ObjectKind.NUMPY_ARRAY, CodegenAction.IDENTITY_OUTPUT): "_convert_callback_array_output_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.CALL_LOCAL_INPUT): "_convert_callback_derived_input_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.IN_PLACE_ARGUMENT): "_convert_callback_derived_writable_argument", - (ObjectKind.DERIVED_TYPE, CodegenAction.IDENTITY_OUTPUT): "_convert_callback_derived_output_argument", - } - ) - _CALLBACK_RESULT_POLICY_DISPATCHER = PolicyActionDispatcher( - { - (ObjectKind.SCALAR, CodegenAction.DIRECT_VALUE): "_convert_callback_scalar_result", - (ObjectKind.NUMPY_ARRAY, CodegenAction.COPY_OUT): "_convert_callback_array_result", - (ObjectKind.DERIVED_TYPE, CodegenAction.WRAPPER_INSTANCE): "_convert_callback_derived_result", - } - ) - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, sharedlib_dirpath, verbose): - """Initialize state collected while building one bridge module.""" - self._additional_exprs = [] - self._additional_functions = [] - self._generator_names_dict = {} - super().__init__(verbose) - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_Module(self, expr): - """ - Create a BindCModule which is compatible with C. - - Create a BindCModule which provides an interface between C and the - Module described by expr. This includes wrapping functions, - interfaces, classes and module variables. - - Parameters - ---------- - expr : x2py.ast.core.Module - The module to be generated. - - Returns - ------- - x2py.ast.bind_c.BindCModule - The C-compatible module. - """ - # Define scope - scope = expr.scope - mod_scope = Scope( - name=f"bind_c_{expr.name}", - used_symbols=scope.local_used_symbols.copy(), - original_symbols=scope.python_names.copy(), - naming_policy=scope.naming_policy, - public_namespace=scope.public_namespace, - symbol_language=self.start_language, - scope_type="module", - ) - name = mod_scope.get_new_name(f"bind_c_{expr.name}") - self.scope = mod_scope - - # Wrap contents - funcs_to_generate = [f for f in expr.funcs if f.is_semantic and not f.is_private] - - wrapped_funcs = [self._visit(f) for f in funcs_to_generate] - init_func = self._wrapped_special_function(expr.init_func, funcs_to_generate, wrapped_funcs) - free_func = self._wrapped_special_function(expr.free_func, funcs_to_generate, wrapped_funcs) - removed_functions = [ - f for f, wrapped in zip(funcs_to_generate, wrapped_funcs, strict=False) if isinstance(wrapped, EmptyNode) - ] - python_exports = self._wrapped_python_exports(expr, funcs_to_generate, wrapped_funcs) - funcs = [f for f in wrapped_funcs if not isinstance(f, EmptyNode)] - interfaces = self._wrapped_interfaces(expr, python_exports) - classes = [self._visit(f) for f in expr.classes] - self._extend_python_exports(python_exports, expr, expr.classes, classes) - variables, variable_accessor_funcs, variable_sources = self._wrapped_module_variables(expr.variables) - self._extend_variable_python_exports( - python_exports, - expr, - variables, - variable_accessor_funcs, - variable_sources, - ) - funcs.extend(variable_accessor_funcs) - variable_getters = [v for v in variables if isinstance(v, BindCArrayVariable | BindCNativeArrayHandleVariable)] - # Import the module and its dependencies (in case they are used for argument types) - imports = self._module_imports(expr, funcs_to_generate) - - # Ensure renamed datatypes are mapped to their new name - self.scope.imports["cls_constructs"].update(expr.scope.imports["cls_constructs"]) - - self._generator_names_dict[expr.name] = name - - self.exit_scope() - - return BindCModule( - name, - variables, - funcs, - variable_wrappers=variable_getters, - init_func=init_func, - free_func=free_func, - overload_sets=interfaces, - classes=classes, - imports=imports, - original_module=expr, - scope=mod_scope, - removed_functions=removed_functions, - python_exports=python_exports, - ) - - @staticmethod - def _wrapped_python_exports(expr, source_functions, wrapped_functions): - """Map wrapped functions to their explicit Python export paths.""" - if not expr.has_explicit_python_exports: - return None - return { - id(wrapped): expr.get_python_exports(source) - for source, wrapped in zip(source_functions, wrapped_functions, strict=True) - if not isinstance(wrapped, EmptyNode) - } - - def _wrapped_interfaces(self, expr, python_exports): - """Wrap overload sets and attach explicit Python export metadata.""" - interfaces = [] - for item in expr.overload_sets: - wrapped = self._visit(item) - if isinstance(wrapped, EmptyNode): - continue - interfaces.append(wrapped) - if python_exports is not None: - python_exports[id(wrapped)] = expr.get_python_exports(item) - return interfaces - - @staticmethod - def _extend_python_exports(python_exports, expr, sources, wrapped_objects): - """Add source-to-wrapper export mappings when explicit exports exist.""" - if python_exports is None: - return - python_exports.update( - { - id(wrapped): expr.get_python_exports(source) - for source, wrapped in zip(sources, wrapped_objects, strict=True) - } - ) - - def _extend_variable_python_exports( - self, - python_exports, - expr, - variables, - accessor_functions, - variable_sources, - ): - """Attach module-variable and accessor export paths to wrapped objects.""" - if python_exports is None: - return - accessor_ids = {id(function) for function in accessor_functions} - for wrapped in (*variables, *accessor_functions): - exports = expr.get_python_exports(variable_sources[id(wrapped)]) - if id(wrapped) in accessor_ids: - source_name = wrapped.original_function.name - exports = tuple((namespace, str(self.scope.get_python_name(source_name))) for namespace, _ in exports) - python_exports[id(wrapped)] = exports - - @staticmethod - def _wrapped_special_function(original, source_functions, wrapped_functions): - """Return the wrapper corresponding to an optional special function.""" - if original is None: - return None - index = next(index for index, function in enumerate(source_functions) if function == original) - return wrapped_functions[index] - - def _wrapped_module_variables(self, module_variables): - """Split wrapped module variables into storage and accessor functions.""" - variables = [] - accessors = [] - sources = {} - for item in module_variables: - if item.is_private: - continue - variable = self._visit(item) - if isinstance(variable, BindCAccessorModuleVariable): - wrapped_accessors = tuple( - function - for function in (variable.getter_function, variable.setter_function) - if function is not None - ) - accessors.extend(wrapped_accessors) - sources.update({id(function): item for function in wrapped_accessors}) - else: - variables.append(variable) - sources[id(variable)] = item - return variables, accessors, sources - - @staticmethod - def _module_imports(module, wrapped_functions): - """Select imports required by a generated bridge module.""" - if module.imports: - return list(module.imports) - if any(function.is_external for function in wrapped_functions): - return [] - return [Import(module.name, target=module, mod=module)] - - def _visit_FunctionDef(self, expr): - """ - Create a C-compatible function which executes the original function. - - Create a function which can be called from C which internally calls the original - function. It does this by wrapping the arguments and the results and unrolling - the body using self._get_function_def_body to ensure optional arguments are - present before accessing them. With all this information a BindCFunctionDef is - created which is C-compatible. - - Functions which cannot be wrapped raise a warning and return an EmptyNode. This - is the case for functions with functions as arguments. - - Parameters - ---------- - expr : FunctionDef - The function to generate. - - Returns - ------- - BindCFunctionDef - The C-compatible function. - """ - if expr.is_private or not expr.is_semantic: - return EmptyNode() - - if self._can_call_existing_bind_c_directly(expr): - return self._direct_bind_c_function(expr) - - orig_name = expr.cls_name or expr.name - name = self.scope.get_new_name(f"bind_c_{orig_name.lower()}") - self._generator_names_dict[expr.name] = name - self._additional_exprs = [] - self._additional_functions = [] - - # Create the scope - func_scope = self.scope.new_child_scope(name, "function") - self.scope = func_scope - - # Wrap the arguments and collect the expressions passed as the call argument. - generated_args, projected_argument_results = self._convert_function_arguments(expr) - - func_arguments = [a["c_arg"] for a in generated_args if a["c_arg"] is not None] - call_arguments = [a["f_arg"] for a in generated_args] - - result_infos, func_call_results = self._convert_function_result(expr) - result_infos.extend(projected_argument_results) - func_results = self._function_result_value(result_infos) - - overload_set = get_direct_overload_set(expr) - - call_target = overload_set or expr - body = self._get_function_def_body(call_target, generated_args, func_call_results) - - body.extend(self._additional_exprs) - self._additional_exprs.clear() - additional_functions = self._additional_functions - self._additional_functions = [] - - self._append_destructor_cleanup(expr, call_arguments, body) - - self.exit_scope() - - imports = self._function_imports(expr) - - func = BindCFunctionDef( - name, - func_arguments, - body, - FunctionDefResult(func_results), - imports=imports, - functions=additional_functions, - scope=func_scope, - original_function=expr, - docstring=expr.docstring, - result_pointer_map=expr.result_pointer_map, - ) - - self.scope.insert_function(func, name) - - return func - - def _convert_function_arguments(self, function): - """Convert every function argument and collect projected results.""" - generated_args = [] - projected_results = [] - for argument in function.arguments: - generated_arg, projected_result = self._convert_function_argument(argument, function) - generated_args.append(generated_arg) - if projected_result is not None: - projected_results.append(projected_result) - return generated_args, projected_results - - def _convert_function_argument(self, argument, function): - """Convert one function argument and its optional projected result.""" - if isinstance(argument.var, FunctionAddress): - return self._convert_argument(argument, function), None - decision = ownership_decision_for_codegen_variable(argument.var) - if decision.projects_result and not decision.python_visible: - return self._convert_hidden_function_argument(argument.var, decision, argument, function) - return self._FUNCTION_ARGUMENT_POLICY_DISPATCHER.dispatch( - self, - argument.var, - argument, - function, - ) - - def _convert_visible_function_argument(self, _var, _decision, argument, function): - """Convert an ordinary Python-visible native argument.""" - generated = self._convert_argument(argument, function) - return generated, None - - def _convert_replacement_function_argument(self, _var, _decision, argument, function): - """Convert one visible argument and collect its replacement result.""" - generated = self._convert_argument(argument, function) - result = self._REPLACEMENT_RESULT_DISPATCHER.dispatch(self, argument.var, generated) - self._additional_exprs.extend(result["body"]) - return generated, result - - def _convert_hidden_function_argument(self, _var, _decision, argument, function): - """Create native output storage for a Python-hidden argument.""" - if argument.bound_argument: - raise ValueError(f"Bound argument {argument.var.name!r} cannot be a hidden output") - result = self._convert_result(argument.var, function.scope) - self._additional_exprs.extend(result["body"]) - keyword = None if self._uses_positional_native_call(function) else argument.var.name - generated = { - "c_arg": None, - "f_arg": FunctionCallArgument(result["f_result"], keyword=keyword), - "body": [], - } - return generated, result - - def _convert_function_result(self, function): - """Convert the explicit function result into bridge result metadata.""" - if function.results.var is NIL: - return [], [] - result = self._convert_result(function.results.var, function.scope) - self._additional_exprs.extend(result["body"]) - call_results = self.scope.collect_all_tuple_elements(result["f_result"]) - return [result], call_results - - def _function_result_value(self, result_infos): - """Build the C-visible result value for converted result metadata.""" - if not result_infos: - return NIL - if len(result_infos) == 1: - return result_infos[0]["c_result"] - return self._pack_function_results(result_infos) - - @staticmethod - def _append_destructor_cleanup(function, call_arguments, body) -> None: - """Append pointer cleanup required by a wrapped destructor.""" - if function.scope.get_python_name(function.name) != "__del__" or not call_arguments: - return - if function.is_external: - body.pop() - body.append(DeallocatePointer(call_arguments[0].value)) - - def _function_imports(self, function): - """Return direct imports required to call an external function.""" - return [] - - def _visit_FunctionOverloadSet(self, expr): - """ - Create an interface containing only C-compatible functions. - - Create an interface containing only functions which can be called from C - from an interface which is not necessarily C-compatible. - - Parameters - ---------- - expr : x2py.ast.core.FunctionOverloadSet - The interface to be wrapped. - - Returns - ------- - x2py.ast.core.FunctionOverloadSet - The C-compatible interface. - """ - functions = [wrapped for item in expr.functions if not isinstance(wrapped := self._visit(item), EmptyNode)] - if not functions: - return EmptyNode() - return FunctionOverloadSet( - expr.name, - functions, - expr.is_argument, - native_name=expr.native_name, - native_names=expr.native_names, - ) - - def _visit_Variable(self, expr): - """ - Create all objects necessary to expose a module variable to C. - - Create and return the objects which must be printed in the wrapping - module in order to expose the variable to C. In the case of scalar - numerical values nothing needs to be done so an EmptyNode is returned. - In the case of numerical arrays a C-compatible function must be created - which returns the array. This is necessary because built-in Fortran - arrays are not C-compatible. In the case of classes a C-compatible - function is also created which returns a pointer to the class object. - - Parameters - ---------- - expr : x2py.ast.variables.Variable - The module variable. - - Returns - ------- - codegen model object - The AST object describing the code which must be printed in - the wrapping module to expose the variable. - """ - if expr.array_interop_policy is not None: - return self._ARRAY_INTEROP_POLICY_DISPATCHER.dispatch( - self, - expr, - expr.array_interop_policy, - "module_variable", - ) - return self._MODULE_VARIABLE_POLICY_DISPATCHER.dispatch(self, expr) - - def _bridge_data_buffer_module_variable(self, subject, _policy): - """Expose an ordinary array module variable through the data-buffer ABI.""" - return self._MODULE_VARIABLE_POLICY_DISPATCHER.dispatch(self, subject) - - def _bridge_descriptor_module_variable(self, subject, policy): - """Expose a native array handle module variable through descriptor ABI.""" - self._validate_descriptor_array_interop_policy(subject, policy) - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - subject, - subject.native_array_handle_policy, - ) - - def _array_module_variable(self, expr, decision): - """Build a borrowed module-array accessor from completed policy.""" - getter_policy = expr.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Module variable {expr.name!r} is missing completed getter policy") - scope = self.scope - func_name = scope.get_new_name("bind_c_" + expr.name.lower()) - func_scope = scope.new_child_scope(func_name, "function") - mod = get_enclosing_module(expr) - assert mod is not None - func_scope.imports["variables"][expr.name] = expr - self.scope = func_scope - getter_value = expr.clone( - expr.name, - ownership_decision=getter_policy, - memory_handling=getter_policy.storage_mode.value, - ) - result = self._MODULE_ARRAY_GETTER_POLICY_DISPATCHER.dispatch_decision( - self, - getter_value, - getter_policy, - expr, - ) - if decision.nullable: - unallocated_body = [ - Assign(result["bind_var"], NIL), - *[ - Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) - for shape_var in result["shape_vars"] - ], - ] - result["body"] = [ - If( - IfSection(ArrayAllocated(expr), result["body"]), - IfSection(convert_to_literal(True), unallocated_body), - ) - ] - self.exit_scope() - func = BindCFunctionDef( - name=func_name, - body=result["body"], - arguments=[], - results=FunctionDefResult(result["c_result"]), - imports=self._module_variable_imports(expr), - scope=func_scope, - original_function=expr, - ) - return expr.clone( - expr.name, - new_class=BindCArrayVariable, - wrapper_function=func, - original_variable=expr, - ) - - def _native_array_module_handle(self, subject, policy): - """Build the Bind-C model for a borrowed native-array module handle.""" - return subject.clone( - subject.name, - new_class=BindCNativeArrayHandleVariable, - operation_functions=self._native_array_module_handle_operations(subject, policy), - native_array_handle_policy=policy, - original_variable=subject, - ) - - def _native_array_field_handle(self, subject, policy): - """Build generated parent-bound operations for a native-array field handle.""" - owner_class = subject.lhs.cls_base - if owner_class is None: - raise ValueError(f"Native array field {subject.name!r} has no containing wrapper class") - return BindCNativeArrayHandleProperty( - owner_class.scope.get_python_name(subject.name), - class_type=subject.lhs.dtype, - operation_functions=self._native_array_field_handle_operations(subject, policy), - original_variable=subject, - owner_class=owner_class, - native_array_handle_policy=policy, - ) - - def _native_array_field_handle_operations(self, subject, policy): - """Generate private parent-bound operations for one descriptor field.""" - operations = [ - ("shape", self._native_array_field_shape_operation(subject, policy)), - ("array_actual", self._native_array_field_pointer_operation(subject, policy, "array_actual")), - ( - "descriptor", - self._native_array_field_descriptor_view_operation(subject, policy, "descriptor") - if policy.descriptor_kind == "pointer" - else self._native_array_field_pointer_operation(subject, policy, "descriptor"), - ), - ( - self._native_array_state_operation_name(policy), - self._native_array_field_state_operation(subject, policy), - ), - ( - "native_byte_order", - self._native_array_field_constant_bool_operation(subject, policy, "native_byte_order"), - ), - ("aligned", self._native_array_field_constant_bool_operation(subject, policy, "aligned")), - ("writeable", self._native_array_field_constant_bool_operation(subject, policy, "writeable")), - ] - if policy.to_numpy != "unsupported": - operation = ( - self._native_array_field_descriptor_view_operation(subject, policy, "to_numpy") - if policy.requires_pointer_c_descriptor_interop - else self._native_array_field_to_numpy_operation(subject, policy) - ) - operations.append(("to_numpy", operation)) - if policy.descriptor_kind == "allocatable": - if policy.allows("deallocate"): - operations.append(("deallocate", self._native_array_field_deallocate_operation(subject, policy))) - if policy.allows("resize"): - operations.append(("resize", self._native_array_field_allocatable_resize_operation(subject, policy))) - elif policy.allows("nullify"): - operations.append(("contiguous", self._native_array_field_contiguous_operation(subject, policy))) - operations.append(("nullify", self._native_array_field_nullify_operation(subject, policy))) - if policy.allows("allocate"): - operations.append(("allocate", self._native_array_field_pointer_allocate_operation(subject, policy))) - if policy.allows("deallocate"): - operations.append( - ("deallocate", self._native_array_field_pointer_deallocate_operation(subject, policy)) - ) - if policy.allows("resize"): - operations.append(("resize", self._native_array_field_pointer_resize_operation(subject, policy))) - return tuple(operations) - - def _native_array_field_operation_context(self, subject, operation): - """Enter one parent-bound field operation and expose ``parent%field``.""" - outer_scope = self.scope - owner_class = subject.lhs.cls_base - class_scope = owner_class.scope - original_name = class_scope.get_new_name(f"__x2py_{subject.name}_{operation}", object_type="wrapper") - func_name = outer_scope.get_new_name( - f"bind_c_{owner_class.name}_{subject.name}_{operation}", - object_type="wrapper", - ) - func_scope = outer_scope.new_child_scope(func_name, "function") - self.scope = func_scope - parent_argument = FunctionDefArgument(subject.lhs.clone(subject.lhs.name), bound_argument=True) - converted_parent = self._convert_argument(parent_argument, subject) - parent = converted_parent["f_arg"].value - field = subject.clone(subject.name, lhs=parent) - return { - "outer_scope": outer_scope, - "class_scope": class_scope, - "owner_class": owner_class, - "original_name": original_name, - "func_name": func_name, - "func_scope": func_scope, - "arguments": [converted_parent["c_arg"]], - "body": list(converted_parent["body"]), - "field": field, - } - - def _finish_native_array_field_operation( - self, - subject, - operation, - context, - result, - *, - original_arguments=(), - ): - """Leave a parent-bound operation scope and create its Bind-C function.""" - self.exit_scope() - original_result = result.original_var if isinstance(result, BindCVariable) else result - if original_result is not NIL: - original_result = original_result.clone( - f"{subject.name}_{operation}_value", - new_class=Variable, - is_argument=False, - is_optional=False, - ) - self._copy_native_array_module_original_result_aliases( - context["func_scope"], - context["class_scope"], - result, - original_result, - ) - original_parent = subject.lhs.clone(subject.lhs.name) - original_function = FunctionDef( - context["original_name"], - [FunctionDefArgument(original_parent, bound_argument=True), *original_arguments], - [], - FunctionDefResult(original_result), - scope=context["class_scope"], - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA: True, - INTERNAL_NATIVE_ARRAY_HANDLE_OWNER_CLASS_METADATA: context["owner_class"], - RUNTIME_RETAIN_RESULT_OWNER_METADATA: operation == "to_numpy", - }, - is_private=True, - ) - return BindCFunctionDef( - context["func_name"], - context["arguments"], - context["body"], - FunctionDefResult(result), - scope=context["func_scope"], - original_function=original_function, - ) - - def _native_array_field_shape_operation(self, subject, policy): - """Return current extents for a descriptor field.""" - context = self._native_array_field_operation_context(subject, "shape") - result, shape_vars = self._native_array_shape_operation_result(subject) - context["body"].extend( - Assign(shape_var, ArrayShapeElement(context["field"], convert_to_literal(index))) - for index, shape_var in enumerate(shape_vars) - ) - return self._finish_native_array_field_operation(subject, "shape", context, result) - - def _native_array_field_pointer_operation(self, subject, policy, operation): - """Return the current data address for an allocated or associated field.""" - context = self._native_array_field_operation_context(subject, operation) - result = self._native_array_module_scalar_result( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - context["body"].extend( - [ - Assign(result, NIL), - If( - IfSection( - self._native_array_present_expr(context["field"], policy), - [CLocFunc(context["field"], result)], - ) - ), - ] - ) - return self._finish_native_array_field_operation(subject, operation, context, result) - - def _native_array_field_null_pointer_operation(self, subject, policy, operation): - """Return an explicit null pointer for an unavailable descriptor operation.""" - context = self._native_array_field_operation_context(subject, operation) - result = self._native_array_module_scalar_result( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - context["body"].append(Assign(result, NIL)) - return self._finish_native_array_field_operation(subject, operation, context, result) - - def _native_array_field_descriptor_view_operation(self, subject, policy, operation): - """Associate a standard output descriptor with ``parent%field``.""" - context = self._native_array_field_operation_context(subject, operation) - argument, descriptor = self._native_array_descriptor_output_argument(subject, policy) - context["arguments"].append(argument) - context["body"].append(AliasAssign(descriptor, context["field"])) - return self._finish_native_array_field_operation(subject, operation, context, NIL) - - def _native_array_field_state_operation(self, subject, policy): - """Return allocated or associated state for a descriptor field.""" - operation = self._native_array_state_operation_name(policy) - context = self._native_array_field_operation_context(subject, operation) - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - context["body"].append(Assign(result, self._native_array_present_expr(context["field"], policy))) - return self._finish_native_array_field_operation(subject, operation, context, result) - - def _native_array_field_contiguous_operation(self, subject, policy): - """Return whether the current pointer field target is contiguous.""" - context = self._native_array_field_operation_context(subject, "contiguous") - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_contiguous"), - ) - context["body"].append(Assign(result, ArrayContiguous(context["field"]))) - return self._finish_native_array_field_operation(subject, "contiguous", context, result) - - def _native_array_field_constant_bool_operation(self, subject, policy, operation): - """Return one completed constant array-storage fact for a field.""" - context = self._native_array_field_operation_context(subject, operation) - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - context["body"].append(Assign(result, convert_to_literal(True, dtype=NumpyBoolType()))) - return self._finish_native_array_field_operation(subject, operation, context, result) - - def _native_array_field_to_numpy_operation(self, subject, policy): - """Expose a descriptor field through the completed array extraction policy.""" - context = self._native_array_field_operation_context(subject, "to_numpy") - getter_policy = subject.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Native array field {subject.name!r} is missing completed getter policy") - data_subject = subject.clone( - subject.name, - new_class=Variable, - ownership_decision=getter_policy, - memory_handling=getter_policy.storage_mode.value, - native_array_handle_policy=None, - array_interop_policy=ArrayInteropPolicy( - abi="data_buffer", - owner=f"field {subject.name} extraction", - ), - ) - local_var = data_subject.clone( - self.scope.get_new_name(subject.name), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.boundary_storage_mode.value, - ) - self.scope.insert_variable(local_var) - result = self._build_borrowed_array_result(data_subject, getter_policy, subject.name, local_var) - context["body"].extend([AliasAssign(local_var, context["field"]), *result["body"]]) - return self._finish_native_array_field_operation(subject, "to_numpy", context, result["c_result"]) - - def _native_array_field_deallocate_operation(self, subject, policy): - """Deallocate an allocatable descriptor field.""" - context = self._native_array_field_operation_context(subject, "deallocate") - context["body"].append(If(IfSection(ArrayAllocated(context["field"]), [Deallocate(context["field"])]))) - return self._finish_native_array_field_operation(subject, "deallocate", context, NIL) - - def _native_array_field_allocatable_resize_operation(self, subject, policy): - """Resize an allocatable descriptor field.""" - context = self._native_array_field_operation_context(subject, "resize") - arguments, original_arguments, shape = self._native_array_field_shape_arguments(subject, context) - context["arguments"].extend(arguments) - context["body"].append(Allocate(context["field"], shape=shape, status="unknown")) - return self._finish_native_array_field_operation( - subject, - "resize", - context, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_field_nullify_operation(self, subject, policy): - """Nullify a pointer descriptor field.""" - context = self._native_array_field_operation_context(subject, "nullify") - context["body"].append(Nullify(context["field"])) - return self._finish_native_array_field_operation(subject, "nullify", context, NIL) - - def _native_array_field_pointer_allocate_operation(self, subject, policy): - """Allocate target storage through a policy-enabled pointer field.""" - context = self._native_array_field_operation_context(subject, "allocate") - arguments, original_arguments, shape = self._native_array_field_shape_arguments(subject, context) - context["arguments"].extend(arguments) - context["body"].append(Allocate(context["field"], shape=shape, status="unallocated")) - return self._finish_native_array_field_operation( - subject, - "allocate", - context, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_field_pointer_deallocate_operation(self, subject, policy): - """Deallocate target storage through a policy-enabled pointer field.""" - context = self._native_array_field_operation_context(subject, "deallocate") - context["body"].append(If(IfSection(ArrayAssociated(context["field"]), [Deallocate(context["field"])]))) - return self._finish_native_array_field_operation(subject, "deallocate", context, NIL) - - def _native_array_field_pointer_resize_operation(self, subject, policy): - """Replace target storage through a policy-enabled pointer field.""" - context = self._native_array_field_operation_context(subject, "resize") - arguments, original_arguments, shape = self._native_array_field_shape_arguments(subject, context) - context["arguments"].extend(arguments) - context["body"].extend( - [ - If(IfSection(ArrayAssociated(context["field"]), [Deallocate(context["field"])])), - Allocate(context["field"], shape=shape, status="unallocated"), - ] - ) - return self._finish_native_array_field_operation( - subject, - "resize", - context, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_field_shape_arguments(self, subject, context): - """Return scalar extent arguments following the bound parent argument.""" - arguments = [] - original_arguments = [] - shape = [] - for index in range(subject.rank): - name = f"extent_{index + 1}" - original = self._native_array_module_scalar_argument(NumpyInt64Type(), name) - converted = self._build_numeric_argument( - original, - ownership_decision_for_codegen_variable(original), - needs_pointer_bridge=False, - direct_memory_handling=StorageMode.STACK.value, - ) - arguments.append(FunctionDefArgument(converted["c_arg"])) - original_arguments.append(FunctionDefArgument(original)) - shape.append(converted["f_arg"]) - return arguments, original_arguments, tuple(shape) - - def _native_array_module_handle_operations(self, subject, policy): - """Generate private module-storage operations for a runtime handle.""" - operations = [ - ("shape", self._native_array_module_shape_operation(subject, policy)), - ("array_actual", self._native_array_module_array_actual_operation(subject, policy)), - ( - "descriptor", - self._native_array_module_descriptor_view_operation(subject, policy, "descriptor") - if policy.descriptor_kind == "pointer" - else self._native_array_module_array_actual_operation(subject, policy, "descriptor"), - ), - ( - self._native_array_state_operation_name(policy), - self._native_array_module_state_operation(subject, policy), - ), - ( - "native_byte_order", - self._native_array_module_constant_bool_operation(subject, policy, "native_byte_order"), - ), - ("aligned", self._native_array_module_constant_bool_operation(subject, policy, "aligned")), - ("writeable", self._native_array_module_constant_bool_operation(subject, policy, "writeable")), - ] - if policy.to_numpy != "unsupported": - if policy.requires_pointer_c_descriptor_interop: - operation = self._native_array_module_descriptor_view_operation(subject, policy, "to_numpy") - else: - operation = self._native_array_module_to_numpy_operation(subject, policy) - operations.append(("to_numpy", operation)) - if policy.descriptor_kind == "allocatable": - if policy.allows("deallocate"): - operations.append(("deallocate", self._native_array_module_deallocate_operation(subject, policy))) - if policy.allows("resize"): - operations.append(("resize", self._native_array_module_allocatable_resize_operation(subject, policy))) - elif policy.allows("nullify"): - operations.append(("contiguous", self._native_array_module_contiguous_operation(subject, policy))) - operations.append(("nullify", self._native_array_module_nullify_operation(subject, policy))) - if policy.allows("allocate"): - operations.append(("allocate", self._native_array_module_pointer_allocate_operation(subject, policy))) - if policy.allows("deallocate"): - operations.append( - ("deallocate", self._native_array_module_pointer_deallocate_operation(subject, policy)) - ) - if policy.allows("resize"): - operations.append(("resize", self._native_array_module_pointer_resize_operation(subject, policy))) - return tuple(operations) - - @staticmethod - def _native_array_state_operation_name(policy): - """Return the runtime state operation name for one descriptor kind.""" - return "allocated" if policy.descriptor_kind == "allocatable" else "associated" - - @staticmethod - def _native_array_module_scalar_result(class_type, name): - """Return a scalar helper result with completed by-value ownership policy.""" - return Variable( - class_type, - name, - ownership_decision=OwnershipDecision( - ObjectKind.SCALAR, - OwnershipOwner.PYTHON, - TransferMode.BY_VALUE, - DestructionPolicy.PYTHON_REFCOUNT, - boundary_storage_mode=StorageMode.STACK, - codegen_action=CodegenAction.DIRECT_VALUE, - python_barrier_action=PythonBarrierAction.NONE, - native_barrier_action=NativeBarrierAction.NONE, - reason="generated native-array handle helper returns a scalar Python value", - ), - ) - - @staticmethod - def _native_array_module_scalar_argument(class_type, name): - """Return a scalar helper argument with completed by-value ownership policy.""" - return Variable( - class_type, - name, - is_argument=True, - ownership_decision=OwnershipDecision( - ObjectKind.SCALAR, - OwnershipOwner.CALLER, - TransferMode.BY_VALUE, - DestructionPolicy.CALLER, - boundary_storage_mode=StorageMode.STACK, - codegen_action=CodegenAction.DIRECT_VALUE, - python_barrier_action=PythonBarrierAction.SCALAR_VALUE, - native_barrier_action=NativeBarrierAction.PASS_VALUE, - reason="generated native-array handle helper consumes a scalar Python value", - ), - ) - - def _native_array_module_shape_operation(self, subject, policy): - """Return a generated operation that reports current native extents.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - "shape", - ) - result_var, shape_vars = self._native_array_shape_operation_result(subject) - body = [ - Assign(shape_var, ArrayShapeElement(subject, convert_to_literal(index))) - for index, shape_var in enumerate(shape_vars) - ] - return self._finish_native_array_module_operation( - subject, - "shape", - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - result_var, - ) - - def _native_array_shape_operation_result(self, subject): - """Return the result variable and extent fields for a generated shape operation.""" - scope = self.scope - if subject.rank == 1: - result = self._native_array_module_scalar_result( - NumpyInt64Type(), - scope.get_new_name(f"{subject.name}_extent_1"), - ) - return result, (result,) - result_type = BindCResultTupleType.get_new(tuple(NumpyInt64Type() for _ in range(subject.rank))) - result = Variable( - result_type, - scope.get_new_name(f"{subject.name}_shape"), - shape=(convert_to_literal(subject.rank),), - ) - shape_vars = tuple( - self._native_array_module_scalar_result( - NumpyInt64Type(), - scope.get_new_name(f"{subject.name}_extent_{index + 1}"), - ) - for index in range(subject.rank) - ) - for index, shape_var in enumerate(shape_vars): - scope.insert_symbolic_alias(IndexedElement(result, convert_to_literal(index)), shape_var) - return result, shape_vars - - def _native_array_module_array_actual_operation(self, subject, policy, operation="array_actual"): - """Return a generated operation that reports a native array data address.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - operation, - ) - result = self._native_array_module_scalar_result( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - body = [Assign(result, NIL)] - address_source = self._native_array_module_address_source(subject) - if address_source is not None: - body.append( - If(IfSection(self._native_array_present_expr(subject, policy), [CLocFunc(address_source, result)])) - ) - return self._finish_native_array_module_operation( - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - result, - ) - - @staticmethod - def _native_array_module_address_source(subject): - """Return the Fortran expression that may be passed to ``c_loc`` for array data.""" - if subject.is_alias or subject.is_target: - return subject - return None - - def _native_array_module_null_pointer_operation(self, subject, policy, operation): - """Return a generated operation that explicitly reports unavailable native handoff.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - operation, - ) - result = self._native_array_module_scalar_result( - BindCPointer(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - return self._finish_native_array_module_operation( - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - [], - [Assign(result, NIL)], - result, - ) - - def _native_array_module_descriptor_view_operation(self, subject, policy, operation): - """Associate a standard output descriptor with a pointer module target.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - operation, - ) - argument, descriptor = self._native_array_descriptor_output_argument(subject, policy) - return self._finish_native_array_module_operation( - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - [argument], - [AliasAssign(descriptor, subject)], - NIL, - ) - - def _native_array_descriptor_output_argument(self, subject, policy): - """Create one TS 29113 pointer output dummy hidden from Python.""" - descriptor_type = BindCNativeArrayDescriptorType.get_new() - output_decision = OwnershipDecision( - ObjectKind.NUMPY_ARRAY, - OwnershipOwner.CALLER, - TransferMode.IN_PLACE, - DestructionPolicy.CALLER, - storage_mode=StorageMode.ALIAS, - boundary_storage_mode=StorageMode.ALIAS, - codegen_action=CodegenAction.IDENTITY_OUTPUT, - mutates_native=True, - reason="generated pointer descriptor-view operation writes a caller-established C descriptor", - ) - descriptor_tuple = Variable( - descriptor_type, - self.scope.get_new_name(f"{subject.name}_descriptor_output"), - is_argument=True, - shape=(convert_to_literal(1),), - ownership_decision=output_decision, - ) - descriptor = subject.clone( - self.scope.get_new_name(f"{subject.name}_descriptor"), - new_class=Variable, - is_argument=True, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ownership_decision=output_decision, - native_array_handle_policy=replace(policy, handle_kind="argument_descriptor"), - ) - self.scope.insert_symbolic_alias(IndexedElement(descriptor_tuple, convert_to_literal(0)), descriptor) - return FunctionDefArgument(BindCVariable(descriptor_tuple, subject)), descriptor - - def _native_array_module_state_operation(self, subject, policy): - """Return a generated operation that reports allocated/associated state.""" - operation = self._native_array_state_operation_name(policy) - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - operation, - ) - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_state"), - ) - body = [Assign(result, self._native_array_present_expr(subject, policy))] - return self._finish_native_array_module_operation( - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - result, - ) - - def _native_array_module_contiguous_operation(self, subject, policy): - """Return whether the current pointer module target is contiguous.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - "contiguous", - ) - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_contiguous"), - ) - return self._finish_native_array_module_operation( - subject, - "contiguous", - original_name, - func_name, - func_scope, - outer_scope, - [], - [Assign(result, ArrayContiguous(subject))], - result, - ) - - @staticmethod - def _native_array_present_expr(subject, policy): - """Return the descriptor-kind-specific Fortran presence expression.""" - if policy.descriptor_kind == "allocatable": - return ArrayAllocated(subject) - return ArrayAssociated(subject) - - def _native_array_module_constant_bool_operation(self, subject, policy, operation): - """Return a generated operation that reports a constant native-storage fact.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - operation, - ) - result = self._native_array_module_scalar_result( - NumpyBoolType(), - self.scope.get_new_name(f"{subject.name}_{operation}"), - ) - body = [Assign(result, convert_to_literal(True, dtype=NumpyBoolType()))] - return self._finish_native_array_module_operation( - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - result, - ) - - def _native_array_module_to_numpy_operation(self, subject, policy): - """Return a generated operation that extracts the module array through the data-buffer ABI.""" - data_subject = self._native_array_data_module_variable(subject) - getter_policy = data_subject.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Module variable {subject.name!r} is missing completed getter policy") - scope = self.scope - original_name = self._generated_module_function_name(f"__x2py_{subject.name}_to_numpy") - func_name = scope.get_new_name(f"bind_c_{original_name.lower()}") - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - func_scope.imports["variables"][subject.name] = subject - getter_value = data_subject.clone( - subject.name, - ownership_decision=getter_policy, - memory_handling=getter_policy.storage_mode.value, - ) - result = self._MODULE_ARRAY_GETTER_POLICY_DISPATCHER.dispatch_decision( - self, - getter_value, - getter_policy, - data_subject, - ) - if getter_policy.nullable: - empty_body = [ - Assign(result["bind_var"], NIL), - *[ - Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) - for shape_var in result["shape_vars"] - ], - ] - result["body"] = [ - If( - IfSection(self._native_array_present_expr(subject, policy), result["body"]), - IfSection(convert_to_literal(True), empty_body), - ) - ] - self.exit_scope() - return BindCFunctionDef( - func_name, - [], - result["body"], - FunctionDefResult(result["c_result"]), - imports=self._module_variable_imports(subject), - scope=func_scope, - original_function=self._native_array_module_original_function( - subject, - original_name, - access="to_numpy", - result_var=data_subject, - scope=scope, - ), - ) - - @staticmethod - def _native_array_data_module_variable(subject): - """Return the ordinary array-data view of a native-array handle variable.""" - return subject.clone( - subject.name, - new_class=Variable, - native_array_handle_policy=None, - array_interop_policy=ArrayInteropPolicy( - abi="data_buffer", - owner=f"variable {subject.name}", - ), - ) - - def _native_array_module_deallocate_operation(self, subject, policy): - """Return a generated operation that deallocates an allocatable module array.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - "deallocate", - ) - body = [If(IfSection(ArrayAllocated(subject), [Deallocate(subject)]))] - return self._finish_native_array_module_operation( - subject, - "deallocate", - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - NIL, - ) - - def _native_array_module_allocatable_resize_operation(self, subject, policy): - """Return a generated operation that resizes an allocatable module array.""" - ( - outer_scope, - original_name, - func_name, - func_scope, - ) = self._native_array_module_operation_context(subject, "resize") - arguments, original_arguments, shape = self._native_array_module_shape_arguments(subject) - body = [Allocate(subject, shape=shape, status="unknown")] - return self._finish_native_array_module_operation( - subject, - "resize", - original_name, - func_name, - func_scope, - outer_scope, - arguments, - body, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_module_nullify_operation(self, subject, policy): - """Return a generated operation that nullifies a pointer module array.""" - outer_scope, original_name, func_name, func_scope = self._native_array_module_operation_context( - subject, - "nullify", - ) - return self._finish_native_array_module_operation( - subject, - "nullify", - original_name, - func_name, - func_scope, - outer_scope, - [], - [Nullify(subject)], - NIL, - ) - - def _native_array_module_pointer_allocate_operation(self, subject, policy): - """Return a generated operation that allocates storage through a pointer module variable.""" - ( - outer_scope, - original_name, - func_name, - func_scope, - ) = self._native_array_module_operation_context(subject, "allocate") - arguments, original_arguments, shape = self._native_array_module_shape_arguments(subject) - body = [Allocate(subject, shape=shape, status="unallocated")] - return self._finish_native_array_module_operation( - subject, - "allocate", - original_name, - func_name, - func_scope, - outer_scope, - arguments, - body, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_module_pointer_deallocate_operation(self, subject, policy): - """Return a generated operation that deallocates storage through a pointer module variable.""" - ( - outer_scope, - original_name, - func_name, - func_scope, - ) = self._native_array_module_operation_context(subject, "deallocate") - body = [If(IfSection(ArrayAssociated(subject), [DeallocatePointer(subject)]))] - return self._finish_native_array_module_operation( - subject, - "deallocate", - original_name, - func_name, - func_scope, - outer_scope, - [], - body, - NIL, - ) - - def _native_array_module_pointer_resize_operation(self, subject, policy): - """Return a generated operation that reallocates storage through a pointer module variable.""" - ( - outer_scope, - original_name, - func_name, - func_scope, - ) = self._native_array_module_operation_context(subject, "resize") - arguments, original_arguments, shape = self._native_array_module_shape_arguments(subject) - body = [ - If(IfSection(ArrayAssociated(subject), [DeallocatePointer(subject)])), - Allocate(subject, shape=shape, status="unallocated"), - ] - return self._finish_native_array_module_operation( - subject, - "resize", - original_name, - func_name, - func_scope, - outer_scope, - arguments, - body, - NIL, - original_arguments=original_arguments, - ) - - def _native_array_module_shape_arguments(self, subject): - """Return scalar extent arguments for generated shape-changing operations.""" - arguments = [] - original_arguments = [] - shape = [] - for index in range(subject.rank): - name = f"extent_{index + 1}" - original = self._native_array_module_scalar_argument(NumpyInt64Type(), name) - converted = self._build_numeric_argument( - original, - ownership_decision_for_codegen_variable(original), - needs_pointer_bridge=False, - direct_memory_handling=StorageMode.STACK.value, - ) - arguments.append(FunctionDefArgument(converted["c_arg"])) - original_arguments.append(FunctionDefArgument(original)) - shape.append(converted["f_arg"]) - return arguments, original_arguments, tuple(shape) - - def _native_array_module_operation_context(self, subject, operation): - """Enter the function scope for one generated module-handle operation.""" - scope = self.scope - if scope is None: - raise ValueError(f"Native array module handle {subject.name!r} needs an active generation scope") - self.scope = scope - original_name = self._generated_module_function_name(f"__x2py_{subject.name}_{operation}") - func_name = scope.get_new_name(f"bind_c_{original_name.lower()}") - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - func_scope.imports["variables"][subject.name] = subject - return scope, original_name, func_name, func_scope - - def _finish_native_array_module_operation( - self, - subject, - operation, - original_name, - func_name, - func_scope, - outer_scope, - arguments, - body, - result, - *, - original_arguments=(), - ): - """Leave an operation scope and create its private Bind-C function.""" - self.exit_scope() - original_result = self._native_array_module_original_result(subject, operation, result) - self._copy_native_array_module_original_result_aliases( - func_scope, - outer_scope, - result, - original_result, - ) - original_function = self._native_array_module_original_function( - subject, - original_name, - access=operation, - result_var=original_result, - arguments=original_arguments, - scope=outer_scope, - ) - return BindCFunctionDef( - func_name, - arguments, - body, - FunctionDefResult(result), - imports=self._module_variable_imports(subject), - scope=func_scope, - original_function=original_function, - ) - - @staticmethod - def _copy_native_array_module_original_result_aliases(func_scope, original_scope, result, original_result): - """Copy tuple result aliases onto generated operation metadata.""" - if ( - result is NIL - or original_result is NIL - or not isinstance(getattr(result, "class_type", None), BindCResultTupleType) - ): - return - for index, alias in enumerate(func_scope.collect_all_tuple_elements(result)): - original_scope.insert_symbolic_alias( - IndexedElement(original_result, convert_to_literal(index)), - alias, - ) - - @staticmethod - def _native_array_module_original_result(subject, operation, result): - """Return the Python-visible result variable for operation wrapper generation.""" - if result is NIL: - return NIL - if operation == "to_numpy": - return result - return result.clone(f"{subject.name}_{operation}_value", new_class=Variable) - - @staticmethod - def _native_array_module_original_function(subject, name, *, access, result_var, arguments=(), scope): - """Return private source metadata for a generated module-handle operation.""" - return FunctionDef( - name, - list(arguments), - [], - FunctionDefResult(result_var), - scope=scope, - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_MODULE_VARIABLE_NAME_METADATA: subject.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: access, - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA: True, - }, - ) - - def _bridge_borrowed_native_array_module_handle(self, subject, policy): - """Expose completed borrowed module storage as a generated runtime handle.""" - return self._native_array_module_handle(subject, policy) - - def _bridge_borrowed_native_array_field_handle(self, subject, policy): - """Expose completed borrowed field storage as a generated runtime handle.""" - return self._native_array_field_handle(subject, policy) - - def _bridge_allocatable_descriptor_argument(self, subject, policy, *args): - """Pass an allocatable TS29113 descriptor through the bind(C) wrapper.""" - return self._bridge_native_array_descriptor_argument(subject, policy, *args) - - def _bridge_owned_allocatable_result_handle(self, subject, policy, *args): - """Return bridge-local data for transfer into persistent CFI owner storage.""" - name = subject.name - scope = self.scope - scope.insert_symbol(name) - local_var = subject.clone( - scope.get_expected_name(name), - new_class=Variable, - memory_handling=StorageMode.HEAP.value, - shape=None, - is_argument=False, - is_optional=False, - ) - scope.insert_variable(local_var, name) - decision = ownership_decision_for_codegen_variable(subject) - result = self._build_heap_copy_return_array_result(subject, decision, name, local_var) - result["f_result"] = local_var - return result - - def _bridge_pointer_descriptor_argument(self, subject, policy, *args): - """Pass a pointer TS29113 descriptor through the bind(C) wrapper.""" - return self._bridge_native_array_descriptor_argument(subject, policy, *args) - - def _bridge_optional_native_array_handle(self, subject, policy, *args): - """Pass an optional TS29113 descriptor through the bind(C) wrapper.""" - return self._bridge_native_array_descriptor_argument(subject, policy, *args) - - def _bridge_native_array_descriptor_argument(self, subject, policy, expr, func): - """Build the Bind-C descriptor pointer tuple selected by completed policy.""" - descriptor_type = self._native_array_descriptor_argument_type(policy) - scope = self.scope - name = subject.name - scope.insert_symbol(name) - descriptor_tuple = Variable( - descriptor_type, - scope.get_new_name(f"{name}_descriptor_arg"), - is_argument=True, - shape=(convert_to_literal(len(descriptor_type)),), - ) - descriptor_dummy = subject.clone( - scope.get_expected_name(name), - new_class=Variable, - is_argument=True, - is_optional=policy.optional_absent, - memory_handling=StorageMode.ALIAS.value if policy.descriptor_kind == "pointer" else StorageMode.HEAP.value, - ) - scope.insert_symbolic_alias(IndexedElement(descriptor_tuple, convert_to_literal(0)), descriptor_dummy) - presence_var = None - if descriptor_type.has_presence: - presence_var = Variable( - BindCPointer(), - scope.get_new_name(f"{name}_present"), - is_argument=True, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ) - scope.insert_symbolic_alias(IndexedElement(descriptor_tuple, convert_to_literal(1)), presence_var) - return { - "c_arg": self._native_array_descriptor_function_argument(expr, descriptor_tuple, subject), - "f_arg": self._native_array_descriptor_call_argument(expr, func, descriptor_dummy), - "body": [], - "optional_presence_var": presence_var, - } - - @staticmethod - def _native_array_descriptor_function_argument(expr, descriptor_tuple, subject): - """Wrap the generated descriptor tuple as the C-visible function argument.""" - return FunctionDefArgument( - BindCVariable(descriptor_tuple, subject), - value=expr.value, - posonly=expr.is_posonly, - kwonly=expr.is_kwonly, - annotation=expr.annotation, - bound_argument=expr.bound_argument, - bound_argument_position=expr.bound_argument_position, - persistent_target=expr.persistent_target, - is_vararg=expr.is_vararg, - is_kwarg=expr.is_kwarg, - ) - - def _native_array_descriptor_call_argument(self, expr, func, descriptor_dummy): - """Return the native call argument for the descriptor dummy.""" - if self._uses_positional_native_call(func): - return FunctionCallArgument(descriptor_dummy) - return FunctionCallArgument(descriptor_dummy, keyword=self._native_argument_keyword(func, expr)) - - @staticmethod - def _native_array_descriptor_argument_type(policy): - """Return the Bind-C tuple shape selected for a handle descriptor argument.""" - return native_array_descriptor_argument_type(policy) - - @staticmethod - def _validate_descriptor_array_interop_policy(subject, policy) -> None: - """Require descriptor ABI dispatch to carry completed native handle policy.""" - handle_policy = subject.native_array_handle_policy - if handle_policy is None: - raise ValueError(f"Descriptor array interop for {subject.name!r} is missing completed handle policy") - if policy.descriptor_kind != handle_policy.descriptor_kind or policy.handle_kind != handle_policy.handle_kind: - raise ValueError( - f"Descriptor array interop for {subject.name!r} disagrees with completed handle policy: " - f"{policy.descriptor_kind}/{policy.handle_kind} != " - f"{handle_policy.descriptor_kind}/{handle_policy.handle_kind}" - ) - - def _borrowed_module_array_getter_result(self, getter_value, _decision, expr): - """Build a borrowed module-array getter result.""" - return self._get_bind_c_array(expr.name, getter_value, expr.shape, pointer_target=True) - - def _visit_DottedVariable(self, expr): - """ - Create all objects necessary to expose a class attribute to C. - - Create the getter and setter functions which expose the class attribute - to C. Return these objects in a BindCClassProperty. - - Parameters - ---------- - expr : DottedVariable - The class attribute. - - Returns - ------- - BindCClassProperty - An object containing the getter and setter functions which expose - the class attribute to C. - """ - if expr.array_interop_policy is not None and expr.array_interop_policy.is_descriptor: - self._validate_descriptor_array_interop_policy(expr, expr.array_interop_policy) - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - expr, - expr.native_array_handle_policy, - ) - - lhs = expr.lhs - getter_policy = expr.getter_ownership_decision - setter_policy = expr.setter_ownership_decision - if getter_policy is None or setter_policy is None: - raise ValueError(f"Field {expr.name!r} is missing completed accessor policy") - class_dtype = lhs.dtype - # ---------------------------------------------------------------------------------- - # Create getter - # ---------------------------------------------------------------------------------- - getter_name = self.scope.get_new_name(f"{class_dtype.name}_{expr.name}_getter".lower()) - getter_scope = self.scope.new_child_scope(getter_name, "function") - self.scope = getter_scope - self.scope.insert_symbol(expr.name) - getter_value = expr.clone( - expr.name, - lhs=expr.lhs, - ownership_decision=getter_policy, - memory_handling=getter_policy.storage_mode.value, - ) - getter_result_info = self._convert_result(getter_value, lhs.cls_base.scope) - getter_result = getter_result_info["c_result"] - - getter_arg_generator = self._convert_argument(FunctionDefArgument(lhs, bound_argument=True), expr) - self_obj = getter_arg_generator["f_arg"].value - getter_arg = getter_arg_generator["c_arg"] - - getter_body = getter_arg_generator["body"] - - attrib = expr.clone(expr.name, lhs=self_obj) - obj = self.scope.find(expr.name) - self._FIELD_GETTER_POLICY_DISPATCHER.dispatch_decision( - self, - expr, - getter_policy, - attrib, - obj, - getter_result_info, - getter_body, - ) - self._additional_exprs.clear() - self.exit_scope() - - getter = BindCFunctionDef( - getter_name, - (getter_arg,), - getter_body, - FunctionDefResult(getter_result), - original_function=expr, - scope=getter_scope, - ) - - setter = self._FIELD_SETTER_POLICY_DISPATCHER.dispatch(self, expr, setter_policy, lhs) - return BindCClassProperty( - lhs.cls_base.scope.get_python_name(expr.name), - getter, - setter, - lhs.dtype, - getter_policy=getter_policy, - setter_policy=setter_policy, - ) - - @staticmethod - def _skip_field_setter(_expr, _setter_policy, _lhs): - """Omit native setter emission when policy exposes no write-through path.""" - - @staticmethod - def _append_value_field_getter(_expr, _getter_policy, attrib, _obj, getter_result_info, getter_body): - """Copy a scalar-like field getter value into the C-visible result.""" - getter_body.append(Assign(getter_result_info["f_result"], attrib)) - getter_body.extend(getter_result_info["body"]) - - @staticmethod - def _append_alias_field_getter(_expr, _getter_policy, attrib, obj, getter_result_info, getter_body): - """Alias a borrowed field getter target before result conversion.""" - getter_body.append(AliasAssign(obj, attrib)) - getter_body.extend(getter_result_info["body"]) - - def _append_borrowed_array_field_getter(self, expr, getter_policy, attrib, obj, getter_result_info, getter_body): - """Alias a borrowed array field and handle nullable allocatable storage.""" - if not getter_policy.nullable: - self._append_alias_field_getter(expr, getter_policy, attrib, obj, getter_result_info, getter_body) - return - unallocated_body = [ - Assign(getter_result_info["bind_var"], NIL), - *[ - Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) - for shape_var in getter_result_info["shape_vars"] - ], - ] - getter_body.append( - If( - IfSection( - ArrayAllocated(attrib), - [AliasAssign(obj, attrib), *getter_result_info["body"]], - ), - IfSection(convert_to_literal(True), unallocated_body), - ) - ) - - def _append_nullable_scalar_field_getter(self, _expr, getter_policy, attrib, _obj, getter_result_info, getter_body): - """Copy a nullable scalar descriptor field through a C data pointer.""" - getter_body.append( - self._nullable_scalar_snapshot_if( - attrib, - self._nullable_scalar_status_func(getter_policy), - getter_result_info["bind_var"], - getter_result_info["copy_var"], - getter_result_info["size_var"], - ) - ) - - def _build_field_setter(self, expr, setter_policy, lhs): - """Build one field setter from completed storage and setter policies.""" - setter_name = self.scope.get_new_name(f"{lhs.dtype.name}_{expr.name}_setter".lower()) - setter_scope = self.scope.new_child_scope(setter_name, "function") - self.scope = setter_scope - self.scope.insert_symbol(expr.name) - setter_value = expr.clone( - expr.name, - lhs=expr.lhs, - memory_handling=setter_policy.storage_mode.value, - ownership_decision=setter_policy, - ) - - setter_arg_generators = ( - self._convert_argument(FunctionDefArgument(lhs, bound_argument=True), expr), - self._convert_argument(FunctionDefArgument(setter_value), expr), - ) - setter_args = (setter_arg_generators[0]["c_arg"], setter_arg_generators[1]["c_arg"]) - setter_args[1].persistent_target = setter_policy.assignment_mode is AssignmentMode.ALIAS - - self_obj = setter_arg_generators[0]["f_arg"].value - set_val = setter_arg_generators[1]["f_arg"].value - - setter_body = setter_arg_generators[0]["body"] + setter_arg_generators[1]["body"] - - attrib = expr.clone(expr.name, lhs=self_obj) - try: - assignment = self._FIELD_ASSIGNMENT_BY_POLICY[setter_policy.assignment_mode] - except KeyError: - raise ValueError( - f"No field setter assignment for completed policy {setter_policy.assignment_mode.value!r}" - ) from None - setter_body.append(assignment(attrib, set_val)) - self.exit_scope() - - return BindCFunctionDef( - setter_name, - setter_args, - setter_body, - original_function=expr, - scope=setter_scope, - ) - - def _visit_ClassDef(self, expr): - """ - Create all objects necessary to expose a class definition to C. - - Create all objects necessary to expose a class definition to C. - - Parameters - ---------- - expr : ClassDef - The class to be wrapped. - - Returns - ------- - BindCClassDef - The wrapped class. - """ - name = expr.name - instance_policy = expr.decorators[RESOLVED_CLASS_INSTANCE_POLICY_METADATA] - self_policy = expr.decorators[RESOLVED_CLASS_SELF_POLICY_METADATA] - func_name = self.scope.get_new_name(f"{name}_bind_c_alloc".lower()) - func_scope = self.scope.new_child_scope(func_name, "function") - - # Allocatable is not returned so it must appear in local scope - local_var = Variable( - expr.class_type, - func_scope.get_new_name(f"{name}_obj"), - cls_base=expr, - memory_handling=instance_policy.boundary_storage_mode.value, - ownership_decision=instance_policy, - ) - func_scope.insert_variable(local_var) - - # Create the C-compatible data pointer - bind_var = Variable( - BindCPointer(), - func_scope.get_new_name("bound_" + name), - memory_handling="alias", - ) - result = BindCVariable(bind_var, local_var) - - # Define the additional steps necessary to define and fill ptr_var - alloc = Allocate(local_var, shape=None, status="unallocated") - c_loc = CLocFunc(local_var, bind_var) - body = [alloc, c_loc] - - new_method = BindCFunctionDef( - func_name, - [], - body, - FunctionDefResult(result), - original_function=None, - scope=func_scope, - ) - - methods = [self._visit(m) for m in expr.methods] - methods = [m for m in methods if not isinstance(m, EmptyNode)] - for i in expr.overload_sets: - for f in i.functions: - self._visit(f) - interfaces = [self._visit(i) for i in expr.overload_sets] - - del_method = expr.methods_as_dict.get("__del__", None) - if del_method is None: - del_name = expr.scope.get_new_name("__del__") - scope = expr.scope.new_child_scope("__del__", scope_type="function") - scope.local_used_symbols["__del__"] = del_name - scope.python_names[del_name] = "__del__" - argument = FunctionDefArgument( - Variable( - expr.class_type, - scope.get_new_name("self"), - cls_base=expr, - memory_handling=self_policy.boundary_storage_mode.value, - ownership_decision=self_policy, - ), - bound_argument=True, - ) - scope.insert_variable(argument.var) - del_method = FunctionDef(del_name, [argument], [Pass()], scope=scope, is_external=True) - methods.append(self._visit(del_method)) - - if any(isinstance(v.class_type, TupleType) for v in expr.attributes): - raise NotImplementedError("Tuples cannot yet be exposed to Python.") - - properties_getters = [ - BindCClassProperty( - expr.scope.get_python_name(m.original_function.name), - m, - None, - expr.class_type, - m.original_function.docstring, - ) - for m in methods - if "property" in m.original_function.decorators - ] - methods = [ - m for m in methods if m not in properties_getters if "property" not in m.original_function.decorators - ] - - # Pseudo-self variable is useful for pre-defined attributes which are not DottedVariables - pseudo_self = Variable( - expr.class_type, - "self", - cls_base=expr, - memory_handling=self_policy.boundary_storage_mode.value, - ownership_decision=self_policy, - ) - properties = [ - self._visit( - v if isinstance(v, DottedVariable) else v.clone(v.name, new_class=DottedVariable, lhs=pseudo_self) - ) - for v in expr.attributes - if not v.is_private and not isinstance(v.class_type, TupleType) - ] - return BindCClassDef( - expr, - new_func=new_method, - methods=methods, - overload_sets=interfaces, - attributes=properties_getters + properties, - docstring=expr.docstring, - class_type=expr.class_type, - decorators=expr.decorators, - superclasses=expr.superclasses, - ) - - # ------------------------------------------------------------------ - # Datatype conversion - # ------------------------------------------------------------------ - - def _convert_argument(self, expr, func): - """ - Extract the C-compatible FunctionDefArgument from the Fortran FunctionDefArgument. - - Extract the C-compatible FunctionDefArgument from the Fortran FunctionDefArgument. - - The explicit datatype dispatch table selects the conversion helper. - - Parameters - ---------- - expr : FunctionDefArgument - An object representing the FunctionDefArgument in the Fortran code which should - be exposed to the C code. - - func : FunctionDef - The function being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to access the argument. - """ - var = expr.var - if isinstance(var, FunctionAddress): - return self._convert_callback_argument(expr, func) - if var.array_interop_policy is not None: - return self._ARRAY_INTEROP_POLICY_DISPATCHER.dispatch( - self, - var, - var.array_interop_policy, - "argument", - expr, - func, - ) - return self._bridge_non_array_argument(var, expr, func) - - def _bridge_data_buffer_argument(self, subject, _policy, expr, func): - """Convert an ordinary array argument through the data-buffer ABI.""" - return self._bridge_non_array_argument(subject, expr, func) - - def _bridge_descriptor_argument(self, subject, policy, expr, func): - """Convert a native array handle argument through descriptor ABI.""" - self._validate_descriptor_array_interop_policy(subject, policy) - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - subject, - subject.native_array_handle_policy, - expr, - func, - ) - - def _bridge_non_array_argument(self, var, expr, func): - """Convert a function argument without native descriptor-handle routing.""" - func_def_argument_dict = self._NATIVE_BARRIER_DISPATCHER.dispatch(self, var, func) - new_var = func_def_argument_dict["c_arg"] - func_def_argument_dict["c_arg"] = FunctionDefArgument( - new_var, - value=expr.value, - posonly=expr.is_posonly, - kwonly=expr.is_kwonly, - annotation=expr.annotation, - bound_argument=expr.bound_argument, - bound_argument_position=expr.bound_argument_position, - persistent_target=expr.persistent_target, - is_vararg=expr.is_vararg, - is_kwarg=expr.is_kwarg, - ) - - if self._uses_positional_native_call(func): - func_def_argument_dict["f_arg"] = FunctionCallArgument(func_def_argument_dict["f_arg"]) - else: - func_def_argument_dict["f_arg"] = FunctionCallArgument( - func_def_argument_dict["f_arg"], - keyword=self._native_argument_keyword(func, expr), - ) - return func_def_argument_dict - - def _convert_callback_argument(self, expr, func): - """Lower one immediate-call dummy procedure to a C callback plus a Fortran adapter.""" - callback = expr.var - if callback.is_optional: - raise ValueError(f"Optional callback argument {callback.name!s} is not supported") - - callback_name = str(callback.name) - c_name = self.scope.get_new_name(f"bound_{callback_name}") - adapter_name = self.scope.get_new_name(f"adapt_{callback_name}") - c_scope = self.scope.new_child_scope(f"{c_name}_interface", "function") - adapter_scope = self.scope.new_child_scope(adapter_name, "function") - - c_arguments = [] - adapter_arguments = [] - adapter_call_arguments = [] - adapter_body = [] - adapter_post_body = [] - abi_arguments = [] - - for argument in callback.arguments: - native_var = argument.var - adapter_var = native_var.clone( - str(native_var.name), - new_class=Variable, - is_argument=True, - is_target=False, - memory_handling="stack", - ) - adapter_scope.insert_variable(adapter_var, name=str(native_var.name)) - adapter_arguments.append(FunctionDefArgument(adapter_var)) - converted = self._convert_callback_abi_argument( - callback_name, - native_var, - adapter_var, - c_scope, - adapter_scope, - ) - c_arguments.extend(converted["c_arguments"]) - adapter_call_arguments.extend(converted["call_arguments"]) - adapter_body.extend(converted["body"]) - adapter_post_body.extend(converted["post_body"]) - abi_arguments.append(converted["abi"]) - - native_result = callback.results.var - abi_result = {"kind": "none", "native": NIL} - if native_result is NIL: - c_result = FunctionDefResult(NIL) - adapter_result = FunctionDefResult(NIL) - else: - adapter_result_var = native_result.clone( - adapter_scope.get_new_name(f"{callback_name}_result"), - new_class=Variable, - is_argument=False, - is_target=native_result.rank > 0 or isinstance(native_result.class_type, CustomDataType), - ) - adapter_scope.insert_variable(adapter_result_var) - adapter_result = FunctionDefResult(adapter_result_var) - converted_result = self._CALLBACK_RESULT_POLICY_DISPATCHER.dispatch( - self, - native_result, - callback_name, - c_scope, - adapter_result_var, - ) - c_result = converted_result["c_result"] - abi_result = converted_result["abi"] - - c_callback = FunctionAddress( - c_name, - c_arguments, - c_result, - is_argument=True, - decorators={ - "x2py_callback_abi": { - "native": callback, - "arguments": abi_arguments, - "result": abi_result, - } - }, - scope=c_scope, - ) - - callback_call = c_callback(*adapter_call_arguments) - if native_result is NIL: - adapter_body.append(callback_call) - adapter_body.extend(adapter_post_body) - elif abi_result["kind"] == "scalar": - adapter_body.append(Assign(adapter_result.var, callback_call)) - adapter_body.extend(adapter_post_body) - else: - result_pointer = Variable( - BindCPointer(), - adapter_scope.get_new_name(f"{callback_name}_result_data"), - memory_handling="stack", - ) - adapter_scope.insert_variable(result_pointer) - adapter_body.append(Assign(result_pointer, callback_call)) - adapter_body.extend(adapter_post_body) - result_view = adapter_result.var.clone( - adapter_scope.get_new_name(f"{callback_name}_result_view"), - new_class=Variable, - is_argument=False, - memory_handling="alias", - is_target=False, - ) - adapter_scope.insert_variable(result_view) - shape = adapter_result.var.alloc_shape if adapter_result.var.rank > 0 else None - adapter_body.append(C_F_Pointer(result_pointer, result_view, shape)) - adapter_body.append(Assign(adapter_result.var, result_view)) - - adapter = FunctionDef( - adapter_name, - adapter_arguments, - adapter_body, - adapter_result, - decorators={"x2py_callback_adapter": callback}, - scope=adapter_scope, - ) - self._additional_functions.append(adapter) - f_arg = ( - FunctionCallArgument(adapter) - if self._uses_positional_native_call(func) - else FunctionCallArgument(adapter, keyword=self._native_argument_keyword(func, expr)) - ) - return { - "c_arg": FunctionDefArgument(c_callback), - "f_arg": f_arg, - "body": [], - } - - @staticmethod - def _uses_positional_native_call(func) -> bool: - """Return whether the native call should avoid Fortran keywords.""" - if not isinstance(func, FunctionDef): - return False - return not FortranToCBridgeGenerator._has_optional_arguments(func) - - @staticmethod - def _native_argument_keyword(func, expr): - """Return the original native keyword for a generated argument.""" - - if getattr(func, "scope", None) is None: - return expr.name - try: - return func.scope.get_python_name(expr.name) - except RuntimeError: - return expr.name - - def _convert_callback_abi_argument(self, callback_name, native_var, adapter_var, c_scope, adapter_scope): - """Dispatch one callback argument to its ABI converter.""" - return self._CALLBACK_ARGUMENT_POLICY_DISPATCHER.dispatch( - self, - native_var, - callback_name, - adapter_var, - c_scope, - adapter_scope, - ) - - @staticmethod - def _convert_callback_scalar_result(native_result, decision, callback_name, c_scope, _adapter_result): - """Build the C ABI result for a scalar callback result.""" - c_result_var = native_result.clone( - c_scope.get_new_name(f"{callback_name}_result"), - new_class=Variable, - is_argument=False, - memory_handling=decision.boundary_storage_mode.value, - ) - c_scope.insert_variable(c_result_var) - return { - "c_result": FunctionDefResult(c_result_var), - "abi": {"kind": "scalar", "native": native_result, "abi": c_result_var}, - } - - def _convert_callback_array_result(self, native_result, decision, callback_name, c_scope, adapter_result): - """Build the pointer ABI for an array callback result.""" - if any(item is None for item in adapter_result.alloc_shape): - raise ValueError(f"Callback {callback_name!r} array result must have an explicit shape") - return self._convert_callback_pointer_result( - native_result, - decision, - callback_name, - c_scope, - kind="array", - ) - - def _convert_callback_derived_result(self, native_result, decision, callback_name, c_scope, _adapter_result): - """Build the pointer ABI for a derived callback result.""" - return self._convert_callback_pointer_result( - native_result, - decision, - callback_name, - c_scope, - kind="derived", - ) - - @staticmethod - def _convert_callback_pointer_result(native_result, _decision, callback_name, c_scope, *, kind): - """Represent an array or derived callback result with one C pointer.""" - c_result_var = Variable( - BindCPointer(), - c_scope.get_new_name(f"{callback_name}_result_data"), - memory_handling="stack", - ) - c_scope.insert_variable(c_result_var) - return { - "c_result": FunctionDefResult(c_result_var), - "abi": {"kind": kind, "native": native_result, "abi": c_result_var}, - } - - def _convert_callback_scalar_argument( - self, - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - ): - """Convert a scalar callback argument to its interoperable ABI.""" - if decision.python_barrier_action is PythonBarrierAction.SCALAR_STORAGE: - return self._convert_callback_scalar_storage_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - copy_in=True, - copy_out=False, - ) - c_var = native_var.clone( - str(native_var.name), - new_class=Variable, - is_argument=True, - memory_handling="stack", - passes_by_value=True, - ) - c_scope.insert_variable(c_var, name=str(native_var.name)) - return { - "c_arguments": [FunctionDefArgument(c_var)], - "call_arguments": [cast_to(adapter_var, c_var.dtype)], - "body": [], - "post_body": [], - "abi": {"kind": "scalar", "native": native_var, "abi": (c_var,)}, - } - - def _convert_callback_scalar_storage_writable_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Expose a writable scalar callback dummy as rank-0 NumPy storage.""" - return self._convert_callback_scalar_storage_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - copy_in=True, - copy_out=True, - ) - - def _convert_callback_scalar_storage_output_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Expose a scalar callback output dummy as rank-0 NumPy storage.""" - return self._convert_callback_scalar_storage_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - copy_in=False, - copy_out=True, - ) - - def _convert_callback_scalar_storage_argument( - self, - native_var, - _decision, - _callback_name, - adapter_var, - c_scope, - adapter_scope, - *, - copy_in, - copy_out, - ): - """Convert a scalar callback argument to pointer-backed Python storage.""" - data = Variable( - BindCPointer(), - c_scope.get_new_name(f"{native_var.name}_data"), - is_argument=True, - memory_handling="stack", - ) - c_scope.insert_variable(data) - data_value, callback_storage = self._callback_pointer_storage(native_var, adapter_var, adapter_scope) - body = [] - post_body = [] - if copy_in: - body.append(Assign(callback_storage, adapter_var)) - body.append(CLocFunc(callback_storage, data_value)) - if copy_out: - post_body.append(Assign(adapter_var, callback_storage)) - return { - "c_arguments": [FunctionDefArgument(data)], - "call_arguments": [data_value], - "body": body, - "post_body": post_body, - "abi": {"kind": "scalar_storage", "native": native_var, "abi": (data,)}, - } - - def _convert_callback_string_input_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Expose a read-only character callback dummy as a Python string.""" - return self._convert_callback_string_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - kind="string", - copy_in=True, - copy_out=False, - ) - - def _convert_callback_string_storage_writable_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Expose a writable character callback dummy as rank-0 bytes storage.""" - return self._convert_callback_string_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - kind="string_storage", - copy_in=True, - copy_out=True, - ) - - def _convert_callback_string_storage_output_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Expose a character callback output dummy as rank-0 bytes storage.""" - return self._convert_callback_string_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - kind="string_storage", - copy_in=False, - copy_out=True, - ) - - def _convert_callback_string_argument( - self, - native_var, - _decision, - _callback_name, - adapter_var, - c_scope, - adapter_scope, - *, - kind, - copy_in, - copy_out, - ): - """Convert a fixed-length character callback argument to pointer ABI data.""" - fixed_len = self._callback_string_length(native_var) - data = Variable( - BindCPointer(), - c_scope.get_new_name(f"{native_var.name}_data"), - is_argument=True, - memory_handling="stack", - ) - length = Variable( - NumpyInt64Type(), - c_scope.get_new_name(f"{native_var.name}_length"), - is_argument=True, - passes_by_value=True, - ) - c_scope.insert_variable(data) - c_scope.insert_variable(length) - data_value, callback_storage = self._callback_pointer_storage(native_var, adapter_var, adapter_scope) - body = [] - post_body = [] - if copy_in: - body.append(Assign(callback_storage, adapter_var)) - body.append(CLocFunc(callback_storage, data_value)) - if copy_out: - post_body.append(Assign(adapter_var, callback_storage)) - return { - "c_arguments": [FunctionDefArgument(data), FunctionDefArgument(length)], - "call_arguments": [ - data_value, - cast_to(FortranCharacterLength(callback_storage), NumpyInt64Type()), - ], - "body": body, - "post_body": post_body, - "abi": {"kind": kind, "native": native_var, "abi": (data, length), "length": fixed_len}, - } - - @staticmethod - def _callback_string_length(native_var): - """Return a fixed callback character length for diagnostics and ABI metadata.""" - if native_var.fortran_character_length not in (None, "*", ":"): - return native_var.fortran_character_length - if native_var.alloc_shape and native_var.alloc_shape[0] is not None: - return native_var.alloc_shape[0] - raise ValueError(f"Callback string argument {native_var.name!r} requires a fixed character length") - - def _convert_callback_array_input_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy an array callback input into adapter-visible pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=True, - copy_in=True, - copy_out=False, - ) - - def _convert_callback_array_writable_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy an array callback argument into and out of pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=True, - copy_in=True, - copy_out=True, - ) - - def _convert_callback_array_output_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy an array callback output from adapter pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=True, - copy_in=False, - copy_out=True, - ) - - def _convert_callback_derived_input_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy a derived callback input into adapter-visible pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=False, - copy_in=True, - copy_out=False, - ) - - def _convert_callback_derived_writable_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy a derived callback argument into and out of pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=False, - copy_in=True, - copy_out=True, - ) - - def _convert_callback_derived_output_argument( - self, native_var, decision, callback_name, adapter_var, c_scope, adapter_scope - ): - """Copy a derived callback output from adapter pointer storage.""" - return self._convert_callback_pointer_argument( - native_var, - decision, - callback_name, - adapter_var, - c_scope, - adapter_scope, - is_array=False, - copy_in=False, - copy_out=True, - ) - - @staticmethod - def _callback_pointer_storage(native_var, adapter_var, adapter_scope): - """Create adapter-side pointer storage for a callback argument.""" - data_value = Variable( - BindCPointer(), - adapter_scope.get_new_name(f"{native_var.name}_data"), - memory_handling="stack", - ) - adapter_scope.insert_variable(data_value) - callback_storage = adapter_var.clone( - adapter_scope.get_new_name(f"{native_var.name}_callback_storage"), - new_class=Variable, - is_argument=False, - is_target=True, - memory_handling="stack", - ) - adapter_scope.insert_variable(callback_storage) - return data_value, callback_storage - - def _convert_callback_pointer_argument( - self, - native_var, - _decision, - _callback_name, - adapter_var, - c_scope, - adapter_scope, - *, - is_array, - copy_in, - copy_out, - ): - """Convert an array or derived callback argument to pointer ABI data.""" - data = Variable( - BindCPointer(), - c_scope.get_new_name(f"{native_var.name}_data"), - is_argument=True, - memory_handling="stack", - ) - c_scope.insert_variable(data) - dimensions = self._callback_array_dimensions(native_var, c_scope) if is_array else [] - data_value, callback_storage = self._callback_pointer_storage(native_var, adapter_var, adapter_scope) - body = [] - post_body = [] - if copy_in: - body.append(Assign(callback_storage, adapter_var)) - body.append(CLocFunc(callback_storage, data_value)) - if copy_out: - post_body.append(Assign(adapter_var, callback_storage)) - shape_arguments = [ - ArrayShapeElement(callback_storage, convert_to_literal(index)) for index in range(native_var.rank) - ] - return { - "c_arguments": [FunctionDefArgument(item) for item in (data, *dimensions)], - "call_arguments": [data_value, *shape_arguments], - "body": body, - "post_body": post_body, - "abi": { - "kind": "array" if is_array else "derived", - "native": native_var, - "abi": (data, *dimensions), - }, - } - - @staticmethod - def _callback_array_dimensions(native_var, c_scope): - """Create C ABI dimension arguments for a callback array.""" - dimensions = [ - Variable( - NumpyInt64Type(), - c_scope.get_new_name(f"{native_var.name}_shape_{index + 1}"), - is_argument=True, - passes_by_value=True, - ) - for index in range(native_var.rank) - ] - for dimension in dimensions: - c_scope.insert_variable(dimension) - return dimensions - - def _convert_native_value_argument(self, var, decision, func): - """Pass a C-visible scalar value through the native call boundary.""" - if decision.kind is not ObjectKind.SCALAR: - raise ValueError(f"Native value barrier only supports scalar arguments, got {decision.kind.value}") - return self._build_numeric_argument( - var, - decision, - needs_pointer_bridge=var.is_optional, - direct_memory_handling=var.memory_handling, - ) - - def _convert_native_call_local_address_argument(self, var, decision, func): - """Pass the address of bridge-owned call-local native storage.""" - if decision.kind is ObjectKind.STRING: - return self._build_string_argument( - var, decision, copy_back=decision.codegen_action is CodegenAction.COPY_IN_OUT - ) - if decision.kind is ObjectKind.SCALAR and decision.codegen_action is CodegenAction.COPY_IN_OUT: - return self._convert_native_scalar_replacement_argument(var, decision, func) - if decision.kind is not ObjectKind.SCALAR: - raise ValueError( - f"Native call-local address barrier only supports scalar or string arguments, got {decision.kind.value}" - ) - if ( - decision.codegen_action is CodegenAction.CALL_LOCAL_INPUT - and decision.descriptor_boundary - and decision.boundary_storage_mode in {StorageMode.HEAP, StorageMode.ALIAS} - ): - return self._convert_native_scalar_descriptor_input_argument(var, decision) - return self._build_numeric_argument( - var, - decision, - needs_pointer_bridge=var.is_optional, - direct_memory_handling=StorageMode.STACK.value, - ) - - def _convert_native_scalar_descriptor_input_argument(self, var, decision): - """Build a nullable call-local descriptor from a scalar C pointer.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ) - presence_var = None - if self._uses_optional_scalar_descriptor_presence(var, decision): - presence_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}_present"), - is_argument=True, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ) - input_var = var.clone( - scope.get_new_name(f"{name}_input"), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ) - descriptor_var = var.clone( - scope.get_new_name(f"{name}_descriptor"), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=decision.boundary_storage_mode.value, - ) - scope.insert_variable(bind_var) - if presence_var is not None: - scope.insert_variable(presence_var) - scope.insert_variable(input_var) - scope.insert_variable(descriptor_var) - body = [C_F_Pointer(bind_var, input_var)] - if decision.boundary_storage_mode is StorageMode.HEAP: - body.append(If(IfSection(ArrayAssociated(input_var), [Assign(descriptor_var, input_var)]))) - else: - body.append(AliasAssign(descriptor_var, input_var)) - if presence_var is not None: - c_arg_var = Variable( - BindCScalarDescriptorType(), - scope.get_new_name(f"{name}_descriptor_input"), - is_argument=True, - shape=(convert_to_literal(2),), - ) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(0)), bind_var) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(1)), presence_var) - return { - "c_arg": BindCVariable(c_arg_var, var), - "f_arg": descriptor_var, - "body": body, - "optional_presence_var": presence_var, - } - return { - "c_arg": BindCVariable(bind_var, var), - "f_arg": descriptor_var, - "body": body, - } - - @staticmethod - def _uses_optional_scalar_descriptor_presence(var, decision): - """Return whether a scalar descriptor needs supplied-vs-None tracking.""" - return bool( - var.is_optional - and decision.kind is ObjectKind.SCALAR - and decision.descriptor_boundary - and decision.nullable - ) - - def _convert_native_storage_address_argument(self, var, decision, func): - """Pass caller/Python-backed storage through the native call boundary.""" - if decision.kind is ObjectKind.STRING: - return self._build_string_storage_argument(var, decision) - if decision.kind is not ObjectKind.SCALAR: - raise ValueError( - f"Native storage-address barrier only supports scalar arguments, got {decision.kind.value}" - ) - return self._build_numeric_argument( - var, - decision, - needs_pointer_bridge=True, - direct_memory_handling=var.memory_handling, - ) - - def _convert_native_raw_address_argument(self, var, decision, func): - """Pass a caller-supplied raw address through the native call boundary.""" - if decision.kind is ObjectKind.STRING: - return self._convert_raw_string_argument(var, decision) - if decision.kind is ObjectKind.NUMPY_ARRAY: - return self._convert_raw_array_argument(var) - if decision.kind is not ObjectKind.SCALAR: - raise ValueError(f"Native raw-address barrier does not support {decision.kind.value} arguments") - return self._build_numeric_argument( - var, - decision, - needs_pointer_bridge=True, - direct_memory_handling=var.memory_handling, - ) - - def _build_numeric_argument(self, var, decision, *, needs_pointer_bridge, direct_memory_handling): - """Build the numeric bridge representation selected by policy dispatch.""" - name = var.name - self.scope.insert_symbol(name) - collisionless_name = self.scope.get_expected_name(name) - if needs_pointer_bridge: - f_arg = var.clone( - collisionless_name, - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling="alias", - ) - new_var = Variable( - BindCPointer(), - self.scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - body = [C_F_Pointer(new_var, f_arg)] - else: - f_arg = var.clone( - collisionless_name, - new_class=Variable, - is_argument=True, - memory_handling=direct_memory_handling, - ) - new_var = f_arg - body = [] - self.scope.insert_variable(f_arg) - return {"c_arg": BindCVariable(new_var, var), "f_arg": f_arg, "body": body} - - def _convert_native_scalar_replacement_argument(self, var, decision, func): - """Copy an immutable Python scalar into mutable native call storage.""" - if decision.descriptor_boundary: - return self._convert_native_scalar_descriptor_input_argument(var, decision) - name = var.name - scope = self.scope - scope.insert_symbol(name) - input_var = var.clone( - scope.get_expected_name(name), - new_class=Variable, - is_argument=True, - is_optional=False, - memory_handling=StorageMode.STACK.value, - ) - local_var = var.clone( - scope.get_new_name(f"{name}_mutable"), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=decision.boundary_storage_mode.value, - ) - scope.insert_variable(input_var) - scope.insert_variable(local_var) - return { - "c_arg": BindCVariable(input_var, var), - "f_arg": local_var, - "body": [Assign(local_var, input_var)], - } - - def _convert_native_wrapper_address_argument(self, var, decision, func): - """Pass a generated wrapper's native address through the native boundary.""" - name = var.name - self.scope.insert_symbol(name) - collisionless_name = self.scope.get_expected_name(name) - f_arg = var.clone( - collisionless_name, - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=decision.boundary_storage_mode.value, - ) - new_var = Variable( - BindCPointer(), - self.scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - body = [C_F_Pointer(new_var, f_arg)] - self.scope.insert_variable(f_arg) - return {"c_arg": BindCVariable(new_var, var), "f_arg": f_arg, "body": body} - - def _convert_native_array_buffer_argument(self, var, decision, func): - """Pass ordinary array-buffer fields through the native boundary.""" - if decision.codegen_action is CodegenAction.COPY_IN_OUT: - return self._convert_native_array_replacement_argument(var, decision, func) - return self._convert_native_array_storage_argument(var, decision, func) - - def _convert_native_array_replacement_argument(self, var, decision, func): - """Copy an immutable Python array into mutable native call storage.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - rank = var.rank - has_itemsize = self._is_character_array(var) - base_shape = [ - scope.get_temporary_variable( - NumpyInt64Type(), - name=f"{name}_base_shape_{index + 1}", - is_argument=True, - ) - for index in range(rank) - ] - itemsize_var = ( - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_itemsize", is_argument=True) - if has_itemsize - else None - ) - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - ) - input_var = var.clone( - scope.get_new_name(f"{name}_input"), - is_argument=False, - is_optional=False, - memory_handling=StorageMode.ALIAS.value, - new_class=Variable, - fortran_character_length=itemsize_var if has_itemsize else var.fortran_character_length, - ) - local_var = var.clone( - scope.get_expected_name(name), - is_argument=False, - is_optional=False, - memory_handling=decision.storage_mode.value, - shape=tuple(base_shape), - new_class=Variable, - ) - for item in (bind_var, input_var, local_var): - scope.insert_variable(item) - - prepare_local = [] - if decision.storage_mode is StorageMode.HEAP: - alloc_var = ( - local_var.clone(local_var.name, new_class=Variable, fortran_character_length=itemsize_var) - if has_itemsize - else local_var - ) - prepare_local.append(Allocate(alloc_var, shape=tuple(base_shape), status="unallocated")) - prepare_local.append(Assign(local_var, input_var)) - pointer_shape = base_shape[::-1] if var.order == "C" else base_shape - body = [ - If( - IfSection( - IsNot(bind_var, NIL), - [C_F_Pointer(bind_var, input_var, pointer_shape), *prepare_local], - ) - ) - ] - descriptor_offset = 2 if has_itemsize else 1 - c_arg_var = Variable( - BindCArrayType.get_new(rank, has_strides=False, has_itemsize=has_itemsize), - scope.get_new_name(), - is_argument=True, - shape=(convert_to_literal(rank + descriptor_offset),), - ) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(0)), bind_var) - if itemsize_var is not None: - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(1)), itemsize_var) - for index, shape_var in enumerate(base_shape): - scope.insert_symbolic_alias( - IndexedElement(c_arg_var, convert_to_literal(index + descriptor_offset)), - shape_var, - ) - return {"c_arg": BindCVariable(c_arg_var, var), "f_arg": local_var, "body": body} - - def _convert_native_array_storage_argument(self, var, decision, func): - """Convert array argument for the current wrapper.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - collisionless_name = scope.get_expected_name(name) - rank = var.rank - order = var.order - allows_strides = var.class_type.allows_strides - has_itemsize = self._is_character_array(var) - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - - if self._is_assumed_rank_array(var): - return self._convert_assumed_rank_array_argument(var, collisionless_name, bind_var) - - base_shape = [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_base_shape_{i + 1}", is_argument=True) - for i in range(rank) - ] - itemsize_var = ( - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_itemsize", is_argument=True) - if has_itemsize - else None - ) - arg_var = var.clone( - collisionless_name, - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - fortran_character_length=itemsize_var if has_itemsize else var.fortran_character_length, - ) - pointer_shape = base_shape[::-1] if order == "C" else base_shape - scope.insert_variable(arg_var) - scope.insert_variable(bind_var) - - stride = ( - [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_stride_{i + 1}", is_argument=True) - for i in range(rank) - ] - if allows_strides - else [] - ) - ubound = ( - [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_ubound_{i + 1}", is_argument=True) - for i in range(rank) - ] - if allows_strides - else [] - ) - - body = [C_F_Pointer(bind_var, arg_var, pointer_shape)] - descriptor_offset = 2 if has_itemsize else 1 - - c_arg_var = Variable( - BindCArrayType.get_new(rank, has_strides=allows_strides, has_itemsize=has_itemsize), - scope.get_new_name(), - is_argument=True, - shape=(convert_to_literal(rank * 3 + descriptor_offset if allows_strides else rank + descriptor_offset),), - ) - - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(0)), bind_var) - if itemsize_var is not None: - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(1)), itemsize_var) - for i, s in enumerate(base_shape): - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(i + descriptor_offset)), s) - if allows_strides: - for i, s in enumerate(ubound): - scope.insert_symbolic_alias( - IndexedElement(c_arg_var, convert_to_literal(i + rank + descriptor_offset)), s - ) - for i, s in enumerate(stride): - scope.insert_symbolic_alias( - IndexedElement(c_arg_var, convert_to_literal(i + 2 * rank + descriptor_offset)), - s, - ) - - start = convert_to_literal(1) # C_F_Pointer leads to default Fortran lbound - indexes = [ - Slice(start, Add(stop, convert_to_literal(1)), step) for step, stop in zip(stride, ubound, strict=False) - ] - f_arg = IndexedElement(arg_var, *indexes) - else: - f_arg = arg_var - - return {"c_arg": BindCVariable(c_arg_var, var), "f_arg": f_arg, "body": body} - - def _convert_raw_array_argument(self, var): - """Associate a caller-supplied raw data address with a Fortran array pointer.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - collisionless_name = scope.get_expected_name(name) - shape = self._raw_array_pointer_shape(var) - pointer_shape = shape[::-1] if var.order == "C" else shape - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - f_arg = var.clone( - collisionless_name, - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - ) - scope.insert_variable(bind_var) - scope.insert_variable(f_arg) - return { - "c_arg": BindCVariable(bind_var, var), - "f_arg": f_arg, - "body": [C_F_Pointer(bind_var, f_arg, pointer_shape)], - } - - @staticmethod - def _raw_array_pointer_shape(var): - """Return the fixed or visible extents needed to associate a raw pointer.""" - shape = tuple(var.alloc_shape or ()) - if len(shape) != var.rank or any(item is None for item in shape): - raise ValueError( - f"Raw array address argument {var.name!r} requires a fixed or visible extent for every axis" - ) - return list(shape) - - def _convert_assumed_rank_array_argument(self, var, collisionless_name, bind_var): - """Convert assumed rank array argument for the current wrapper.""" - name = var.name - scope = self.scope - rank = _MAX_SUPPORTED_ASSUMED_RANK - allows_strides = var.class_type.allows_strides - scope.insert_variable(bind_var) - rank_var = scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_rank", is_argument=True) - shape_vars = [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_base_shape_{i + 1}", is_argument=True) - for i in range(rank) - ] - ubound_vars = ( - [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_ubound_{i + 1}", is_argument=True) - for i in range(rank) - ] - if allows_strides - else [] - ) - stride_vars = ( - [ - scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_stride_{i + 1}", is_argument=True) - for i in range(rank) - ] - if allows_strides - else [] - ) - rank_vars = {} - for actual_rank in range(1, rank + 1): - rank_type = var.class_type.switch_rank(actual_rank, "F") - rank_vars[actual_rank] = Variable( - rank_type, - scope.get_new_name(f"{collisionless_name}_rank{actual_rank}"), - is_argument=False, - is_optional=False, - memory_handling="alias", - ) - scope.insert_variable(rank_vars[actual_rank]) - - descriptor_type = BindCArrayType.get_new(rank, has_strides=allows_strides, has_rank=True) - c_arg_var = Variable( - descriptor_type, - scope.get_new_name(), - is_argument=True, - shape=(convert_to_literal(len(descriptor_type)),), - ) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(0)), bind_var) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(1)), rank_var) - offset = 2 - for i, s in enumerate(shape_vars): - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(i + offset)), s) - if allows_strides: - for i, s in enumerate(ubound_vars): - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(i + rank + offset)), s) - for i, s in enumerate(stride_vars): - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(i + 2 * rank + offset)), s) - - placeholder = var.clone( - collisionless_name, - is_argument=False, - is_optional=False, - memory_handling="alias", - new_class=Variable, - ) - return { - "c_arg": BindCVariable(c_arg_var, var), - "f_arg": placeholder, - "body": [], - "assumed_rank": { - "allows_strides": allows_strides, - "bind_var": bind_var, - "rank_var": rank_var, - "rank_vars": rank_vars, - "shape_vars": shape_vars, - "stride_vars": stride_vars, - "ubound_vars": ubound_vars, - }, - } - - def _convert_raw_string_argument(self, var, decision): - """Associate a caller-supplied raw character address with fixed string storage.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - fixed_len = var.alloc_shape[0] - if fixed_len is None: - raise ValueError(f"Raw string address argument {name!r} requires a fixed String[n] length") - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - array_var = Variable( - NumpyNDArrayType.get_new(CharType(), 1, None), - scope.get_new_name(name), - memory_handling="alias", - ) - f_arg = var.clone( - scope.get_expected_name(name), - is_argument=False, - is_optional=False, - memory_handling="stack", - new_class=Variable, - ) - scope.insert_variable(bind_var) - scope.insert_variable(array_var) - scope.insert_variable(f_arg) - body = [ - C_F_Pointer(bind_var, array_var, (fixed_len,)), - Assign(f_arg, FortranTransfer(array_var, f_arg)), - ] - post_body = [] - if decision.mutates_native: - raw_slice = IndexedElement(array_var, Slice(None, Add(fixed_len, convert_to_literal(1)))) - post_body = [Assign(raw_slice, FortranTransfer(f_arg, raw_slice, fixed_len))] - return { - "c_arg": BindCVariable(bind_var, var), - "f_arg": f_arg, - "body": body, - "post_body": post_body, - } - - def _build_string_storage_argument(self, var, decision): - """Associate caller-provided rank-0 NumPy bytes storage with fixed string storage.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - fixed_len = var.alloc_shape[0] - if fixed_len is None: - raise ValueError(f"String storage argument {name!r} requires a fixed String[n] length") - bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - array_var = Variable( - NumpyNDArrayType.get_new(CharType(), 1, None), - scope.get_new_name(name), - memory_handling="alias", - ) - f_arg = var.clone( - scope.get_expected_name(name), - is_argument=False, - is_optional=False, - memory_handling="stack", - new_class=Variable, - ) - scope.insert_variable(bind_var) - scope.insert_variable(array_var) - scope.insert_variable(f_arg) - body = [ - C_F_Pointer(bind_var, array_var, (fixed_len,)), - Assign(f_arg, FortranTransfer(array_var, f_arg)), - ] - post_body = [] - if decision.mutates_native: - storage_slice = IndexedElement(array_var, Slice(None, Add(fixed_len, convert_to_literal(1)))) - post_body = [Assign(storage_slice, FortranTransfer(f_arg, storage_slice, fixed_len))] - return { - "c_arg": BindCVariable(bind_var, var), - "f_arg": f_arg, - "body": body, - "post_body": post_body, - } - - def _build_string_argument(self, var, decision, *, copy_back): - """Build string argument storage selected by strict policy dispatch.""" - name = var.name - scope = self.scope - scope.insert_symbol(name) - collisionless_name = scope.get_expected_name(name) - rank = var.rank - pointer_type = BindCPointer() if decision.mutates_native else FinalType.get_new(BindCPointer()) - bind_var = Variable( - pointer_type, - scope.get_new_name(f"bound_{name}"), - is_argument=True, - is_optional=False, - memory_handling="alias", - ) - shape_var = scope.get_temporary_variable(NumpyInt64Type(), name=f"{name}_size", is_argument=True) - array_var = Variable( - NumpyNDArrayType.get_new(CharType(), 1, None), - scope.get_new_name(name), - memory_handling="alias", - ) - scope.insert_variable(bind_var) - scope.insert_variable(array_var) - - fixed_len = var.alloc_shape[0] - has_fixed_length = fixed_len is not None - buffer_extent = Add(fixed_len if has_fixed_length else shape_var, convert_to_literal(1)) - pointer_extent = buffer_extent if copy_back else fixed_len if has_fixed_length else buffer_extent - if has_fixed_length: - fixed_var = var.clone( - collisionless_name, - is_argument=False, - is_optional=False, - memory_handling="stack", - new_class=Variable, - ) - scope.insert_variable(fixed_var) - body = [ - C_F_Pointer(bind_var, array_var, (pointer_extent,)), - Assign(fixed_var, FortranTransfer(array_var, fixed_var)), - ] - f_arg = fixed_var - else: - arg_var = var.clone( - collisionless_name, - is_argument=False, - is_optional=False, - memory_handling="stack", - shape=(shape_var,), - new_class=Variable, - ) - scope.insert_variable(arg_var) - body = [ - C_F_Pointer(bind_var, array_var, (buffer_extent,)), - Assign(arg_var, FortranTransfer(array_var, arg_var)), - ] - f_arg = arg_var - - post_body = [] - absent_body = [] - result_bind_var = None - if copy_back: - result_bind_var = Variable( - BindCPointer(), - scope.get_new_name(f"returned_{name}"), - memory_handling="alias", - ownership_decision=var.ownership_decision, - ) - payload_slice = IndexedElement(array_var, Slice(None, buffer_extent)) - post_body = [ - Assign(payload_slice, FortranTransfer(f_arg, payload_slice, shape_var)), - Assign(IndexedElement(array_var, buffer_extent), C_NULL_CHAR()), - Assign(result_bind_var, bind_var), - ] - if var.is_optional: - absent_body = [Assign(result_bind_var, NIL)] - - c_arg_var = Variable( - BindCArrayType.get_new(rank, has_strides=False), - scope.get_new_name(), - is_argument=True, - shape=(convert_to_literal(2),), - ) - - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(0)), bind_var) - scope.insert_symbolic_alias(IndexedElement(c_arg_var, convert_to_literal(1)), shape_var) - - return { - "c_arg": BindCVariable(c_arg_var, var), - "f_arg": f_arg, - "body": body, - "post_body": post_body, - "absent_body": absent_body, - "bind_var": bind_var, - "result_bind_var": result_bind_var, - } - - def _convert_result(self, orig_var, orig_func_scope): - """ - Get the code and variables necessary to translate a `Variable` to a C-compatible Variable. - - Get the code and variables necessary to translate a `Variable` which is returned - from a function to a `Variable` which can be called from C. A variable `local_var` is - created. This variable can be retrieved using its name which matches the name of `orig_var` - the variable that was originally returned. `local_var` should be used to retrieve the - result of the function call. It will generally be a clone of the return variable but some - properties (such as the memory handling) may be modified. A variable describing the - object which should be returned from the BindCFunctionDef may also be created if necessary. - Finally AST nodes are also created to describe any code which is needed to convert the - `local_var` to the returned variable. - - Parameters - ---------- - orig_var : Variable - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result: - - c_result: The Variable which should be used in a FunctionDefResult from the wrapped - function. - - body: The code which is needed to convert the local_var to the returned variable - saved in c_result. - - f_result: The Variable which should be used in a FunctionCall to collect the results - from the Fortran function. - """ - if orig_var.array_interop_policy is not None: - return self._ARRAY_INTEROP_POLICY_DISPATCHER.dispatch( - self, - orig_var, - orig_var.array_interop_policy, - "result", - orig_func_scope, - ) - return self._bridge_non_array_result(orig_var, orig_func_scope) - - def _bridge_data_buffer_result(self, subject, _policy, orig_func_scope): - """Convert an ordinary array result through the data-buffer ABI.""" - return self._bridge_non_array_result(subject, orig_func_scope) - - def _bridge_descriptor_result(self, subject, policy, orig_func_scope): - """Convert a native array handle result through descriptor ABI.""" - self._validate_descriptor_array_interop_policy(subject, policy) - return self._NATIVE_ARRAY_HANDLE_DISPATCHER.dispatch( - self, - subject, - subject.native_array_handle_policy, - orig_func_scope, - ) - - def _bridge_non_array_result(self, orig_var, orig_func_scope): - """Convert a function result without native descriptor-handle routing.""" - return self._RESULT_POLICY_DISPATCHER.dispatch(self, orig_var, orig_func_scope) - - def _convert_scalar_result(self, orig_var, decision, orig_func_scope): - """Convert scalar result for the current wrapper.""" - if decision.descriptor_boundary and decision.nullable: - return self._build_snapshot_copy_scalar_result( - orig_var, - self._nullable_scalar_status_func_for_storage(decision.boundary_storage_mode), - source_storage_mode=decision.boundary_storage_mode, - ) - name = orig_var.name - self.scope.insert_symbol(name) - local_var = orig_var.clone( - self.scope.get_expected_name(name), - new_class=Variable, - is_argument=False, - is_optional=False, - memory_handling=( - StorageMode.STACK.value - if decision.native_barrier_action is NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS - else orig_var.memory_handling - ), - ) - return { - "body": [], - "c_result": BindCVariable(local_var, orig_var), - "f_result": local_var, - } - - def _convert_snapshot_scalar_result(self, orig_var, decision, orig_func_scope): - """Copy a pointer scalar result into detached Python-visible storage.""" - return self._build_snapshot_copy_scalar_result( - orig_var, - self._nullable_scalar_status_func(decision), - source_storage_mode=decision.storage_mode, - ) - - def _convert_owned_custom_type_result(self, orig_var, decision, orig_func_scope): - """Convert an owned custom result through native value storage.""" - return self._convert_custom_type_result( - orig_var, - decision, - orig_func_scope, - decision.storage_mode, - borrowed=False, - ) - - def _convert_borrowed_custom_type_result(self, orig_var, decision, orig_func_scope): - """Convert a borrowed custom result through alias boundary storage.""" - return self._convert_custom_type_result( - orig_var, - decision, - orig_func_scope, - decision.boundary_storage_mode, - borrowed=True, - ) - - def _convert_custom_type_result( - self, - orig_var, - decision, - orig_func_scope, - local_storage_mode, - *, - borrowed, - ): - """Build the concrete custom result representation selected by policy.""" - name = orig_var.name - scope = self.scope - scope.insert_symbol(name) - memory_handling = local_storage_mode.value - local_var = orig_var.clone( - scope.get_expected_name(name), - new_class=Variable, - memory_handling=memory_handling, - is_argument=False, - is_optional=False, - ) - # Allocatable is not returned so it must appear in local scope - scope.insert_variable(local_var, name) - - # Create the C-compatible data pointer - bind_var = Variable(BindCPointer(), scope.get_new_name("bound_" + name), memory_handling="alias") - - if borrowed: - ptr_var = orig_var - body = [CLocFunc(ptr_var, bind_var)] - else: - # Create an array variable which can be passed to CLocFunc - ptr_var = Variable( - orig_var.class_type, - scope.get_new_name(name + "_ptr"), - memory_handling="alias", - ) - scope.insert_variable(ptr_var) - alloc = Allocate(ptr_var, shape=None, status="unallocated") - copy = Assign(ptr_var, local_var) - cloc = CLocFunc(ptr_var, bind_var) - body = [alloc, copy, cloc] - - return { - "body": body, - "c_result": BindCVariable(bind_var, orig_var), - "f_result": local_var, - } - - def _convert_array_result(self, orig_var, decision, orig_func_scope): - """Convert array result for the current wrapper.""" - name = orig_var.name - scope = self.scope - scope.insert_symbol(name) - memory_handling = decision.boundary_storage_mode.value - - shape = orig_var.shape if memory_handling == "stack" else None - - # Allocatable is not returned so it must appear in local scope - local_var = orig_var.clone( - scope.get_expected_name(name), - new_class=Variable, - memory_handling=memory_handling, - shape=shape, - is_argument=False, - is_optional=False, - ) - scope.insert_variable(local_var, name) - - result = self._NDARRAY_RESULT_DISPATCHER.dispatch( - self, - orig_var, - name, - local_var, - ) - - result["f_result"] = local_var - - return result - - def _convert_string_result(self, orig_var, decision, orig_func_scope): - """Convert string result for the current wrapper.""" - name = orig_var.name - scope = self.scope - scope.insert_symbol(name) - memory_handling = decision.boundary_storage_mode.value - - # Allocatable is not returned so it must appear in local scope - local_var = orig_var.clone( - scope.get_expected_name(name), - new_class=Variable, - memory_handling=memory_handling, - is_argument=False, - is_optional=False, - ) - scope.insert_variable(local_var, name) - - # Create the C-compatible data pointer - bind_var = Variable(BindCPointer(), scope.get_new_name("bound_" + name), memory_handling="alias") - - shape_var = Variable(NumpyInt32Type(), scope.get_new_name(f"{name}_len")) - scope.insert_variable(shape_var) - - # Create an array variable which can be passed to CLocFunc - ptr_var = Variable( - NumpyNDArrayType.get_new(CharType(), 1, None), - scope.get_new_name(name + "_ptr"), - memory_handling="alias", - ) - elem_var = Variable(CharType(), scope.get_new_name(name + "_elem")) - scope.insert_variable(ptr_var) - scope.insert_variable(elem_var) - - # Define the additional steps necessary to define and fill ptr_var - copy_body = [ - Assign(shape_var, Add(ArraySize(local_var), convert_to_literal(1))), - Assign(bind_var, c_malloc(Mul(BindCSizeOf(elem_var), shape_var))), - If( - IfSection( - IsNot(bind_var, NIL), - [ - C_F_Pointer(bind_var, ptr_var, [shape_var]), - Assign(ptr_var, FortranTransfer(local_var, ptr_var, shape_var)), - Assign(IndexedElement(ptr_var, shape_var), C_NULL_CHAR()), - ], - ) - ), - ] - if decision.nullable and decision.descriptor_boundary: - body = [ - If( - IfSection(self._nullable_scalar_status_func(decision)(local_var), copy_body), - IfSection(convert_to_literal(True), [Assign(bind_var, NIL)]), - ) - ] - else: - body = copy_body - - return { - "c_result": BindCVariable(bind_var, orig_var), - "body": body, - "f_array": ptr_var, - "f_result": local_var, - } - - # ------------------------------------------------------------------ - # Node builders - # ------------------------------------------------------------------ - - def _build_snapshot_copy_scalar_result( - self, - orig_var, - status_func=None, - source_expr=None, - source_storage_mode=None, - ): - """Build snapshot copy scalar result nodes.""" - name = orig_var.name - scope = self.scope - local_storage = source_storage_mode or StorageMode.ALIAS - if source_expr is None: - scope.insert_symbol(name) - source_expr = orig_var.clone( - scope.get_expected_name(name), - new_class=Variable, - is_argument=False, - memory_handling=local_storage.value, - is_optional=False, - ) - scope.insert_variable(source_expr) - bind_var = Variable(BindCPointer(), scope.get_new_name(f"bound_{name}"), memory_handling="alias") - copy_var = orig_var.clone( - scope.get_new_name(f"{name}_copy"), - new_class=Variable, - is_argument=False, - memory_handling="alias", - is_optional=False, - ) - size_var = orig_var.clone( - scope.get_new_name(f"{name}_element"), - new_class=Variable, - is_argument=False, - memory_handling="stack", - is_optional=False, - ) - for variable in (copy_var, size_var): - scope.insert_variable(variable) - status = status_func or ArrayAssociated - body = [self._nullable_scalar_snapshot_if(source_expr, status, bind_var, copy_var, size_var)] - return { - "body": body, - "c_result": BindCVariable(bind_var, orig_var), - "f_result": source_expr, - "bind_var": bind_var, - "copy_var": copy_var, - "size_var": size_var, - } - - @staticmethod - def _nullable_scalar_snapshot_if(source_expr, status_func, bind_var, copy_var, size_var): - """Build a nullable scalar snapshot branch from a completed descriptor policy.""" - copy_body = [ - Assign(bind_var, c_malloc(BindCSizeOf(size_var))), - If( - IfSection( - IsNot(bind_var, NIL), - [C_F_Pointer(bind_var, copy_var), Assign(copy_var, source_expr)], - ) - ), - ] - return If( - IfSection(status_func(source_expr), copy_body), - IfSection(convert_to_literal(True), [Assign(bind_var, NIL)]), - ) - - @staticmethod - def _nullable_scalar_status_func(decision): - """Return the native descriptor presence check selected by completed policy.""" - return FortranToCBridgeGenerator._nullable_scalar_status_func_for_storage(decision.storage_mode) - - @staticmethod - def _nullable_scalar_status_func_for_storage(storage_mode): - """Return the presence check for one completed descriptor storage mode.""" - if storage_mode is StorageMode.HEAP: - return ArrayAllocated - if storage_mode is StorageMode.ALIAS: - return ArrayAssociated - value = getattr(storage_mode, "value", storage_mode) - raise ValueError(f"Nullable scalar snapshot requires heap or alias storage, got {value}") - - def _build_snapshot_copy_array_result(self, orig_var, _decision, name, local_var): - """Build snapshot copy array result nodes.""" - return self._get_pointer_snapshot_bind_c_array(name, orig_var, local_var) - - def _build_borrowed_array_result(self, orig_var, _decision, name, local_var): - """Build borrowed array result nodes.""" - return self._get_bind_c_array(name, orig_var, local_var.shape, local_var, source_var=local_var) - - def _build_copy_return_array_result(self, orig_var, decision, name, local_var): - """Dispatch copy-return emission from completed boundary storage.""" - try: - handler_name = self._COPY_RETURN_ARRAY_BY_STORAGE[decision.boundary_storage_mode] - except KeyError: - raise ValueError( - f"No array copy-return handler for completed storage {decision.boundary_storage_mode.value!r}" - ) from None - return getattr(self, handler_name)(orig_var, decision, name, local_var) - - def _build_stack_copy_return_array_result(self, orig_var, _decision, name, local_var): - """Copy a fixed-shape native result into Python-owned storage.""" - result = self._get_bind_c_array(name, orig_var, local_var.shape, source_var=local_var) - result["body"].append(self._array_result_copy_section(result, local_var)) - return result - - def _build_heap_copy_return_array_result(self, orig_var, _decision, name, local_var): - """Copy an allocated native result and release its native storage.""" - copy_shape = tuple(ArrayShapeElement(local_var, convert_to_literal(index)) for index in range(local_var.rank)) - result = self._get_bind_c_array(name, orig_var, copy_shape, source_var=local_var) - result["body"].append(self._array_result_copy_section(result, local_var)) - allocated_body = [*result["body"], Deallocate(local_var)] - unallocated_body = [ - Assign(result["bind_var"], NIL), - *[Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) for shape_var in result["shape_vars"]], - ] - result["body"] = [ - If( - IfSection(ArrayAllocated(local_var), allocated_body), - IfSection(convert_to_literal(True), unallocated_body), - ) - ] - return result - - def _build_scalar_replacement_result(self, orig_var, decision, generated_arg): - """Return the mutable native scalar temporary as a replacement value.""" - local_var = generated_arg["f_arg"].value - if decision.descriptor_boundary and decision.nullable: - return self._build_snapshot_copy_scalar_result( - orig_var, - self._nullable_scalar_status_func_for_storage(decision.boundary_storage_mode), - source_expr=local_var, - source_storage_mode=decision.boundary_storage_mode, - ) - # The copy-in temporary becomes the Bind(C) function result. It must - # therefore be declared by the result signature, not a second time as - # an ordinary function local. - self.scope.remove_variable(local_var, remove_symbol=False) - return { - "c_result": BindCVariable(local_var, orig_var), - "body": [], - "f_result": local_var, - } - - def _build_array_replacement_result(self, orig_var, decision, generated_arg): - """Copy a mutable native array temporary into Python-owned result storage.""" - local_var = generated_arg["f_arg"].value - result_shape = ( - tuple(ArrayShapeElement(local_var, convert_to_literal(index)) for index in range(local_var.rank)) - if decision.storage_mode is StorageMode.HEAP - else local_var.shape - ) - result = self._get_bind_c_array( - orig_var.name, - orig_var, - result_shape, - source_var=local_var, - ) - result["body"].append(self._array_result_copy_section(result, local_var)) - if decision.storage_mode is StorageMode.HEAP: - allocated_body = [*result["body"], Deallocate(local_var)] - unallocated_body = [ - Assign(result["bind_var"], NIL), - *[ - Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) - for shape_var in result["shape_vars"] - ], - ] - result["body"] = [ - If( - IfSection(ArrayAllocated(local_var), allocated_body), - IfSection(convert_to_literal(True), unallocated_body), - ) - ] - result["f_result"] = local_var - return result - - @staticmethod - def _build_string_replacement_result(orig_var, decision, generated_arg): - """Build string replacement result nodes.""" - return { - "c_result": BindCVariable(generated_arg["result_bind_var"], orig_var), - "body": [], - "f_result": generated_arg["f_arg"].value, - } - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _array_result_copy_section(self, result, source_var): - """Copy a native array into the returned C-compatible result buffer.""" - if self._is_character_array(source_var): - copy_expr = FortranTransfer(source_var, result["f_array"], result["byte_count"]) - else: - copy_expr = source_var - return If(IfSection(IsNot(result["bind_var"], NIL), [Assign(result["f_array"], copy_expr)])) - - @staticmethod - def _is_character_array(var): - """Return whether ``var`` stores fixed-width character array elements.""" - return isinstance(var.class_type, NumpyNDArrayType) and isinstance(var.dtype, CharType) - - @staticmethod - def _has_optional_arguments(func: FunctionDef) -> bool: - """Return whether has optional arguments.""" - return any(getattr(argument.var, "is_optional", False) for argument in func.arguments) - - def _get_function_def_body(self, func, generated_args, results, handled=()): - """ - Get the body of the bind c function definition. - - Get the body of the bind c function definition by inserting if blocks - to check the presence of optional variables. Once we have ascertained - the presence of the variables the original function is called. This - code slices array variables to ensure the correct step. - - Parameters - ---------- - func : FunctionDef - The function which should be called. - - generated_args : list[dict] - A list containing the dictionaries returned by _convert_argument. - - results : list of Variables - The Variables where the result of the function call will be saved. - - handled : tuple - A list of all variables which have been handled (checked to see if they - are present). - - Returns - ------- - list - A list of codegen nodes describing the body of the function. - """ - optional_block = self._optional_argument_presence_block(func, generated_args, results, handled) - if optional_block is not None: - return optional_block - return self._get_required_function_def_body(func, generated_args, results) - - def _optional_argument_presence_block(self, func, generated_args, results, handled): - """Build the next optional-argument presence branch, if any remains.""" - next_optional_arg = self._next_optional_generated_arg(generated_args, handled) - if next_optional_arg is None: - return None - - args = generated_args.copy() - optional_var = self._optional_presence_var(next_optional_arg) - handled += (next_optional_arg,) - true_section = IfSection( - IsNot(optional_var, NIL), - self._get_function_def_body(func, args, results, handled), - ) - args.remove(next_optional_arg) - false_section = IfSection( - convert_to_literal(True), - [ - *next_optional_arg.get("absent_body", ()), - *self._get_function_def_body(func, args, results, handled), - ], - ) - return [If(true_section, false_section)] - - @staticmethod - def _next_optional_generated_arg(generated_args, handled): - """Return the next generated optional argument that has not been branched.""" - for generated_arg in generated_args: - if generated_arg["c_arg"] is None or generated_arg in handled: - continue - original_var = getattr(generated_arg["c_arg"].var, "original_var", generated_arg["c_arg"].var) - if getattr(original_var, "is_optional", False): - return generated_arg - return None - - def _optional_presence_var(self, generated_arg): - """Return the C-side value used to test optional argument presence.""" - optional_var = generated_arg.get("optional_presence_var") or generated_arg["c_arg"].var - optional_var = getattr(optional_var, "new_var", optional_var) - if isinstance(optional_var.class_type, BindCArrayType): - return self.scope.collect_tuple_element(IndexedElement(optional_var, convert_to_literal(0))) - return optional_var - - def _get_required_function_def_body(self, func, generated_args, results): - """Build the body once all optional arguments have been resolved.""" - args = [a["f_arg"] for a in generated_args] - body = [line for a in generated_args for line in a["body"]] - post_body = [line for a in generated_args for line in a.get("post_body", ())] - - selected, native_name, args = self._selected_native_call_args(func, args) - if re.sub(r"\s+", "", native_name).casefold() == "assignment(=)": - lhs, rhs = func.native_arguments(selected, args) - return [*body, Assign(lhs.value, rhs.value), *post_body] - - if getattr(func, "is_external", False): - args = self._positional_native_arguments(args) - - selected_func = selected or func - if len(results) == 1 and self._uses_allocatable_function_result_helper(selected_func, results[0]): - helper = self._allocatable_function_result_helper(results[0]) - self._additional_functions.append(helper) - return [*body, helper(func(*args), results[0]), *post_body] - - if any(arg.get("assumed_rank") for arg in generated_args): - return [*body, *self._assumed_rank_dispatch(func, generated_args, results), *post_body] - - return [*body, *self._native_call_body(func, args, results), *post_body] - - @staticmethod - def _selected_native_call_args(func, args): - """Resolve overload calls and normalize their native argument order.""" - if not isinstance(func, FunctionOverloadSet): - return None, "", args - selected = func.point(args) - native_name = func.native_name_for(selected) - return selected, native_name, FortranToCBridgeGenerator._positional_native_arguments(args) - - @staticmethod - def _positional_native_arguments(args): - """Return native call arguments without generated bridge keywords.""" - return [FunctionCallArgument(arg.value) for arg in args] - - @staticmethod - def _native_call_body(func, args, results): - """Handle native call body for the current generation context.""" - if len(results) == 1: - res = results[0] - func_call = AliasAssign(res, func(*args)) if res.is_alias else Assign(res, func(*args)) - else: - func_call = Assign(results, func(*args)) - return [func_call] - - def _assumed_rank_dispatch(self, func, generated_args, results): - """Handle assumed rank dispatch for the current generation context.""" - dispatch_args = [arg for arg in generated_args if arg.get("assumed_rank")] - return self._assumed_rank_dispatch_level(func, generated_args, results, dispatch_args, {}, 0) - - def _assumed_rank_dispatch_level(self, func, generated_args, results, dispatch_args, replacements, index): - """Handle assumed rank dispatch level for the current generation context.""" - if index == len(dispatch_args): - args = [ - self._replacement_function_argument(arg["f_arg"], replacements[arg["f_arg"].value]) - if arg.get("assumed_rank") - else arg["f_arg"] - for arg in generated_args - ] - return self._native_call_body(func, args, results) - - dispatch_arg = dispatch_args[index] - info = dispatch_arg["assumed_rank"] - sections = [] - for rank in range(1, _MAX_SUPPORTED_ASSUMED_RANK + 1): - rank_var = info["rank_vars"][rank] - f_arg = self._assumed_rank_argument_view(info, rank_var, rank) - replacements[dispatch_arg["f_arg"].value] = f_arg - nested_body = self._assumed_rank_dispatch_level( - func, - generated_args, - results, - dispatch_args, - replacements, - index + 1, - ) - del replacements[dispatch_arg["f_arg"].value] - sections.append( - CaseSection( - convert_to_literal(rank, dtype=NumpyInt64Type()), - [ - C_F_Pointer(info["bind_var"], rank_var, info["shape_vars"][:rank]), - *nested_body, - ], - ) - ) - sections.append(CaseSection(None, [Return(None)])) - return [SelectCase(info["rank_var"], *sections)] - - @staticmethod - def _replacement_function_argument(original, value): - """Handle replacement function argument for the current generation context.""" - return FunctionCallArgument(value, keyword=original.keyword) - - @staticmethod - def _assumed_rank_argument_view(info, rank_var, rank): - """Handle assumed rank argument view for the current generation context.""" - if not info["allows_strides"]: - return rank_var - start = convert_to_literal(1) - indexes = [ - Slice(start, Add(stop, convert_to_literal(1)), step) - for step, stop in zip(info["stride_vars"][:rank], info["ubound_vars"][:rank], strict=False) - ] - return IndexedElement(rank_var, *indexes) - - @classmethod - def _uses_allocatable_function_result_helper(cls, func, result): - """Return whether uses allocatable function result helper.""" - func_result = getattr(getattr(func, "results", None), "var", NIL) - result_decision = ownership_decision_for_codegen_variable(result) - func_result_decision = ownership_decision_for_codegen_variable(func_result) if func_result is not NIL else None - if ( - result_decision.kind in {ObjectKind.SCALAR, ObjectKind.STRING} - and result_decision.descriptor_boundary - and result_decision.nullable - and result_decision.storage_mode is StorageMode.HEAP - and func_result_decision is not None - ): - return ( - func_result_decision.kind is result_decision.kind - and func_result_decision.descriptor_boundary - and func_result_decision.nullable - and func_result_decision.storage_mode is StorageMode.HEAP - ) - return ( - result.is_ndarray - and cls._is_allocatable_copy_return_result(result) - and func_result is not NIL - and getattr(func_result, "is_ndarray", False) - and cls._is_allocatable_copy_return_result(func_result) - ) - - def _allocatable_function_result_helper(self, result): - """Handle allocatable function result helper for the current generation context.""" - helper_name = self.scope.get_new_name(f"x2py_collect_{result.name}") - helper_scope = self.scope.new_child_scope(helper_name, "function") - storage_mode = ownership_decision_for_codegen_variable(result).storage_mode.value - value = result.clone( - helper_scope.get_new_name(f"{result.name}_value"), - new_class=Variable, - memory_handling=storage_mode, - is_argument=False, - is_optional=False, - ) - target = result.clone( - helper_scope.get_new_name(f"{result.name}_target"), - new_class=Variable, - memory_handling=storage_mode, - is_argument=False, - is_optional=False, - ) - value_arg = FunctionDefArgument(value) - value_arg.make_const() - target_arg = FunctionDefArgument(target) - return FunctionDef( - helper_name, - [value_arg, target_arg], - [If(IfSection(ArrayAllocated(value), [Assign(target, value)]))], - scope=helper_scope, - ) - - def _direct_bind_c_function(self, expr): - """Handle direct bind c function for the current generation context.""" - external_name = expr.bind_c_external_name - func = BindCFunctionDef( - external_name, - expr.arguments, - [], - expr.results, - is_header=True, - scope=expr.scope, - original_function=expr, - docstring=expr.docstring, - result_pointer_map=expr.result_pointer_map, - bind_c_external_name=external_name, - ) - self.scope.insert_symbol(external_name, object_type="function") - self.scope.insert_function(func, external_name) - return func - - @classmethod - def _can_call_existing_bind_c_directly(cls, expr): - """Return whether can call existing bind c directly.""" - if not expr.bind_c_external_name or expr.is_private or not expr.is_semantic: - return False - if expr.is_external or cls._has_optional_arguments(expr): - return False - if any(argument.bound_argument for argument in expr.arguments): - return False - if not cls._is_direct_bind_c_result(expr.results.var): - return False - return all(cls._is_direct_bind_c_argument(argument.var) for argument in expr.arguments) - - @staticmethod - def _is_direct_bind_c_result(var): - """Return whether is direct bind c result.""" - if var is NIL: - return True - return var.rank == 0 and isinstance(var.class_type, FixedSizeNumericType) - - @staticmethod - def _is_direct_bind_c_argument(var): - """Return whether is direct bind c argument.""" - return ( - var.rank == 0 - and var.memory_handling == "stack" - and getattr(var, "passes_by_value", False) - and isinstance(var.class_type, FixedSizeNumericType) - ) - - @staticmethod - def _is_allocatable_copy_return_result(var): - """Return whether is allocatable copy return result.""" - decision = ownership_decision_for_codegen_variable(var) - return FortranToCBridgeGenerator._ALLOCATABLE_RESULT_HELPER_DISPATCHER.dispatch_decision( - FortranToCBridgeGenerator, var, decision - ) - - @staticmethod - def _uses_heap_allocatable_result_helper(var, decision): - """Return whether a copy-return array result needs allocatable helper collection.""" - return decision.storage_mode is StorageMode.HEAP - - @staticmethod - def _skips_allocatable_result_helper(_var, _decision): - """Return whether a non-copy-return array result skips allocatable helper collection.""" - return False - - @staticmethod - def _is_assumed_rank_array(var): - """Return whether is assumed rank array.""" - return bool(getattr(var, "assumed_rank", False) and var.is_ndarray) - - def _pack_function_results(self, result_infos): - """Handle pack function results for the current generation context.""" - result_type = BindCResultTupleType.get_new(tuple(info["c_result"].class_type for info in result_infos)) - result_var = Variable( - result_type, - self.scope.get_new_name("results"), - shape=(convert_to_literal(len(result_infos)),), - is_temp=True, - ) - for index, info in enumerate(result_infos): - self.scope.insert_symbolic_alias(IndexedElement(result_var, convert_to_literal(index)), info["c_result"]) - return result_var - - @staticmethod - def _module_variable_imports(expr): - """Handle module variable imports for the current generation context.""" - mod = get_enclosing_module(expr) - assert mod is not None - if mod.imports: - return [] - return [Import(mod.name, AsName(expr, expr.name), mod=mod)] - - def _generated_module_function_name(self, public_name: str): - """Handle generated module function name for the current generation context.""" - return self.scope.get_new_public_name( - public_name, - object_type="function", - owner=f"module variable accessor {public_name}", - ) - - def _scalar_module_variable(self, expr, _decision): - """Handle scalar module variable for the current generation context.""" - getter = self._scalar_module_getter(expr) - setter = ( - self._scalar_module_setter(expr) - if expr.setter_ownership_decision.setter_action is SetterAction.WRITE_THROUGH - else None - ) - return expr.clone( - expr.name, - new_class=BindCAccessorModuleVariable, - getter_function=getter, - setter_function=setter, - ) - - @staticmethod - def _literal_module_constant(expr, _decision): - """Expose one literal scalar or string constant without native storage.""" - return expr.clone(expr.name, new_class=BindCModuleConstant) - - def _copied_derived_module_constant(self, expr, decision): - """Expose one derived constant through a wrapper-owned value copy.""" - if decision.transfer is not TransferMode.WRAPPER_INSTANCE: - raise ValueError(f"Derived module constant {expr.name!r} is missing completed copy policy") - if expr.setter_ownership_decision.setter_action is not SetterAction.OMIT: - raise ValueError(f"Derived module constant {expr.name!r} unexpectedly exposes a setter") - return expr.clone( - expr.name, - new_class=BindCAccessorModuleVariable, - getter_function=self._derived_module_copy_getter(expr), - setter_function=None, - ) - - def _derived_module_variable(self, expr, decision): - """Expose one addressable native module object through a borrowed wrapper.""" - if decision.boundary_storage_mode is not StorageMode.ALIAS: - raise ValueError(f"Derived module variable {expr.name!r} is missing completed Aliased storage") - if expr.setter_ownership_decision.setter_action is not SetterAction.REJECT_REPLACEMENT: - raise ValueError(f"Derived module variable {expr.name!r} unexpectedly exposes replacement") - return expr.clone( - expr.name, - new_class=BindCAccessorModuleVariable, - getter_function=self._derived_module_getter(expr), - setter_function=None, - ) - - def _derived_module_copy_getter(self, expr): - """Return a pointer to an owned copy of a derived module value.""" - getter_policy = expr.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Module variable {expr.name!r} is missing completed getter policy") - value_type = expr.class_type.underlying_type if isinstance(expr.class_type, FinalType) else expr.class_type - scope = self.scope - public_name = f"get_{expr.name}" - original_name = self._generated_module_function_name(public_name) - func_name = scope.get_new_name("bind_c_" + public_name.lower()) - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - source_value = expr.clone( - expr.name, - class_type=value_type, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - local_var = expr.clone( - f"{expr.name}_snapshot", - class_type=value_type, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - pointer_var = Variable( - value_type, - func_scope.get_new_name(f"{expr.name}_snapshot_ptr"), - memory_handling="alias", - ) - bind_var = Variable(BindCPointer(), func_scope.get_new_name(f"bound_{expr.name}"), memory_handling="alias") - for variable in (local_var, pointer_var): - func_scope.insert_variable(variable, name=str(variable.name)) - func_scope.imports["variables"][expr.name] = expr - self.exit_scope() - - original_result = expr.clone( - f"{expr.name}_value", - class_type=value_type, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - original_function = FunctionDef( - original_name, - [], - [], - FunctionDefResult(original_result), - scope=scope, - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_MODULE_VARIABLE_NAME_METADATA: expr.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: "get", - }, - ) - return BindCFunctionDef( - func_name, - [], - [ - Assign(local_var, source_value), - Allocate(pointer_var, shape=None, status="unallocated"), - Assign(pointer_var, local_var), - CLocFunc(pointer_var, bind_var), - ], - FunctionDefResult(BindCVariable(bind_var, local_var)), - imports=self._module_variable_imports(expr), - scope=func_scope, - original_function=original_function, - ) - - def _derived_module_getter(self, expr): - """Return the C address of an Aliased derived module object.""" - getter_policy = expr.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Module variable {expr.name!r} is missing completed getter policy") - scope = self.scope - public_name = f"get_{expr.name}" - original_name = self._generated_module_function_name(public_name) - func_name = scope.get_new_name("bind_c_" + public_name.lower()) - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - getter_value = expr.clone( - expr.name, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.boundary_storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - bind_var = Variable(BindCPointer(), func_scope.get_new_name(f"bound_{expr.name}"), memory_handling="alias") - func_scope.imports["variables"][expr.name] = expr - self.exit_scope() - - original_result = expr.clone( - f"{expr.name}_value", - is_argument=False, - is_optional=False, - memory_handling=getter_policy.boundary_storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - original_function = FunctionDef( - original_name, - [], - [], - FunctionDefResult(original_result), - scope=scope, - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_MODULE_VARIABLE_NAME_METADATA: expr.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: "get", - }, - ) - return BindCFunctionDef( - func_name, - [], - [CLocFunc(getter_value, bind_var)], - FunctionDefResult(BindCVariable(bind_var, getter_value)), - imports=self._module_variable_imports(expr), - scope=func_scope, - original_function=original_function, - ) - - def _scalar_module_getter(self, expr): - """Handle scalar module getter for the current generation context.""" - getter_policy = expr.getter_ownership_decision - if getter_policy is None: - raise ValueError(f"Module variable {expr.name!r} is missing completed getter policy") - scope = self.scope - public_name = f"get_{expr.name}" - original_name = self._generated_module_function_name(public_name) - func_name = scope.get_new_name("bind_c_" + public_name.lower()) - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - func_scope.imports["variables"][expr.name] = expr - if getter_policy.codegen_action is CodegenAction.SNAPSHOT_COPY: - source_value = expr.clone( - expr.name, - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - result_info = self._build_snapshot_copy_scalar_result( - expr, - self._nullable_scalar_status_func(getter_policy), - source_expr=source_value, - ) - body = result_info["body"] - result = result_info["c_result"] - else: - result = expr.clone( - func_scope.get_new_name(f"{expr.name}_value"), - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - func_scope.insert_variable(result) - body = [Assign(result, expr)] - self.exit_scope() - original_result = expr.clone( - f"{expr.name}_value", - is_argument=False, - is_optional=False, - memory_handling=getter_policy.storage_mode.value, - ownership_decision=getter_policy, - new_class=Variable, - ) - original_function = FunctionDef( - original_name, - [], - [], - FunctionDefResult(original_result), - scope=scope, - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_MODULE_VARIABLE_NAME_METADATA: expr.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: "get", - }, - ) - return BindCFunctionDef( - func_name, - [], - body, - FunctionDefResult(result), - imports=self._module_variable_imports(expr), - scope=func_scope, - original_function=original_function, - ) - - def _scalar_module_setter(self, expr): - """Handle scalar module setter for the current generation context.""" - setter_policy = expr.setter_ownership_decision - if setter_policy is None: - raise ValueError(f"Module variable {expr.name!r} is missing completed setter policy") - scope = self.scope - public_name = f"set_{expr.name}" - original_name = self._generated_module_function_name(public_name) - func_name = scope.get_new_name("bind_c_" + public_name.lower()) - func_scope = scope.new_child_scope(func_name, "function") - self.scope = func_scope - value = expr.clone( - func_scope.get_new_name("value"), - is_argument=True, - is_optional=False, - memory_handling=setter_policy.storage_mode.value, - ownership_decision=setter_policy, - new_class=Variable, - ) - func_scope.insert_variable(value) - func_scope.imports["variables"][expr.name] = expr - body = [Assign(expr, value)] - self.exit_scope() - original_value = expr.clone( - "value", - is_argument=True, - is_optional=False, - memory_handling=setter_policy.storage_mode.value, - ownership_decision=setter_policy, - new_class=Variable, - ) - original_function = FunctionDef( - original_name, - [FunctionDefArgument(original_value)], - [], - FunctionDefResult(NIL), - scope=scope, - decorators={ - RUNTIME_HOLD_GIL_METADATA: True, - INTERNAL_MODULE_VARIABLE_NAME_METADATA: expr.name, - INTERNAL_MODULE_VARIABLE_ACCESS_METADATA: "set", - }, - ) - return BindCFunctionDef( - func_name, - [FunctionDefArgument(value)], - body, - FunctionDefResult(NIL), - imports=self._module_variable_imports(expr), - scope=func_scope, - original_function=original_function, - ) - - def _get_pointer_snapshot_bind_c_array(self, name, orig_var, pointer_var): - """Return a snapshot-copy bind C array for descriptor-backed storage.""" - dtype = orig_var.dtype - rank = orig_var.rank - order = orig_var.order - scope = self.scope - - bind_var = Variable(BindCPointer(), scope.get_new_name("bound_" + name), memory_handling="alias") - shape_vars = [Variable(NumpyInt32Type(), scope.get_new_name(f"{name}_shape_{i + 1}")) for i in range(rank)] - - numpy_dtype = numpy_precision_map[(dtype.primitive_type, dtype.precision)] - ptr_var = Variable( - NumpyNDArrayType.get_new(numpy_dtype, rank, order), - scope.get_new_name(name + "_ptr"), - memory_handling="alias", - ) - elem_var = Variable(dtype, scope.get_new_name(name + "_elem")) - scope.insert_variable(ptr_var) - scope.insert_variable(elem_var) - - shape_assignments = [ - Assign( - shape_var, - cast_to(ArrayShapeElement(pointer_var, convert_to_literal(index)), NumpyInt32Type()), - ) - for index, shape_var in enumerate(shape_vars) - ] - size = reduce(Mul, [BindCSizeOf(elem_var), *shape_vars]) - copy_body = [ - *shape_assignments, - Assign(bind_var, c_malloc(size)), - If( - IfSection( - IsNot(bind_var, NIL), - [ - C_F_Pointer(bind_var, ptr_var, shape_vars if order == "F" else shape_vars[::-1]), - Assign(ptr_var, pointer_var), - ], - ) - ), - ] - unassociated_body = [ - Assign(bind_var, NIL), - *[Assign(shape_var, convert_to_literal(0, dtype=NumpyInt32Type())) for shape_var in shape_vars], - ] - body = [ - If( - IfSection(ArrayAssociated(pointer_var), copy_body), - IfSection(convert_to_literal(True), unassociated_body), - ) - ] - - result_var = Variable( - BindCArrayType.get_new(rank, has_strides=False), - scope.get_new_name(), - shape=(rank + 1,), - ) - c_result = BindCVariable(result_var, orig_var) - for descriptor in (result_var, c_result): - scope.insert_symbolic_alias(IndexedElement(descriptor, convert_to_literal(0)), bind_var) - for index, shape_var in enumerate(shape_vars): - scope.insert_symbolic_alias(IndexedElement(descriptor, convert_to_literal(index + 1)), shape_var) - - return { - "c_result": c_result, - "body": body, - "f_array": ptr_var, - "bind_var": bind_var, - "shape_vars": shape_vars, - } - - def _get_bind_c_array(self, name, orig_var, shape, pointer_target=False, source_var=None): - """ - Get all the objects necessary to return an array from the BindCFunctionDef. - - In the case of an array, C cannot represent the array natively. Rather it is - stored in a pointer. This function therefore creates a variable to represent - that pointer. Additionally information about the shape and strides of the array - are necessary. The assignment expressions which define the shapes and strides - are then stored in `body` along with the allocation of the pointer. The - Fortran-accessible array is returned so that it can be filled differently - depending on what type is described by the array (e.g. if the array describes - an array a simple copy is required, but if the array describes a set then the - elements need to be added one by one. - - Parameters - ---------- - name : str - The stem of the names of the objects that should be created. - - orig_var : Variable - An object representing the variable or an element of the variable from the - FunctionDefResult being wrapped. This is used to obtain the dtype, rank - and order of the array that should be created. - - shape : tuple[model object] - A tuple describing the shape that the array should be allocated to. - - pointer_target : bool, default=False - Indicates if the data in orig_var is a target of the pointer that will be - created. - - Returns - ------- - dict - A dictionary describing the objects necessary to collect the result: - - c_result: The Variable which should be used in a FunctionDefResult from the wrapped - function. - - body: The code which is needed to convert the local_var to the returned variable - saved in c_result. - - f_array: The Fortran-accessible array that will be returned. This is where the data - should be copied to. - """ - rank = orig_var.rank - scope = self.scope - has_itemsize = self._is_character_array(orig_var) - # Create the C-compatible data pointer - bind_var = Variable(BindCPointer(), scope.get_new_name("bound_" + name), memory_handling="alias") - itemsize_var = Variable(NumpyInt64Type(), scope.get_new_name(f"{name}_itemsize")) if has_itemsize else None - shape_vars = [Variable(NumpyInt32Type(), scope.get_new_name(f"{name}_shape_{i + 1}")) for i in range(rank)] - f_array, elem_var = self._bind_c_array_result_storage(name, orig_var, pointer_target, has_itemsize) - shape = self._bind_c_array_result_shape(f_array, rank, shape) - body = [Assign(s_v, cast_to(s, NumpyInt32Type())) for s_v, s in zip(shape_vars, shape, strict=False)] - body.extend(self._bind_c_array_itemsize_body(itemsize_var, source_var, pointer_target, orig_var)) - byte_count = reduce(Mul, [itemsize_var, *shape_vars]) if itemsize_var is not None else None - body.extend( - self._bind_c_array_pointer_body( - orig_var, - bind_var, - elem_var, - shape_vars, - byte_count, - pointer_target, - f_array, - ) - ) - _result_var, c_result = self._bind_c_array_result_descriptor( - rank, - has_itemsize, - bind_var, - itemsize_var, - shape_vars, - orig_var, - ) - - return { - "c_result": c_result, - "body": body, - "f_array": f_array, - "bind_var": bind_var, - "byte_count": byte_count, - "itemsize_var": itemsize_var, - "shape_vars": shape_vars, - } - - def _bind_c_array_result_storage(self, name, orig_var, pointer_target, has_itemsize): - """Create the Fortran-side array storage used for bind-C array results.""" - if pointer_target: - return orig_var, None - dtype = orig_var.dtype - rank = orig_var.rank - order = orig_var.order - numpy_dtype = dtype if has_itemsize else numpy_precision_map[(dtype.primitive_type, dtype.precision)] - ptr_rank = 1 if has_itemsize else rank - ptr_var = Variable( - NumpyNDArrayType.get_new(numpy_dtype, ptr_rank, order), - self.scope.get_new_name(name + "_ptr"), - memory_handling="alias", - fortran_character_length=1 if has_itemsize else None, - ) - elem_var = Variable(dtype, self.scope.get_new_name(name + "_elem")) - self.scope.insert_variable(ptr_var) - self.scope.insert_variable(elem_var) - return ptr_var, elem_var - - @staticmethod - def _bind_c_array_result_shape(f_array, rank, shape): - """Fill unspecified bind-C result dimensions from the emitted Fortran array.""" - if shape is None: - return tuple(ArrayShapeElement(f_array, convert_to_literal(index)) for index in range(rank)) - return tuple( - ArrayShapeElement(f_array, convert_to_literal(index)) if dim is None else dim - for index, dim in enumerate(shape) - ) - - @staticmethod - def _bind_c_array_itemsize_body(itemsize_var, source_var, pointer_target, orig_var): - """Return itemsize assignment nodes for fixed-width character array results.""" - if itemsize_var is None: - return [] - length_source = source_var - if length_source is None and isinstance(pointer_target, Variable): - length_source = pointer_target - if length_source is None: - length_source = orig_var - return [Assign(itemsize_var, cast_to(FortranCharacterLength(length_source), NumpyInt64Type()))] - - def _bind_c_array_pointer_body(self, orig_var, bind_var, elem_var, shape_vars, byte_count, pointer_target, f_array): - """Create pointer association or allocation nodes for a bind-C result.""" - if pointer_target: - return [CLocFunc(self._bind_c_array_pointer_source(orig_var), bind_var)] - pointer_shape = ( - [byte_count] if byte_count is not None else self._bind_c_array_pointer_shape(orig_var, shape_vars) - ) - size_terms = ( - [BindCSizeOf(elem_var), byte_count] if byte_count is not None else [BindCSizeOf(elem_var), *shape_vars] - ) - return [ - Assign(bind_var, c_malloc(reduce(Mul, size_terms))), - If(IfSection(IsNot(bind_var, NIL), [C_F_Pointer(bind_var, f_array, pointer_shape)])), - ] - - @staticmethod - def _bind_c_array_pointer_shape(orig_var, shape_vars): - """Return pointer shape respecting Fortran or C-oriented storage.""" - return shape_vars if orig_var.order == "F" else shape_vars[::-1] - - @staticmethod - def _bind_c_array_pointer_source(orig_var): - """Return the addressable source for borrowed bind-C result arrays.""" - if ownership_decision_for_codegen_variable(orig_var).storage_mode is StorageMode.HEAP: - return IndexedElement( - orig_var, - *(ArrayLowerBound(orig_var, convert_to_literal(index)) for index in range(orig_var.rank)), - ) - return orig_var - - def _bind_c_array_result_descriptor(self, rank, has_itemsize, bind_var, itemsize_var, shape_vars, orig_var): - """Create and alias the bind-C array result descriptor.""" - descriptor_offset = 2 if has_itemsize else 1 - result_var = Variable( - BindCArrayType.get_new(rank, has_strides=False, has_itemsize=has_itemsize), - self.scope.get_new_name(), - shape=(rank + descriptor_offset,), - ) - c_result = BindCVariable(result_var, orig_var) - for descriptor in (result_var, c_result): - self.scope.insert_symbolic_alias(IndexedElement(descriptor, convert_to_literal(0)), bind_var) - if itemsize_var is not None: - self.scope.insert_symbolic_alias(IndexedElement(descriptor, convert_to_literal(1)), itemsize_var) - for i, s in enumerate(shape_vars): - self.scope.insert_symbolic_alias( - IndexedElement(descriptor, convert_to_literal(i + descriptor_offset)), s - ) - return result_var, c_result diff --git a/x2py/codegen/codegen.py b/x2py/codegen/codegen.py deleted file mode 100644 index b6dd46a14..000000000 --- a/x2py/codegen/codegen.py +++ /dev/null @@ -1,27 +0,0 @@ -"""Small container passed from semantic lowering to wrapper generation.""" - -from __future__ import annotations - - -class Codegen: - """Hold the generated module and scope used by `BindingPipeline`.""" - - def __init__(self, name, ast, scope): - self._name = name - self._scope = scope - self._ast = ast - - @property - def name(self): - """Return the Python extension module name.""" - return self._name - - @property - def scope(self): - """Return the root codegen scope.""" - return self._scope - - @property - def ast(self): - """Return the lowered codegen module AST.""" - return self._ast diff --git a/x2py/codegen/generator.py b/x2py/codegen/generator.py deleted file mode 100644 index a5b936e8e..000000000 --- a/x2py/codegen/generator.py +++ /dev/null @@ -1,55 +0,0 @@ -"""Shared visitor base for bridge and binding generators.""" - -from x2py.utilities.visitor import ClassVisitor - -from .scope import Scope - -__all__ = ("BindingGenerator", "BridgeGenerator") - - -class _Generator(ClassVisitor): - """Dispatch codegen model nodes to `_visit_` methods.""" - - start_language = None - target_language = None - generator_kind = "generator" - - def __init__(self, verbose): - self._scope = None - self._verbose = verbose - - @property - def scope(self): - """Return the current generation scope.""" - return self._scope - - @scope.setter - def scope(self, scope): - """Set the current generation scope.""" - assert isinstance(scope, Scope) - self._scope = scope - - def exit_scope(self): - """Return to the enclosing generation scope.""" - self._scope = self._scope.parent_scope - - def generate(self, expr): - """Generate a bridge or binding model for `expr`.""" - return self._visit(expr) - - def _visit_not_supported(self, expr): - """Raise an error when no visitor supports the model type.""" - msg = f"_visit_{type(expr).__name__} is not yet implemented for {self.generator_kind} : {type(self)}\n" - raise NotImplementedError(msg) - - -class BindingGenerator(_Generator): - """Base class for generators that create target-language bindings.""" - - generator_kind = "binding generator" - - -class BridgeGenerator(_Generator): - """Base class for generators that create language bridges.""" - - generator_kind = "bridge generator" diff --git a/x2py/codegen/models/__init__.py b/x2py/codegen/models/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/x2py/codegen/models/core.py b/x2py/codegen/models/core.py deleted file mode 100644 index e65068199..000000000 --- a/x2py/codegen/models/core.py +++ /dev/null @@ -1,4807 +0,0 @@ -""" -Module containing the core X2py AST nodes which are used in the syntactic -and semantic stages of X2py, and are relevant to all target languages. These -include model objects representing variable assignment, code blocks, and memory -allocation. Relationship bookkeeping lives in standalone helper functions, -without a shared model base class. -""" - -import inspect - -from itertools import chain -from typing import ClassVar - -from .datatypes import ( - CustomDataType, - FinalType, - Type, - NumpyBoolType, - NumpyInt32Type, - TupleType, - PrimitiveIntegerType, - NumpyInt64Type, - StringType, - _find_direct_model_parent, - _find_model_parent, - attach_model_child, - detach_model_child, - init_model_object, - is_model_class, - is_model_object, - register_model_class, - iterable, -) -from .datatypes import ( - Literal, - NIL, - NumpyNDArrayType, - convert_to_literal, -) -from .datatypes import GenericType - -__all__ = ( - "Add", - "AliasAssign", - "Allocate", - "And", - "ArithmeticOperator", - "ArrayAllocated", - "ArrayAssociated", - "ArrayContiguous", - "ArrayLowerBound", - "ArrayShapeElement", - "ArraySize", - "AsName", - "Assign", - "AugAssign", - "BinaryBooleanOperator", - "BinaryOperator", - "BooleanOperator", - "CaseSection", - "ClassDef", - "CodeBlock", - "Comment", - "CommentBlock", - "ComparisonOperator", - "Deallocate", - "Declare", - "Div", - "DottedVariable", - "EmptyNode", - "Eq", - "FortranCharacterLength", - "Function", - "FunctionAddress", - "FunctionCall", - "FunctionCallArgument", - "FunctionDef", - "FunctionDefArgument", - "FunctionDefResult", - "FunctionOverloadSet", - "Ge", - "Gt", - "If", - "IfSection", - "IfTernaryOperator", - "Import", - "In", - "IndexedElement", - "Is", - "IsNot", - "Le", - "Lt", - "Minus", - "Module", - "ModuleHeader", - "Mul", - "Ne", - "Not", - "Nullify", - "Operator", - "Or", - "Pass", - "PythonTuple", - "Return", - "SelectCase", - "SeparatorComment", - "Slice", - "Symbol", - "UnaryBooleanOperator", - "UnaryOperator", - "UnarySub", - "Variable", - "get_direct_assignment", - "get_direct_function_argument", - "get_direct_module", - "get_direct_overload_set", - "get_enclosing_class", - "get_enclosing_function", - "get_enclosing_module", - "is_in_overload_set", -) - - -def make_operator_class(name, base, op): - return type( - name, - (base,), - { - "__slots__": (), - "__module__": __name__, - "op": op, - }, - ) - - -# ============================================================================== -class Operator: - __slots__ = ("_args", "_class_type", "_shape") - _attribute_nodes = ("_args",) - op = None - _DEFAULT = object() - - def __init__(self, *args, shape=_DEFAULT, class_type=_DEFAULT): - self._args = tuple(args) - - self._shape = args[0]._shape if shape is self._DEFAULT else shape - self._class_type = args[0]._class_type if class_type is self._DEFAULT else class_type - - init_model_object(self) - - @property - def args(self): - return self._args - - def __str__(self): - return repr(self) - - -class UnaryOperator(Operator): - __slots__ = () - - def __repr__(self): - return f"{self.op}{self.args[0]!r}" - - -class BinaryOperator(Operator): - __slots__ = () - - def __repr__(self): - return f"{self.args[0]!r} {self.op} {self.args[1]!r}" - - -class BooleanOperator(Operator): - __slots__ = () - - def __init__(self, *args): - super().__init__(*args, shape=None, class_type=NumpyBoolType()) - - def __repr__(self): - return f" {self.op} ".join(repr(a) for a in self.args) - - -class UnaryBooleanOperator(BooleanOperator, UnaryOperator): - __slots__ = () - - def __init__(self, arg): - super().__init__(arg) - - def __repr__(self): - return UnaryOperator.__repr__(self) - - -class BinaryBooleanOperator(BooleanOperator, BinaryOperator): - __slots__ = () - - def __init__(self, arg1, arg2): - super().__init__(arg1, arg2) - - -class ArithmeticOperator(BinaryOperator): - __slots__ = () - - -class ComparisonOperator(BinaryBooleanOperator): - __slots__ = () - - -UnarySub = make_operator_class("UnarySub", UnaryOperator, "-") - -Not = make_operator_class("Not", UnaryBooleanOperator, "not ") - -Add = make_operator_class("Add", ArithmeticOperator, "+") -Mul = make_operator_class("Mul", ArithmeticOperator, "*") -Minus = make_operator_class("Minus", ArithmeticOperator, "-") -Div = make_operator_class("Div", ArithmeticOperator, "/") - -Eq = make_operator_class("Eq", ComparisonOperator, "==") -Ne = make_operator_class("Ne", ComparisonOperator, "!=") -Lt = make_operator_class("Lt", ComparisonOperator, "<") -Le = make_operator_class("Le", ComparisonOperator, "<=") -Gt = make_operator_class("Gt", ComparisonOperator, ">") -Ge = make_operator_class("Ge", ComparisonOperator, ">=") - -And = make_operator_class("And", BooleanOperator, "and") -Or = make_operator_class("Or", BooleanOperator, "or") -Is = make_operator_class("Is", BinaryBooleanOperator, "is") -IsNot = make_operator_class("IsNot", BinaryBooleanOperator, "is not") -In = make_operator_class("In", BinaryBooleanOperator, "in") - - -# ============================================================================== -class IfTernaryOperator(Operator): - """ - Represent a ternary conditional operator in the code. - - Represent a ternary conditional operator in the code, - of the form (a if cond else b). - - Parameters - ---------- - cond : model object - The condition which determines which result is returned. - value_true : model object - The value returned if the condition is true. - value_false : model object - The value returned if the condition is false. - - Examples - -------- - >>> from x2py.ast.internals import Symbol - >>> from x2py.ast.core import Assign - >>> from x2py.ast.operators import IfTernaryOperator - >>> n = Symbol('n') - >>> x = 5 if n > 1 else 2 - >>> IfTernaryOperator(Gt(n > 1), 5, 2) - IfTernaryOperator(Gt(n > 1), 5, 2) - """ - - __slots__ = () - - def __init__(self, cond, value_true, value_false): - super().__init__(cond, value_true, value_false, shape=value_true._shape, class_type=value_true._class_type) - - @property - def cond(self): - return self._args[0] - - @property - def value_true(self): - return self._args[1] - - @property - def value_false(self): - return self._args[2] - - def __str__(self): - return f"(({self.value_true}) if ({self.cond}) else ({self.value_false})" - - -# ============================================================================== -class Symbol(str): - """ - Class representing a symbol in the code. - - Symbolic placeholder for a Python variable, which has a name but no type yet. - This is very generic, and it can also represent a function or a module. - - Parameters - ---------- - name : str - Name of the symbol. - - is_temp : bool - Indicates if the symbol is a temporary object. This either means that the - symbol represents an object originally named `_` in the code, or that the - symbol represents an object created by X2py in order to assign a - temporary object. This is sometimes necessary to facilitate the translation. - - Examples - -------- - >>> from x2py.ast.internals import Symbol - >>> x = Symbol('x') - x - """ - - __slots__ = ("_is_temp",) - _model_immutable = True - - def __new__(cls, name, is_temp=False): - return super().__new__(cls, name) - - def __init__(self, name, is_temp=False): - self._is_temp = is_temp - init_model_object(self) - - @property - def is_temp(self): - """ - Indicates if this symbol represents a temporary variable created by X2py, - and was not present in the original Python code [default value : False]. - """ - return self._is_temp - - -class Variable: - """ - Represents a typed variable. - - Represents a variable in the code and stores all useful properties which allow - for easy usage of this variable. - - Parameters - ---------- - class_type : Type - The Python type of the variable. - - name : str, list, DottedName - The name of the variable represented. This can be either a string - or a dotted name, when using a Class attribute. - - memory_handling : str, default: 'stack' - 'heap' is used for arrays, if we need to allocate memory on the heap. - 'stack' if memory should be allocated on the stack, represents stack arrays and scalars. - 'alias' if object allows access to memory stored in another variable. - - is_target : bool, default: False - Indicates if object is pointed to by another variable. - - is_optional : bool, default: False - Indicates if object is an optional argument of a function. - - is_private : bool, default: False - Indicates if object is private within a Module. - - projected_output : bool, default: False - True when a compact semantic contract projects this visible writable - argument into Python returns without preserving source-level output syntax. - - passes_by_value : bool, default: False - True when a native scalar dummy has Fortran ``value`` ABI. - - fortran_array_category : str, optional - Native Fortran array category preserved as ABI metadata. Python - extraction and native handoff policy comes from ``ownership_decision``. - - fortran_callback_access : str, optional - Exact callback dummy declaration access for Fortran adapter signatures. - - fortran_character_length : object, optional - Native Fortran character element length for character scalars and arrays. - - fortran_source_shape : tuple, optional - Native Fortran source dimensions preserved for ABI-sensitive declarations. - - ownership_decision : object, default: None - Central ownership policy decision preserved from semantic lowering. - - getter_ownership_decision : object, default: None - Completed policy used when this field or module variable is read. - - setter_ownership_decision : object, default: None - Completed policy used when this field is assigned through a generated setter. - - native_array_handle_policy : object, default: None - Completed policy for native allocatable or pointer array handle lowering. - - array_interop_policy : object, default: None - Completed selector for ordinary data-buffer versus native descriptor array ABI. - - shape : tuple, default: None - The shape of the array. A tuple whose elements indicate the number of elements along - each of the dimensions of an array. The elements of the tuple should be None or model objects. - - cls_base : class, default: None - Class base if variable is an object or an object member. - - is_argument : bool, default: False - Indicates if object is the argument of a function. - - is_temp : bool, default: False - Indicates if this symbol represents a temporary variable created by X2py, - and was not present in the original Python code. - - Examples - -------- - >>> from x2py.ast.datatypes import NumpyInt64Type, NumpyFloat64Type - >>> from x2py.ast.core import Variable - >>> Variable(NumpyInt64Type(), 'n') - n - >>> n = 4 - >>> Variable(NumpyFloat64Type(), 'x', shape=(n,2), memory_handling='heap') - x - >>> Variable(NumpyInt64Type(), DottedName('matrix', 'n_rows')) - matrix.n_rows - """ - - __slots__ = ( - "_alloc_shape", - "_array_interop_policy", - "_assumed_rank", - "_class_type", - "_cls_base", - "_default_value", - "_fortran_array_category", - "_fortran_callback_access", - "_fortran_character_length", - "_fortran_source_shape", - "_getter_ownership_decision", - "_is_argument", - "_is_optional", - "_is_private", - "_is_target", - "_is_temp", - "_memory_handling", - "_name", - "_native_array_handle_policy", - "_ownership_decision", - "_passes_by_value", - "_projected_output", - "_setter_ownership_decision", - "_shape", - ) - _attribute_nodes = () - - def __init__( - self, - class_type, - name, - *, - memory_handling="stack", - is_target=False, - is_optional=False, - is_private=False, - passes_by_value=False, - fortran_array_category=None, - fortran_callback_access=None, - fortran_character_length=None, - fortran_source_shape=None, - getter_ownership_decision=None, - ownership_decision=None, - setter_ownership_decision=None, - native_array_handle_policy=None, - array_interop_policy=None, - projected_output=False, - assumed_rank=False, - shape=None, - cls_base=None, - default_value=None, - is_argument=False, - is_temp=False, - ): - init_model_object(self) - - # ------------ Variable Properties --------------- - # if class attribute - if isinstance(name, str): - name = name.split(""".""") - if len(name) == 1: - name = Symbol(name[0]) - else: - raise ValueError(name) - - assert isinstance(name, Symbol) - self._name = name - - if memory_handling not in ("heap", "stack", "alias"): - raise ValueError("memory_handling must be 'heap', 'stack' or 'alias'") - self._memory_handling = memory_handling - - if not isinstance(is_target, bool): - raise TypeError("is_target must be a boolean.") - self.is_target = is_target - - if not isinstance(is_optional, bool): - raise TypeError("is_optional must be a boolean.") - self._is_optional = is_optional - - if not isinstance(is_private, bool): - raise TypeError("is_private must be a boolean.") - self._is_private = is_private - - if not isinstance(passes_by_value, bool): - raise TypeError("passes_by_value must be a boolean.") - self._passes_by_value = passes_by_value - self._fortran_array_category = fortran_array_category - if fortran_callback_access not in (None, "read", "write", "readwrite", "unspecified"): - raise ValueError( - "fortran_callback_access must be one of None, 'read', 'write', 'readwrite', or 'unspecified'" - ) - self._fortran_callback_access = fortran_callback_access - self._fortran_character_length = fortran_character_length - self._fortran_source_shape = tuple(fortran_source_shape or ()) - self._getter_ownership_decision = getter_ownership_decision - self._ownership_decision = ownership_decision - self._setter_ownership_decision = setter_ownership_decision - self._native_array_handle_policy = native_array_handle_policy - self._array_interop_policy = array_interop_policy - if not isinstance(projected_output, bool): - raise TypeError("projected_output must be a boolean.") - self._projected_output = projected_output - if not isinstance(assumed_rank, bool): - raise TypeError("assumed_rank must be a boolean.") - self._assumed_rank = assumed_rank - self._cls_base = cls_base - self._default_value = default_value - self._is_argument = is_argument - self._is_temp = is_temp - - # ------------ model object Properties --------------- - assert isinstance(class_type, Type) - rank = class_type.rank - - if rank == 0: - assert shape is None - - elif shape is None: - shape = tuple(None for i in range(class_type.container_rank)) - - self._alloc_shape = shape - self._class_type = class_type - self._shape = self.process_shape(shape) - - def process_shape(self, shape): - """ - Simplify the provided shape and ensure it has the expected format. - - The provided shape is the shape used to create the object, and it can - be a long expression. In most cases where the shape is required the - provided shape is inconvenient, or it might have become invalid. This - function therefore replaces those expressions with calls to the function - `ArrayShapeElement`. - - Parameters - ---------- - shape : iterable of int - The array shape to be simplified. - - Returns - ------- - tuple - The simplified array shape. - """ - if self.rank == 0: - return None - if not hasattr(shape, "__iter__"): - shape = [shape] - - new_shape = [None] * len(shape) - for i, s in enumerate(shape): - if isinstance(s, Literal) and isinstance(s.dtype.primitive_type, PrimitiveIntegerType): - new_shape[i] = s - elif isinstance(s, int): - new_shape[i] = convert_to_literal(s) - elif is_model_object(s): - new_shape[i] = s - elif s is not None: - raise ValueError(s) - return tuple(new_shape) - - @property - def name(self): - """Name of the variable""" - return self._name - - @property - def alloc_shape(self): - """Shape of the variable at allocation - - The shape used in x2py is usually simplified to contain - only Literals and ArraySizes but the shape for - the allocation of x cannot be `Shape(x)` - """ - return self._alloc_shape - - @property - def memory_handling(self): - """Indicates whether a Variable has a dynamic size""" - return self._memory_handling - - @memory_handling.setter - def memory_handling(self, memory_handling): - if memory_handling not in ("heap", "stack", "alias"): - raise ValueError("memory_handling must be 'heap', 'stack' or 'alias'") - self._memory_handling = memory_handling - - @property - def is_alias(self): - """Indicates if variable is an alias""" - return self.memory_handling == "alias" - - @property - def on_heap(self): - """Indicates if memory is allocated on the heap""" - return self.memory_handling == "heap" - - @property - def on_stack(self): - """Indicates if memory is allocated on the stack""" - return self.memory_handling == "stack" - - @property - def is_stack_array(self): - """Indicates if the variable is located on stack and is an array""" - return self.on_stack and self.rank > 0 - - @property - def cls_base(self): - """Class from which the Variable inherits""" - return self._cls_base - - @property - def default_value(self): - """Source-level literal value associated with this variable, when any.""" - return self._default_value - - @property - def is_temp(self): - """ - Indicates if this symbol represents a temporary variable created by X2py, - and was not present in the original Python code [default value : False]. - """ - return self._is_temp - - @property - def is_target(self): - """Indicates if the data in this Variable is - shared with (pointed at by) another Variable - """ - return self._is_target - - @is_target.setter - def is_target(self, is_target): - if not isinstance(is_target, bool): - raise TypeError("is_target must be a boolean.") - self._is_target = is_target - - @property - def is_optional(self): - """Indicates if the Variable is optional - in this context - """ - return self._is_optional - - @property - def is_private(self): - """Indicates if the Variable is private - within the Module - """ - return self._is_private - - @property - def passes_by_value(self): - """True when the native scalar dummy uses Fortran ``value`` ABI.""" - return self._passes_by_value - - @property - def projected_output(self): - """True when this visible writable argument is projected as a Python result.""" - return self._projected_output - - @property - def fortran_array_category(self): - """Native Fortran array category carried as ABI metadata.""" - return self._fortran_array_category - - @property - def fortran_callback_access(self): - """Exact callback dummy declaration access for Fortran adapter signatures.""" - return self._fortran_callback_access - - @property - def fortran_character_length(self): - """Native Fortran character element length, when this variable stores character data.""" - return self._fortran_character_length - - @property - def fortran_source_shape(self): - """Native Fortran source dimensions used by ABI-sensitive printers.""" - return self._fortran_source_shape - - @property - def ownership_decision(self): - """Central ownership policy decision for this variable.""" - return self._ownership_decision - - @property - def getter_ownership_decision(self): - """Completed policy used by a generated getter.""" - return self._getter_ownership_decision - - @property - def setter_ownership_decision(self): - """Completed ownership policy used by a generated field setter.""" - return self._setter_ownership_decision - - @property - def native_array_handle_policy(self): - """Completed policy for native allocatable or pointer array handles.""" - return self._native_array_handle_policy - - @property - def array_interop_policy(self): - """Completed selector for the generated array ABI lane.""" - return self._array_interop_policy - - @property - def assumed_rank(self): - """True when this array represents a Fortran ``dimension(..)`` dummy.""" - return self._assumed_rank - - @property - def is_argument(self): - """Indicates whether the Variable is - a function argument in this context - """ - return self._is_argument - - def declare_as_argument(self): - """ - Indicate that the variable is used as an argument. - - This function is called by FunctionDefArgument to ensure that - arguments are correctly flagged as such. - """ - self._is_argument = True - - @property - def is_ndarray(self): - """ - User friendly method to check if the variable is a numpy.ndarray. - - User friendly method to check if the variable is an ndarray. - """ - return isinstance(self.class_type, NumpyNDArrayType) and not self.class_type.raw - - @property - def is_raw_array(self): - """Whether the variable is represented directly as a C array or pointer.""" - return isinstance(self.class_type, NumpyNDArrayType) and self.class_type.raw - - def __str__(self): - return str(self.name) - - def __repr__(self): - return f"{type(self).__name__}({self.name}, type={self.class_type!r})" - - def __hash__(self): - return hash((type(self).__name__, self._name)) - - def clone(self, name, new_class=None, **kwargs): - """ - Create a clone of the current variable. - - Create a new Variable object of the chosen class - with the provided name and options. All non-specified - options will match the current instance. - - Parameters - ---------- - name : str - The name of the new Variable. - new_class : type, optional - The class type of the new Variable (e.g. DottedVariable). - The default is the same class type. - **kwargs : dict - Dictionary containing any keyword-value - pairs which are valid constructor keywords. - - Returns - ------- - Variable - The cloned variable. - """ - - cls = self.__class__ if new_class is None else new_class - - args = inspect.signature(Variable.__init__) - new_kwargs = {k: getattr(self, "_" + k) for k in args.parameters if "_" + k in dir(self)} - new_kwargs.update(kwargs) - new_kwargs["name"] = name - if "shape" not in kwargs: - new_kwargs["shape"] = self.alloc_shape - - return cls(**new_kwargs) - - def rename(self, newname): - """Forbidden method for renaming the variable""" - # The name is part of the hash so it must never change - raise RuntimeError("Cannot modify hash definition") - - @is_temp.setter - def is_temp(self, is_temp): - if not isinstance(is_temp, bool): - raise TypeError("is_temp must be a boolean") - if is_temp: - raise ValueError("Variables cannot become temporary") - self._is_temp = is_temp - - -class IndexedElement: - """ - Represents an indexed object in the code. - - Represents an object which is a subset of a base object. The - indexed object is retrieved by passing indices to the base - object using the `[]` syntax. - - In the semantic stage, the base object is an array, tuple or - list. This function then determines the new rank and shape of - the data block. - - In the syntactic stage, this object is more versatile, it - stores anything which is indexed using `[]` syntax. This can - additionally include classes, maps, etc. - - Parameters - ---------- - base : Variable | Symbol | DottedName - The object being indexed. - - *indices : tuple of model object - The values used to index the base. - - Examples - -------- - >>> from x2py.ast.core import Variable, IndexedElement - >>> from x2py.ast.datatypes import NumpyInt64Type - >>> A = Variable(NumpyInt64Type(), 'A', shape=(2,3), rank=2) - >>> i = Variable(NumpyInt64Type(), 'i') - >>> j = Variable(NumpyInt64Type(), 'j') - >>> IndexedElement(A, (i, j)) - IndexedElement(A, i, j) - >>> IndexedElement(A, i, j) == A[i, j] - True - """ - - __slots__ = ("_class_type", "_indices", "_is_slice", "_label", "_shape") - _attribute_nodes = ("_label", "_indices", "_shape") - - def __init__(self, base, *indices): - self._label = base - self._shape = None - - rank = base.class_type.container_rank - assert len(indices) <= rank - - if any(not isinstance(a, int | Slice) and not is_model_object(a) for a in indices): - raise TypeError("Index is not of valid type") - - if len(indices) < rank: - indices = indices + tuple([Slice(None, None)] * (rank - len(indices))) - self._indices = tuple(convert_to_literal(a) if isinstance(a, int) else a for a in indices) - else: - self._indices = tuple(convert_to_literal(a) if isinstance(a, int) else a for a in indices) - - if isinstance(base.class_type, TupleType): - assert ( - len(self._indices) == 1 - and isinstance(self._indices[0], Literal) - and isinstance(self._indices[0].dtype.primitive_type, PrimitiveIntegerType) - ) - self._class_type = base.class_type[self._indices[0]] - self._is_slice = False - self._shape = None - else: - self._class_type = base.class_type.element_type - self._is_slice = False - self._shape = (1,) - - init_model_object(self) - - @property - def base(self): - """The object which is indexed""" - return self._label - - @property - def indices(self): - """A tuple of indices used to index the variable""" - return self._indices - - def __str__(self): - indices = ",".join(str(i) for i in self.indices) - return f"{self.base}[{indices}]" - - def __repr__(self): - indices = ",".join(repr(i) for i in self.indices) - return f"{self.base!r}[{indices}]" - - def __hash__(self): - return hash((self.base, self._indices)) - - -class FortranCharacterLength: - """Represent the Fortran ``len(value)`` intrinsic for character storage.""" - - __slots__ = ("_arg", "_class_type", "_shape") - _attribute_nodes = ("_arg",) - - def __init__(self, arg): - self._arg = arg - self._class_type = NumpyInt32Type() - self._shape = None - init_model_object(self) - - @property - def arg(self): - """The character scalar or array whose element length is requested.""" - return self._arg - - -class DottedVariable(Variable): - """ - Class representing a dotted variable. - - Represents a dotted variable. This is usually - a variable which is a member of a class - - E.g. - a = AClass() - a.b = 3 - - In this case b is a DottedVariable where the lhs is a. - - Parameters - ---------- - *args : tuple - See x2py.ast.variable.Variable. - - lhs : Variable - The Variable on the right of the '.'. - - **kwargs : dict - See x2py.ast.variable.Variable. - """ - - __slots__ = ("_lhs",) - _attribute_nodes = ("_lhs",) - - def __init__(self, *args, lhs, **kwargs): - self._lhs = lhs - super().__init__(*args, **kwargs) - - @property - def lhs(self): - """The object before the final dot in the - dotted variable - - e.g. for the DottedVariable: - a.b - The lhs is a - """ - return self._lhs - - def __hash__(self): - return hash((type(self).__name__, self.name, self.lhs)) - - def __str__(self): - return str(self.lhs) + "." + str(self.name) - - def __repr__(self): - lhs = repr(self.lhs) - name = str(self.name) - class_type = repr(self.class_type) - classname = type(self).__name__ - return f"{classname}({lhs}.{name}, type={class_type})" - - -class AsName: - """ - Represents a renaming of an object, used with Import. - - A class representing the renaming of an object such as a function or a - variable. This usually occurs during an Import. - - Parameters - ---------- - obj : model object or model type - The variable, function, or module being renamed. - local_alias : str - Name of variable or function in this context. - """ - - __slots__ = ("_local_alias", "_obj", "_source_name") - _attribute_nodes = () - - def __init__(self, obj, local_alias, *, source_name=None): - assert (is_model_object(obj) and not isinstance(obj, Symbol)) or is_model_class(obj) - self._obj = obj - self._local_alias = local_alias - self._source_name = source_name - init_model_object(self) - - @property - def name(self): - """The original name of the object""" - if self._source_name is not None: - return self._source_name - obj = self._obj - if isinstance(obj, str | Symbol): - return obj - return obj.name - - @property - def local_alias(self): - """ - The local_alias name of the object. - - The name used to identify the object in the local scope. - """ - return self._local_alias - - @property - def object(self): - """The underlying object described by this AsName""" - return self._obj - - def __repr__(self): - return f"{self.object} as {self.local_alias}" - - def __eq__(self, string): - if isinstance(string, str): - return string == self.local_alias - if isinstance(string, AsName): - return string.local_alias == self.local_alias - return self is string - - def __ne__(self, string): - return not self == string - - def __hash__(self): - return hash(self.local_alias) - - -class Assign: - """ - Represents variable assignment for code generation. - - Class representing an assignment node, where the result of an expression - (rhs: right hand side) is saved into a variable (lhs: left hand side). - - Parameters - ---------- - lhs : model object - In the syntactic stage: - Object representing the lhs of the expression. These should be - singular objects, such as one would use in writing code. Notable types - include Symbol, and IndexedElement. Types that - subclass these types are also supported. - In the semantic stage: - Variable or IndexedElement. - - rhs : model object - In the syntactic stage: - Object representing the rhs of the expression. - In the semantic stage : - model object with the same shape as the lhs. - - Examples - -------- - >>> from x2py.ast.datatypes import NumpyInt64Type - >>> from x2py.ast.internals import symbols - >>> from x2py.ast.variable import Variable - >>> from x2py.ast.core import Assign - >>> x, y, z = symbols('x, y, z') - >>> Assign(x, y) - x := y - >>> Assign(x, 0) - x := 0 - >>> A = Variable(NumpyInt64Type(), 'A', rank = 2) - >>> Assign(x, A) - x := A - >>> Assign(A[0,1], x) - IndexedElement(A, 0, 1) := x - """ - - __slots__ = ("_lhs", "_rhs") - _attribute_nodes = ("_lhs", "_rhs") - - def __init__(self, lhs, rhs): - if isinstance(lhs, tuple | list): - lhs = tuple(lhs) - self._lhs = lhs - self._rhs = rhs - init_model_object(self) - - def __str__(self): - return f"{self.lhs} := {self.rhs}" - - def __repr__(self): - return f"{self.lhs!r} := {self.rhs!r}" - - @property - def lhs(self): - return self._lhs - - @property - def rhs(self): - return self._rhs - - @property - def is_alias(self): - """Returns True if the assignment is an alias.""" - - # TODO to be improved when handling classes - - lhs = self.lhs - rhs = self.rhs - cond = isinstance(rhs, Variable) and rhs.rank > 0 - cond = cond or isinstance(rhs, IndexedElement) - cond = cond and isinstance(lhs, Symbol) - return cond or (isinstance(rhs, Variable) and rhs.is_alias) - - -# ------------------------------------------------------------------------------ -class Allocate: - """ - Represents memory allocation for code generation. - - Represents memory allocation (usually of an array) for code generation. - This is relevant to low-level target languages, such as C or Fortran, - where the programmer must take care of heap memory allocation. - - Parameters - ---------- - variable : x2py.ast.core.Variable - The typed variable (usually an array) that needs memory allocation. - - shape : int or iterable or None - Shape of the array after allocation (None for scalars). - - status : str {'allocated'|'unallocated'|'unknown'} - Variable allocation status at object creation. - - like : model object, optional - A model object describing the amount of memory which must be allocated. - In C this provides the size which will be passed to malloc. In Fortran - this provides the source argument of the allocate function. - - alloc_type : str {'init'|'reserve'|'resize'}, optional - Specifies the memory allocation strategy for containers with dynamic memory management. - This parameter is relevant for any container type where memory allocation patterns - need to be specified based on usage. - - - 'init' refers to direct allocation with predefined data (e.g., `x = [1, 2, 4]`). - - 'reserve' refers to cases where the container will be appended to. - - 'resize' refers to cases where the container is populated via indexed elements. - - 'function' refers to cases where the container is allocated in a function. It is - still useful to have an allocate node in this case for easy determination - of where deallocations are needed. - - Notes - ----- - An object of this class is immutable, although it contains a reference to a - mutable Variable object. - """ - - __slots__ = ("_alloc_type", "_like", "_order", "_shape", "_status", "_variable") - _attribute_nodes = ("_variable", "_like") - - # ... - def __init__(self, variable, *, shape, status, like=None, alloc_type=None): - if not isinstance(variable, Variable): - raise TypeError(f"Can only allocate a 'Variable' object, got {type(variable)} instead") - - if variable.on_stack: - # Variable may only be a pointer in the wrapper - raise ValueError("Variable must be allocatable") - - if shape and not isinstance(shape, int | tuple | list): - raise TypeError(f"Cannot understand 'shape' parameter of type '{type(shape)}'") - - assert variable.class_type.shape_is_compatible(shape) - - if not isinstance(status, str): - raise TypeError(f"Cannot understand 'status' parameter of type '{type(status)}'") - - if status not in ("allocated", "unallocated", "unknown"): - raise ValueError(f"Value of 'status' not allowed: '{status}'") - - assert alloc_type in (None, "init", "reserve", "resize", "function") - assert alloc_type in (None, "function") - - self._variable = variable - self._shape = shape - self._order = variable.order - self._status = status - self._like = like - self._alloc_type = alloc_type - init_model_object(self) - - # ... - - @property - def variable(self): - """ - The variable to be allocated. - - The variable to be allocated. - """ - return self._variable - - @property - def shape(self): - """ - The shape that the variable should be allocated to. - - The shape that the variable should be allocated to. - """ - return self._shape - - @property - def order(self): - """ - The order that the variable will be allocated with. - - The order that the variable will be allocated with. - """ - return self._order - - @property - def status(self): - """ - The allocation status of the variable before this allocation. - - The allocation status of the variable before this allocation. - One of {'allocated'|'unallocated'|'unknown'}. - """ - return self._status - - @property - def like(self): - """ - model object describing the amount of memory needed for the allocation. - - A model object describing the amount of memory which must be allocated. - In C this provides the size which will be passed to malloc. In Fortran - this provides the source argument of the allocate function. - """ - return self._like - - @property - def alloc_type(self): - """ - Determines the allocation type for homogeneous containers. - - Returns a string that indicates the allocation type used for memory allocation. - The value is either 'init' for containers initialized with predefined data, - 'reserve' for containers populated through appending, and 'resize' for containers - populated through indexed element assignment. - """ - return self._alloc_type - - def __str__(self): - return f"Allocate({self.variable}, shape={self.shape}, order={self.order}, status={self.status})" - - def __eq__(self, other): - if isinstance(other, Allocate): - return ( - (self.variable is other.variable) - and (self.shape == other.shape) - and (self.order == other.order) - and (self.status == other.status) - ) - return False - - def __hash__(self): - return hash((id(self.variable), self.shape, self.order, self.status)) - - -# ------------------------------------------------------------------------------ -class Nullify: - """Fortran pointer nullification statement.""" - - __slots__ = ("_variable",) - _attribute_nodes = ("_variable",) - - def __init__(self, variable): - if not isinstance(variable, Variable): - raise TypeError(f"Can only nullify a 'Variable' object, got {type(variable)} instead") - self._variable = variable - init_model_object(self) - - @property - def variable(self): - return self._variable - - -# ------------------------------------------------------------------------------ -class Deallocate: - """ - Class representing memory deallocation. - - Represents memory deallocation (usually of an array) for code generation. - This is relevant to low-level target languages, such as C or Fortran, - where the programmer must take care of heap memory deallocation. - - Parameters - ---------- - variable : x2py.ast.core.Variable - The typed variable (usually an array) that needs memory deallocation. - - Notes - ----- - An object of this class is immutable, although it contains a reference to a - mutable Variable object. - """ - - __slots__ = ("_variable",) - _attribute_nodes = ("_variable",) - - # ... - def __init__(self, variable): - if not isinstance(variable, Variable): - raise TypeError(f"Can only allocate a 'Variable' object, got {type(variable)} instead") - - self._variable = variable - init_model_object(self) - - # ... - - @property - def variable(self): - return self._variable - - def __eq__(self, other): - if isinstance(other, Deallocate): - return self.variable is other.variable - return False - - def __hash__(self): - return hash(id(self.variable)) - - -# ------------------------------------------------------------------------------ -class CodeBlock: - """ - Represents a block of statements. - - Represents a list of statements for code generation. Each statement - represents a line of code. - - Parameters - ---------- - body : iterable - The lines of code to be grouped together. - - unravelled : bool, default=False - Indicates whether the loops in the code have already been unravelled. - This is useful for printing in languages which don't support vector - expressions. - """ - - __slots__ = ("_body", "_unravelled") - _attribute_nodes = ("_body",) - - def __init__(self, body, unravelled=False): - ls = [] - for i in body: - if isinstance(i, CodeBlock): - ls += i.body - elif i is not None and not isinstance(i, EmptyNode): - ls.append(i) - if not isinstance(unravelled, bool): - raise TypeError("unravelled must be a boolean") - self._body = tuple(ls) - self._unravelled = unravelled - init_model_object(self) - - @property - def body(self): - return self._body - - @property - def unravelled(self): - """Indicates whether the vector syntax of python - has been unravelled into for loops - """ - return self._unravelled - - @property - def lhs(self): - return self.body[-1].lhs - - def __repr__(self): - return f"CodeBlock({self.body})" - - -class AliasAssign: - """ - Representing assignment of an alias to its local_alias. - - Represents aliasing for code generation. An alias is any statement of the - form `lhs := rhs` where lhs is a pointer and rhs is a local_alias. In other words - the contents of `lhs` will change if the contents of `rhs` are modified. - - Parameters - ---------- - lhs : model object - In the syntactic stage: - Object representing the lhs of the expression. These should be - singular objects, such as one would use in writing code. Notable types - include Symbol, and IndexedElement. Types that - subclass these types are also supported. - In the semantic stage: - Variable. - - rhs : Symbol | Variable, IndexedElement - The local_alias of the assignment. A Symbol in the syntactic stage, - a Variable or a Slice of an array in the semantic stage. - - Examples - -------- - >>> from x2py.ast.internals import Symbol - >>> from x2py.ast.core import AliasAssign - >>> from x2py.ast.core import Variable - >>> n = Variable(NumpyInt64Type(), 'n') - >>> x = Variable(NumpyInt64Type(), 'x', rank=1, shape=[n]) - >>> y = Symbol('y') - >>> AliasAssign(y, x) - """ - - __slots__ = ("_lhs", "_rhs") - _attribute_nodes = ("_lhs", "_rhs") - - def __init__(self, lhs, rhs): - if not lhs.is_alias: - raise TypeError("lhs must be a pointer") - - if isinstance(rhs, FunctionCall) and not rhs.funcdef.results.var.is_alias: - raise TypeError("A pointer cannot point to the address of a temporary variable") - - self._lhs = lhs - self._rhs = rhs - init_model_object(self) - - def __str__(self): - return f"{self.lhs} := {self.rhs}" - - @property - def lhs(self): - return self._lhs - - @property - def rhs(self): - return self._rhs - - -class AugAssign(Assign): - r""" - Represents augmented variable assignment for code generation. - - Represents augmented variable assignment for code generation. - Augmented variable assignment is an assignment which modifies the - variable using its initial value rather than simply replacing the - value; for example via an addition (`+=`). - - Parameters - ---------- - lhs : Symbol | model object - Object representing the lhs of the expression. - In the syntactic stage this may be a Symbol, or an IndexedElement. - In later stages the object should inherit from model object and be fully - typed. - - op : str - Operator (+, -, /, \*, %). - - rhs : model object - Object representing the rhs of the expression. - - Examples - -------- - >>> from x2py.ast.core import Variable - >>> from x2py.ast.core import AugAssign - >>> s = Variable(NumpyInt64Type(), 's') - >>> t = Variable(NumpyInt64Type(), 't') - >>> AugAssign(s, '+', 2 * t + 1) - s += 1 + 2*t - """ - - __slots__ = ("_op",) - _accepted_operators: ClassVar = { - "+": Add, - } - - def __init__(self, lhs, op, rhs): - if op not in self._accepted_operators: - raise TypeError("Unrecognized Operator") - - self._op = op - - super().__init__(lhs, rhs) - - def __repr__(self): - return f"{self.lhs} {self.op}= {self.rhs}" - - @property - def op(self): - """ - Get the string describing the operator which modifies the lhs variable. - - Get the string describing the operator which modifies the lhs variable. - """ - return self._op - - def to_basic_assign(self): - """ - Convert the AugAssign to an Assign. - - Convert the AugAssign to an Assign. - E.g. convert: - a += b - to: - a = a + b - - Returns - ------- - Assign - An assignment equivalent to the AugAssign. - """ - return Assign(self.lhs, self._accepted_operators[self._op](self.lhs, self.rhs)) - - -class Module: - """ - Represents a module in the code. - - The X2py node representing a Python module. A module consists of everything - inside a given Python file. - - Parameters - ---------- - name : str - Name of the module. - - variables : list - List of the variables that appear in the block. - - funcs : list - A list of FunctionDef instances. - - init_func : FunctionDef, default: None - The function which initialises the module (expressions in the - python module which are executed on import). - - free_func : FunctionDef, default: None - The function which frees any variables allocated in the module. - - overload_sets : list - A list of FunctionOverloadSet instances. - - classes : list - A list of ClassDef instances. - - imports : list, tuple - List of needed imports. - - scope : Scope - The scope of the module. - - is_external : bool - Indicates if the Module's definition is found elsewhere. - This is notably the case for gFTL extensions. - - Examples - -------- - >>> from x2py.ast.variable import Variable - >>> from x2py.ast.core import FunctionDefArgument, Assign, FunctionDefResult - >>> from x2py.ast.core import ClassDef, FunctionDef, Module - >>> from x2py.ast.operators import Add, Minus - >>> from x2py.codegen.models.datatypes import convert_to_literal - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> z = Variable(NumpyFloat64Type(), 'z') - >>> t = Variable(NumpyFloat64Type(), 't') - >>> a = Variable(NumpyFloat64Type(), 'a') - >>> b = Variable(NumpyFloat64Type(), 'b') - >>> body = [Assign(z,Add(x,a))] - >>> args = [FunctionDefArgument(arg) for arg in [x,y,a,b]] - >>> results = [FunctionDefResult(res) for res in [z,t]] - >>> translate = FunctionDef('translate', args, results, body) - >>> attributes = [x,y] - >>> methods = [translate] - >>> Point = ClassDef('Point', attributes, methods) - >>> incr = FunctionDef('incr', [FunctionDefArgument(x)], [FunctionDefResult(y)], [Assign(y,Add(x,convert_to_literal(1)))]) - >>> decr = FunctionDef('decr', [FunctionDefArgument(x)], [FunctionDefResult(y)], [Assign(y,Minus(x,convert_to_literal(1)))]) - >>> Module('my_module', [], [incr, decr], classes = [Point]) - Module(my_module, [], [FunctionDef(), FunctionDef()], [], [ClassDef(Point, (x, y), (FunctionDef(),), [public], (), [], [])], ()) - """ - - __slots__ = ( - "_classes", - "_free_func", - "_funcs", - "_imports", - "_init_func", - "_internal_dictionary", - "_is_external", - "_name", - "_overload_sets", - "_python_exports", - "_variable_inits", - "_variables", - ) - _attribute_nodes = ( - "_variables", - "_funcs", - "_overload_sets", - "_classes", - "_imports", - "_init_func", - "_free_func", - "_variable_inits", - ) - - def __init__( - self, - name, - variables, - funcs, - init_func=None, - free_func=None, - overload_sets=(), - classes=(), - imports=(), - scope=None, - is_external=False, - python_exports=None, - ): - if not isinstance(name, str): - raise TypeError("name must be a string") - - if not iterable(variables): - raise TypeError("variables must be an iterable") - for i in variables: - if not isinstance(i, Variable): - raise TypeError("Only a Variable instance is allowed.") - - if not iterable(funcs): - raise TypeError("funcs must be an iterable") - - for i in funcs: - if not isinstance(i, FunctionDef): - raise TypeError("Only a FunctionDef instance is allowed.") - - if not iterable(classes): - raise TypeError("classes must be an iterable") - for i in classes: - if not isinstance(i, ClassDef): - raise TypeError("Only a ClassDef instance is allowed.") - - if not iterable(overload_sets): - raise TypeError("overload_sets must be an iterable") - for i in overload_sets: - if not isinstance(i, FunctionOverloadSet): - raise TypeError("Only a FunctionOverloadSet instance is allowed.") - - NoneType = type(None) - assert isinstance(init_func, NoneType | FunctionDef) - - if not isinstance(free_func, NoneType | FunctionDef): - raise TypeError("free_func must be a FunctionDef") - - if not iterable(imports): - raise TypeError("imports must be an iterable") - imports = list(imports) - for i in classes: - imports += i.imports - imports = dict.fromkeys(imports) # for unicity and ordering - imports = tuple(imports.keys()) - - assert isinstance(is_external, bool) - - self._name = name - self._variables = variables - self._variable_inits = [None] * len(variables) - self._funcs = funcs - self._init_func = init_func - self._free_func = free_func - self._overload_sets = overload_sets - self._classes = classes - self._imports = imports - self._is_external = is_external - self._python_exports = None if python_exports is None else dict(python_exports) - - def get_name(o): - """Get the syntactic/Python name of the object""" - n = o.name - return scope.get_python_name(n) if scope else n - - self._internal_dictionary = {get_name(v): v for v in variables} - self._internal_dictionary.update({get_name(f): f for f in funcs}) - self._internal_dictionary.update({get_name(i): i for i in overload_sets}) - self._internal_dictionary.update({get_name(c): c for c in classes}) - - import_mods = { - i.source: [t.object for t in i.target if isinstance(t.object, Module)] - for i in imports - if isinstance(i, Import) - } - self._internal_dictionary.update({v: t[0] for v, t in import_mods.items() if t}) - - init_model_object(self, scope=scope) - - @property - def name(self): - """Name of the module""" - return self._name - - @property - def variables(self): - """Module global variables""" - return self._variables - - @property - def init_func(self): - """The function which initialises the module (expressions in the - python module which are executed on import) - """ - return self._init_func - - @property - def free_func(self): - """The function which frees any variables allocated in the module""" - return self._free_func - - @property - def funcs(self): - """Any functions defined in the module""" - return self._funcs - - @property - def overload_sets(self): - """Any overload_sets defined in the module""" - return self._overload_sets - - @property - def classes(self): - """Any classes defined in the module""" - return self._classes - - @property - def imports(self): - """Any imports in the module""" - return self._imports - - def get_python_exports(self, obj): - """Return ``(namespace, name)`` locations exported for one object.""" - if self._python_exports is None: - name = self.scope.get_python_name(obj.name) if self.scope else obj.name - return (((), str(name)),) - return self._python_exports.get(id(obj), ()) - - @property - def has_explicit_python_exports(self): - """Whether export locations came from an entry `.pyi` contract.""" - return self._python_exports is not None - - @property - def declarations(self): - """ - Get the declarations of all variables in the module. - - Get the declarations of all variables in the module. - """ - return [ - Declare(i, value=v, module_variable=True) - for i, v in zip(self.variables, self._variable_inits, strict=False) - ] - - @property - def body(self): - """Returns the functions, overload_sets and classes defined - in the module - """ - return self.overload_sets + self.funcs + self.classes - - def __getitem__(self, arg): - assert isinstance(arg, str) - args = arg.split(".") - result = self._internal_dictionary[args[0]] - for key in args[1:]: - result = result[key] - return result - - def __contains__(self, arg): - assert isinstance(arg, str | Symbol) - args = str(arg).split(".") - current_pos = self._internal_dictionary - key = args[0] - result = key in self._internal_dictionary - i = 1 - while i < len(args) and result: - current_pos = current_pos[key] - key = args[i] - result = key in current_pos - i += 1 - return result - - def keys(self): - """Returns the names of all objects accessible directly in this module""" - return self._internal_dictionary.keys() - - @property - def is_external(self): - """ - Indicate if the Module's definition is found elsewhere. - - This is notably the case for gFTL extensions. - """ - return self._is_external - - -class ModuleHeader: - """ - Represents the header file for a module. - - This class is simply a wrapper around a module. It is helpful to differentiate - between headers and sources when printing. - - Parameters - ---------- - module : Module - The module described by the header. - - See Also - -------- - Module : The module itself. - - Examples - -------- - >>> from x2py.ast.variable import Variable - >>> from x2py.ast.core import FunctionDefArgument, Assign, FunctionDefResult - >>> from x2py.ast.core import ClassDef, FunctionDef, Module - >>> from x2py.ast.operators import Add, Minus - >>> from x2py.codegen.models.datatypes import convert_to_literal - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> z = Variable(NumpyFloat64Type(), 'z') - >>> t = Variable(NumpyFloat64Type(), 't') - >>> a = Variable(NumpyFloat64Type(), 'a') - >>> b = Variable(NumpyFloat64Type(), 'b') - >>> body = [Assign(z,Add(x,a))] - >>> args = [FunctionDefArgument(arg) for arg in [x,y,a,b]] - >>> results = [FunctionDefResult(res) for res in [z,t]] - >>> translate = FunctionDef('translate', args, results, body) - >>> attributes = [x,y] - >>> methods = [translate] - >>> Point = ClassDef('Point', attributes, methods) - >>> incr = FunctionDef('incr', [FunctionDefArgument(x)], [FunctionDefResult(y)], [Assign(y,Add(x,convert_to_literal(1)))]) - >>> decr = FunctionDef('decr', [FunctionDefArgument(x)], [FunctionDefResult(y)], [Assign(y,Minus(x,convert_to_literal(1)))]) - >>> Module('my_module', [], [incr, decr], classes = [Point]) - >>> ModuleHeader(mod) - Module(my_module, [], [FunctionDef(), FunctionDef()], [], [ClassDef(Point, (x, y), (FunctionDef(),), [public], (), [], [])], ()) - """ - - __slots__ = ("_module",) - _attribute_nodes = ("_module",) - - def __init__(self, module): - if not isinstance(module, Module): - raise TypeError("module must be a Module") - - self._module = module - init_model_object(self) - - @property - def module(self): - return self._module - - -class FunctionCallArgument: - """ - An argument passed in a function call. - - Class describing an argument passed to a function in a - function call. - - Parameters - ---------- - value : model object - The expression passed as an argument. - keyword : str, optional - If the argument is passed by keyword then this - is that keyword. - """ - - __slots__ = ("_keyword", "_value") - _attribute_nodes = ("_value",) - - def __init__(self, value, keyword=None): - self._value = value - self._keyword = keyword - init_model_object(self) - - @property - def value(self): - """The value passed as argument""" - return self._value - - @property - def keyword(self): - """The keyword used to pass the argument""" - return self._keyword - - @property - def has_keyword(self): - """Indicates whether the argument was passed by keyword""" - return self._keyword is not None - - def __repr__(self): - if self.has_keyword: - return f"FunctionCallArgument({self.keyword} = {self.value!r})" - return f"FunctionCallArgument({self.value!r})" - - def __str__(self): - if self.has_keyword: - return f"{self.keyword} = {self.value}" - return f"{self.value}" - - -class FunctionDefArgument: - """ - model object describing the argument of a function. - - An object describing the argument of a function described - by a FunctionDef. This object stores all the information - which describes an argument but is superfluous for a Variable. - - Parameters - ---------- - name : Symbol, Variable, FunctionAddress - The name of the argument. - - value : model object, optional - The default value of the argument. - - posonly : bool, default: False - Indicates if the argument must be passed by position. - - kwonly : bool, default: False - Indicates if the argument must be passed by keyword. - - annotation : str, optional - The type annotation describing the argument. - - bound_argument : bool, default: False - Indicates if the argument is the passed object bound to a method call. - - bound_argument_position : int, optional - The position of the passed-object dummy in the native procedure - signature before wrapper normalization. - - persistent_target : bool, default: False - Indicates if the object passed as this argument becomes a target. - This argument will usually only be passed by the wrapper. - - is_vararg : bool, default: False - Indicates if the argument represents a variadic argument. - - is_kwarg : bool, default: False - Indicates if the argument represents a set of keyword arguments. - - See Also - -------- - FunctionDef : The class where these objects will be stored. - - Examples - -------- - >>> from x2py.ast.core import FunctionDefArgument - >>> n = FunctionDefArgument('n') - >>> n - n - """ - - __slots__ = ( - "_annotation", - "_bound_argument", - "_bound_argument_position", - "_is_kwarg", - "_is_vararg", - "_kwonly", - "_name", - "_persistent_target", - "_posonly", - "_value", - "_var", - "_writable", - ) - _attribute_nodes = ("_value", "_var") - - def __init__( - self, - name, - *, - value=None, - posonly=False, - kwonly=False, - annotation=None, - bound_argument=False, - bound_argument_position=None, - persistent_target=False, - is_vararg=False, - is_kwarg=False, - ): - if isinstance(name, Variable | FunctionAddress): - self._var = name - self._name = name.name - elif isinstance(name, Symbol): - self._var = name - self._name = name - else: - raise TypeError("Name must be a Symbol, Variable or FunctionAddress") - if not isinstance(bound_argument, bool): - raise TypeError("bound_argument must be a boolean") - if bound_argument_position is not None and not isinstance(bound_argument_position, int): - raise TypeError("bound_argument_position must be an integer or None") - if bound_argument_position is not None and not bound_argument: - raise ValueError("bound_argument_position requires bound_argument=True") - self._value = value - self._posonly = posonly - self._kwonly = kwonly - self._annotation = annotation - self._persistent_target = persistent_target - self._bound_argument = bound_argument - self._bound_argument_position = bound_argument_position - self._is_vararg = is_vararg - self._is_kwarg = is_kwarg - - if isinstance(name, Variable): - name.declare_as_argument() - - if isinstance(self.var, Variable): - self._writable = ( - (self.var.rank > 0 or isinstance(self.var.class_type, CustomDataType)) - and not isinstance(self.var.class_type, FinalType) - and not isinstance(self.var.class_type, TupleType) - ) - else: - # If var is not a Variable it is a FunctionAddress - self._writable = False - - init_model_object(self) - - @property - def name(self): - """The name of the argument""" - return self._name - - @property - def var(self): - """The variable representing the argument - (available after the semantic treatment) - """ - return self._var - - @property - def is_posonly(self): - """ - Indicates if the argument must be passed by position. - - Indicates if the argument must be passed by position. - """ - return self._posonly - - @property - def is_kwonly(self): - """ - Indicates if the argument must be passed by keyword. - - Indicates if the argument must be passed by keyword. - """ - return self._kwonly - - @property - def annotation(self): - """ - The argument annotation providing dtype information. - - The argument annotation providing dtype information. - """ - return self._annotation - - @property - def value(self): - """The default value of the argument""" - return self._value - - @property - def has_default(self): - """Indicates whether the argument has a default value - (if not then it must be provided) - """ - return self._value is not None - - @property - def writable(self): - """ - Indicates whether the argument may be modified by the function. - - True if the argument may be modified in the function. False if - the argument remains constant in the function. - """ - return self._writable - - def make_const(self): - """ - Indicate that the argument does not change in the function. - - Indicate that the argument does not change in the function by - modifying the writable flag. - """ - self._writable = False - - @property - def persistent_target(self): - """ - Indicate if the object passed as this argument becomes a target. - - Indicate if the object passed as this argument becomes a pointer target after - a call to the function associated with this argument. This may be the case - in class methods. - """ - return self._persistent_target - - @persistent_target.setter - def persistent_target(self, persistent_target): - self._persistent_target = persistent_target - - @property - def bound_argument(self): - """ - Indicate if the argument is bound to the function call. - - Indicate if the argument is bound to the function call. This is - the case if the argument is the first argument of a method of a - class. - """ - return self._bound_argument - - @property - def bound_argument_position(self): - """Position of the passed-object dummy in the native signature.""" - return self._bound_argument_position - - @bound_argument.setter - def bound_argument(self, bound): - if not isinstance(bound, bool): - raise TypeError("bound must be a boolean") - self._bound_argument = bound - - def __str__(self): - name = str(self.name) - if self.is_vararg: - name = f"*{name}" - if self.is_kwarg: - name = f"**{name}" - - if self.has_default: - return f"{name}={self.value}" - return name - - def __repr__(self): - name = repr(self.name) - if self.is_vararg: - name = f"*{name}" - if self.is_kwarg: - name = f"**{name}" - - if self.has_default: - return f"FunctionDefArgument({name}={self.value})" - return f"FunctionDefArgument({name})" - - @property - def is_vararg(self): - """ - True if the argument represents a variadic argument. - - True if the argument represents a variadic argument. - """ - return self._is_vararg - - @property - def is_kwarg(self): - """ - True if the argument represents a set of keyword arguments. - - True if the argument represents a set of keyword arguments. - """ - return self._is_kwarg - - -class FunctionDefResult: - """ - model object describing the result of a function. - - An object describing the result of a function described - by a FunctionDef. This object stores all the information - which describes an result but is superfluous for a Variable. - - Parameters - ---------- - var : Variable - The variable which represents the returned value. - - annotation : str, default: None - The type annotation describing the argument. - - See Also - -------- - FunctionDef : The class where these objects will be stored. - - Examples - -------- - >>> from x2py.ast.core import FunctionDefResult - >>> n = FunctionDefResult('n') - >>> n - n - """ - - __slots__ = ("_annotation", "_is_argument", "_var") - _attribute_nodes = ("_var",) - - def __init__(self, var, *, annotation=None): - self._var = var - self._annotation = annotation - - if not isinstance(var, Variable) and var is not NIL: - raise TypeError(f"Var must be a Variable not a {type(var)}") - self._is_argument = getattr(var, "is_argument", False) - - init_model_object(self) - - @property - def var(self): - """ - The variable representing the result. - - The variable which represents the result. This variable is only - available after the semantic stage. - """ - return self._var - - @property - def annotation(self): - """ - The result annotation providing dtype information. - - The annotation which provides all information about the data - types, rank, etc, necessary to fully define the result. - """ - return self._annotation - - @property - def is_argument(self): - """ - Indicates if the result was declared as an argument. - - Indicates if the result of the function was initially declared - as an argument of the same function. If this is the case then - the result may be printed simply as a writable argument. - """ - return self._is_argument - - def __len__(self): - return 0 if self.var is None else 1 - - def __repr__(self): - return f"FunctionDefResult({self.var!r})" - - def __str__(self): - return str(self.var) - - def __bool__(self): - return self.var is not NIL - - -class FunctionCall: - """ - Represents a function call in the code. - - A node which holds all information necessary to represent a function - call in the code. - - Parameters - ---------- - func : FunctionDef - The function being called. - - args : list of FunctionCallArgument - The arguments passed to the function. - - current_function : FunctionDef, default: None - The function where the call takes place. - """ - - __slots__ = ( - "_arguments", - "_class_type", - "_func_name", - "_funcdef", - "_overload_set", - "_overload_set_name", - "_shape", - ) - _attribute_nodes = ("_arguments", "_funcdef", "_overload_set") - - def __init__(self, func, args, current_function=None): - for a in args: - assert not isinstance(a, FunctionDefArgument) - # Ensure all arguments are of type FunctionCallArgument - args = [a if isinstance(a, FunctionCallArgument) else FunctionCallArgument(a) for a in args] - - # ... - if not isinstance(func, FunctionDef | FunctionOverloadSet): - raise TypeError("> expecting a FunctionDef or an FunctionOverloadSet") - - if isinstance(func, FunctionOverloadSet): - self._overload_set = func - self._overload_set_name = func.name - func = func.point(args) - else: - self._overload_set = None - - name = func.name - # ... - if current_function == name: - func.set_recursive() - - if not isinstance(args, tuple | list): - raise TypeError("args must be a list or tuple") - - # add the missing argument in the case of optional arguments - f_args = func.arguments - if len(args) != len(f_args): - # Collect dict of keywords and values (initialised as default) - f_args_dict = {a.name: (a.name, a.value) if a.has_default else None for a in f_args} - keyword_args = [] - for i, a in enumerate(args): - if a.keyword is None: - # Replace default positional arguments with provided arguments - f_args_dict[f_args[i].name] = a - else: - keyword_args = args[i:] - break - - for a in keyword_args: - # Replace default arguments with provided keyword arguments - f_args_dict[a.keyword] = a - - args = [ - (FunctionCallArgument(keyword=a[0], value=a[1]) if isinstance(a, tuple) else a) - for a in f_args_dict.values() - ] - - # Handle function as argument - arg_vals = [None if a is None else a.value for a in args] - args = [ - ( - FunctionCallArgument( - FunctionAddress(av.name, av.arguments, av.results, scope=av.scope), - keyword=a.keyword, - ) - if isinstance(av, FunctionDef) - else a - ) - for a, av in zip(args, arg_vals, strict=False) - ] - - if current_function == func.name and len(func.results) > 0 and not is_model_object(func.results): - raise RuntimeError("Recursive functions with results must declare a result variable.") - - self._funcdef = func - self._arguments = args - self._func_name = func.name - self._shape = func.results.var.shape - self._class_type = func.results.var.class_type - - init_model_object(self) - - @property - def args(self): - """List of FunctionCallArguments provided to the function call - (contains default values after semantic stage) - """ - return self._arguments - - @property - def funcdef(self): - """The function called by this function call""" - return self._funcdef - - @property - def overload_set(self): - """The interface called by this function call""" - return self._overload_set - - @property - def func_name(self): - """The name of the function called by this function call""" - return self._func_name - - @property - def overload_set_name(self): - """The name of the interface called by this function call""" - return self._overload_set_name - - @property - def is_alias(self): - """ - Check if the result of the function call is an alias type. - - Check if the result of the function call is an alias type. - """ - assert len(self._funcdef.results) == 1 - return self._funcdef.results.var.is_alias - - def __repr__(self): - args = ", ".join(str(a) for a in self.args) - return f"{self.func_name}({args})" - - -class Return: - """ - Represents a return statement in a function in the code. - - Represents a return statement in a function in the code. - - Parameters - ---------- - expr : model object - The expression to return. - - stmt : model object - Any assign statements in the case of expression return. - """ - - __slots__ = ("_expr", "_n_returns", "_stmt") - _attribute_nodes = ("_expr", "_stmt") - - def __init__(self, expr, stmt=None): - assert stmt is None or isinstance(stmt, CodeBlock) - assert expr is None or is_model_object(expr) or isinstance(expr, Symbol) - - self._expr = expr - self._stmt = stmt - - self._n_returns = 0 if expr is NIL else 1 if not hasattr(expr, "__iter__") else len(expr) - - init_model_object(self) - - @property - def expr(self): - return self._expr - - @property - def stmt(self): - return self._stmt - - def __repr__(self): - code = repr(self.stmt) + ";" if self.stmt else "" - return code + f"Return({self.expr!r})" - - -class FunctionDef: - """ - Represents a function definition. - - model object containing all the information necessary to describe a function. - This information should provide enough information to print a functionally - equivalent function in any target language. - - Parameters - ---------- - name : str - The name of the function. - - arguments : iterable of FunctionDefArgument - The arguments to the function. - - body : iterable - The body of the function. - - results : FunctionDefResult, optional - The direct outputs of the function. - - global_vars : list of Symbols - Variables which will not be passed into the function. - - cls_name : str - The alternative name of the function required for classes. - - is_static : bool - True for static functions. Needed for iso_c_binding interface. - - imports : list, tuple - A list of needed imports. - - decorators : dict - A dictionary whose keys are the names of decorators and whose values - contain their implementation. - - headers : list,tuple - A list of headers describing the function. - - is_recursive : bool - True for a function which calls itself. - - is_pure : bool - True for a function without side effect. - - is_elemental : bool - True for a function that is elemental. - - is_private : bool - True for a function that is private. - - is_header : bool - True for a function which has no body available. - - is_external : bool - True for a function which cannot be explicitly imported or renamed. - - is_imported : bool, default : False - True for a function that is imported. - - functions : list, tuple - A list of functions defined within this function. - - overload_sets : list, tuple - A list of overload_sets defined within this function. - - result_pointer_map : dict[FunctionDefResult, list[int]] - A dictionary connecting any pointer results to the index of the possible target arguments. - - docstring : str - The doc string of the function. - - bind_c_external_name : str, optional - Existing Fortran ``bind(C, name=...)`` symbol that may be called - directly when its ABI is safe. - - type_bound_name : str, optional - Native Fortran type-bound binding name used when dispatching through a - passed-object argument. - - scope : parser.scope.Scope - The scope containing all objects scoped to the inside of this function. - - See Also - -------- - FunctionDefArgument : The type used to store the arguments. - - Examples - -------- - >>> from x2py.ast.variable import Variable - >>> from x2py.ast.core import FunctionDefArgument, FunctionDefResult - >>> from x2py.ast.core import Assign, FunctionDef - >>> from x2py.ast.operators import Add - >>> from x2py.codegen.models.datatypes import convert_to_literal - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> args = [FunctionDefArgument(x)] - >>> results = [FunctionDefResult(y)] - >>> body = [Assign(y,Add(x,convert_to_literal(1)))] - >>> FunctionDef('incr', args, results, body) - FunctionDef(incr, (x,), (y,), [y := x + 1], [], [], None, False, function) - - One can also use parametrized argument, using FunctionDefArgument - - >>> from x2py.ast.core import Variable - >>> from x2py.ast.core import Assign - >>> from x2py.ast.core import FunctionDef - >>> from x2py.ast.core import FunctionDefArgument - >>> n = FunctionDefArgument('n', value=4) - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> args = [x, n] - >>> results = [y] - >>> body = [Assign(y,x+n)] - >>> FunctionDef('incr', args, results, body) - FunctionDef(incr, (x, n=4), (y,), [y := 1 + x], [], [], None, False, function, []) - """ - - __slots__ = ( - "_arguments", - "_bind_c_external_name", - "_body", - "_cls_name", - "_decorators", - "_docstring", - "_functions", - "_global_vars", - "_headers", - "_imports", - "_is_elemental", - "_is_external", - "_is_header", - "_is_imported", - "_is_private", - "_is_pure", - "_is_recursive", - "_is_semantic", - "_is_static", - "_name", - "_overload_sets", - "_result_pointer_map", - "_results", - "_type_bound_name", - ) - - _attribute_nodes = ( - "_arguments", - "_results", - "_body", - "_global_vars", - "_imports", - "_functions", - "_overload_sets", - ) - - def __init__( - self, - name, - arguments, - body, - results=None, - *, - global_vars=(), - cls_name=None, - is_static=False, - imports=(), - decorators=None, - headers=(), - is_recursive=False, - is_pure=False, - is_elemental=False, - is_private=False, - is_header=False, - is_external=False, - is_imported=False, - functions=(), - overload_sets=(), - result_pointer_map=None, - docstring=None, - bind_c_external_name=None, - type_bound_name=None, - scope=None, - ): - if result_pointer_map is None: - result_pointer_map = {} - if decorators is None: - decorators = {} - if isinstance(name, str): - name = Symbol(name) - elif isinstance(name, tuple | list): - name_ = [] - for i in name: - if isinstance(i, str): - name_.append(Symbol(i)) - else: - raise TypeError("Function name must be Symbol or string") - name = tuple(name_) - else: - raise TypeError("Function name must be Symbol or string") - - # arguments - - if not iterable(arguments): - raise TypeError("arguments must be an iterable") - if not all(isinstance(a, FunctionDefArgument) for a in arguments): - raise TypeError("arguments must be all be FunctionDefArguments") - - [a.var for a in arguments] - - # body - - if iterable(body): - body = CodeBlock(body) - assert isinstance(body, CodeBlock) - - # results - if results is None: - results = FunctionDefResult(NIL) - assert isinstance(results, FunctionDefResult) - - if cls_name and not isinstance(cls_name, str): - raise TypeError("cls_name must be a string") - - if not isinstance(is_static, bool): - raise TypeError("Expecting a boolean for is_static attribute") - - if not iterable(imports): - raise TypeError("imports must be an iterable") - - if not isinstance(decorators, dict): - raise TypeError("decorators must be a dict") - - if not isinstance(is_pure, bool): - raise TypeError("Expecting a boolean for pure") - - if not isinstance(is_elemental, bool): - raise TypeError("Expecting a boolean for elemental") - - if not isinstance(is_private, bool): - raise TypeError("Expecting a boolean for private") - - if not isinstance(is_header, bool): - raise TypeError("Expecting a boolean for header") - - if functions: - for i in functions: - if not isinstance(i, FunctionDef): - raise TypeError("Expecting a FunctionDef") - - self._name = name - self._arguments = arguments - self._results = results - self._body = body - self._global_vars = global_vars - self._cls_name = cls_name - self._is_static = is_static - self._imports = imports - self._decorators = decorators - self._headers = headers - self._is_recursive = is_recursive - self._is_pure = is_pure - self._is_elemental = is_elemental - self._is_private = is_private - self._is_header = is_header - self._is_external = is_external - self._is_imported = is_imported - self._functions = functions - self._overload_sets = overload_sets - self._result_pointer_map = result_pointer_map - self._docstring = docstring - self._bind_c_external_name = bind_c_external_name - self._type_bound_name = type_bound_name - init_model_object(self, scope=scope) - self._is_semantic = True - - @property - def name(self): - """Name of the function""" - return self._name - - @property - def arguments(self): - """List of variables which are the function arguments""" - return self._arguments - - @property - def results(self): - """List of variables which are the function results""" - return self._results - - @property - def body(self): - """ - CodeBlock containing all the statements in the function. - - Return a CodeBlock containing all the statements in the function. - """ - return self._body - - @body.setter - def body(self, body): - if iterable(body): - body = CodeBlock(body) - elif not isinstance(body, CodeBlock): - raise TypeError("body must be an iterable or a CodeBlock") - detach_model_child(self, self._body) - self._body = body - attach_model_child(self, self._body) - - @property - def local_vars(self): - """ - List of variables defined in the function. - - A list of all variables which are local to the function. This - includes arguments, results, and variables defined inside the - function. - """ - scope = self.scope - local_vars = scope.variables.values() - result_vars = [self.results.var] - return tuple( - local_var for local_var in local_vars if local_var not in result_vars and not local_var.is_argument - ) - - @property - def global_vars(self): - """List of global variables used in the function""" - return self._global_vars - - @property - def cls_name(self): - """ - String containing an alternative name for the function if it is a class method. - - If a function is a class method then in some languages an alternative name is - required. For example in Fortran a name is required for the definition of the - class in the module. This name is different from the name of the method which - is used when calling the function via the class variable. - """ - return self._cls_name - - @cls_name.setter - def cls_name(self, cls_name): - self._cls_name = cls_name - - @property - def type_bound_name(self): - """Native Fortran binding name used for type-bound dispatch.""" - return self._type_bound_name - - @property - def imports(self): - """List of imports in the function""" - return self._imports - - @property - def decorators(self): - """List of decorators applied to the function""" - return self._decorators - - @property - def headers(self): - """List of headers applied to the function""" - return self._headers - - @property - def is_recursive(self): - """Returns True if the function is recursive (i.e. calls itself) - and False otherwise""" - return self._is_recursive - - @property - def is_pure(self): - """Returns True if the function is marked as pure and False otherwise - Pure functions must not have any side effects. - In other words this means that the result must be the same no matter - how many times the function is called - e.g: - >>> a = f() - >>> a = f() - - gives the same result as - >>> a = f() - - This is notably not true for I/O functions - """ - return self._is_pure - - @property - def is_elemental(self): - """returns True if the function is marked as elemental and - False otherwise - An elemental function is a function with a single scalar operator - and a scalar return value which can also be called on an array. - When it is called on an array it returns the result of the function - called elementwise on the array""" - return self._is_elemental - - @property - def is_private(self): - """True if the function should not be exposed to - other modules. This includes the wrapper module and - means that the function cannot be used in an import - or exposed to python""" - return self._is_private - - @property - def is_header(self): - """True if the implementation of the function body - is not provided False otherwise""" - return self._is_header - - @property - def is_external(self): - """ - Indicates if the function is from an external library. - - Indicates if the function is from an external library which has no - associated imports. Such functions must be declared locally to - satisfy the compiler. For example this method returns True if the - function is exposed through a pyi file and describes a method from - a f77 module. - """ - return self._is_external - - @is_external.setter - def is_external(self, is_external): - assert isinstance(is_external, bool) - self._is_external = is_external - - @property - def is_imported(self): - """ - Indicates if the function was imported from another file. - - Indicates if the function was imported from another file. - """ - return self._is_imported - - @property - def is_inline(self): - """True if the function should be printed inline""" - return False - - @property - def is_static(self): - """ - Indicates if the function is static. - - Indicates if the function is static. - """ - return self._is_static - - @property - def is_semantic(self): - """ - Indicates if the function was created with semantic information. - - Indicates if the function has been annotated with type descriptors - in the semantic stage. - """ - return self._is_semantic - - @property - def functions(self): - """List of functions within this function""" - return self._functions - - @property - def overload_sets(self): - """List of overload_sets within this function""" - return self._overload_sets - - @property - def docstring(self): - """ - The docstring of the function. - - The docstring of the function. - """ - return self._docstring - - @property - def bind_c_external_name(self): - """Existing Fortran ``bind(C)`` external symbol for direct C calls.""" - return self._bind_c_external_name - - def set_recursive(self): - """Mark the function as a recursive function""" - self._is_recursive = True - - def clone(self, newname, **new_kwargs): - """ - Create an almost identical FunctionDef with name `newname`. - - Create an almost identical FunctionDef with name `newname`. - Additional parameters can be passed to alter the resulting - FunctionDef. - - Parameters - ---------- - newname : str - New name for the FunctionDef. - - **new_kwargs : dict - Any new keyword arguments to be passed to the new FunctionDef. - - Returns - ------- - FunctionDef - The clone of the function definition. - """ - args, kwargs = self.__getnewargs_ex__() - kwargs.update(new_kwargs) - cls = type(self) - - args = (newname, *args[1:]) - return cls(*args, **kwargs) - - def __getnewargs_ex__(self): - """ - This method returns the positional and keyword arguments used to create - an instance of this class. This is used by clone and can be used for pickling. - See https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__ - """ - args = (self._name, self._arguments, self._body) - - kwargs = { - "results": self._results, - "global_vars": self._global_vars, - "cls_name": self._cls_name, - "is_static": self._is_static, - "imports": self._imports, - "decorators": self._decorators, - "headers": self._headers, - "is_recursive": self._is_recursive, - "is_pure": self._is_pure, - "is_elemental": self._is_elemental, - "is_private": self._is_private, - "is_header": self._is_header, - "functions": self._functions, - "is_external": self._is_external, - "is_imported": self._is_imported, - "overload_sets": self._overload_sets, - "docstring": self._docstring, - "bind_c_external_name": self._bind_c_external_name, - "type_bound_name": self._type_bound_name, - "scope": self._scope, - } - return args, kwargs - - def __str__(self): - args = ", ".join(str(a) for a in self.arguments) - return f"{self.name}({args}) -> {self.results}" - - @property - def result_pointer_map(self): - """ - A dictionary connecting any pointer results to the index of the possible target arguments. - - A dictionary whose keys are FunctionDefResult objects and whose values are a list of - integers. The integers specify the position of the argument which is a target of the - FunctionDefResult. - """ - return self._result_pointer_map - - def __call__(self, *args, **kwargs): - arguments = [a if isinstance(a, FunctionCallArgument) else FunctionCallArgument(a) for a in args] - arguments += [FunctionCallArgument(a, keyword=key) for key, a in kwargs.items()] - return FunctionCall(self, arguments) - - -class FunctionOverloadSet: - """ - Class representing an interface function. - - A class representing an interface function. An interface function represents - a Python function which accepts multiple types. In low-level languages this - is a collection of functions. - - Parameters - ---------- - name : str - The name of the interface function. - - functions : iterable - The internal functions that can be accessed via the interface. - - is_argument : bool - True if the interface is used for a function argument. - - is_imported : bool - True if the interface is imported from another file. - - syntactic_node : FunctionDef, default: None - The syntactic node that is not annotated. - - Examples - -------- - >>> from x2py.ast.core import FunctionOverloadSet, FunctionDef - >>> f = FunctionDef('F', [], [], []) - >>> FunctionOverloadSet('I', [f]) - """ - - __slots__ = ( - "_functions", - "_is_argument", - "_is_imported", - "_name", - "_native_name", - "_native_names", - "_syntactic_node", - ) - _attribute_nodes = ("_functions",) - - def __init__( - self, - name, - functions, - is_argument=False, - is_imported=False, - native_name=None, - native_names=None, - syntactic_node=None, - ): - if not isinstance(name, str): - raise TypeError("Expecting an str") - - assert iterable(functions) - functions = tuple(functions) - if not functions: - raise ValueError(f"Function overload set {name!r} must contain at least one function") - if not all(isinstance(function, FunctionDef) for function in functions): - raise TypeError("Function overload set entries must be FunctionDef instances") - if type(self) is FunctionOverloadSet: - source_functions = tuple(getattr(function, "original_function", function) for function in functions) - self._validate_dispatch_signatures(name, source_functions) - - self._name = name - if native_names is None: - native_names = (native_name or name,) * len(functions) - else: - native_names = tuple(native_names) - if len(native_names) != len(functions): - raise ValueError("Function overload set native names must align with its functions") - self._native_names = native_names - self._native_name = native_name or (native_names[0] if len(set(native_names)) == 1 else name) - self._functions = functions - self._is_argument = is_argument - self._is_imported = is_imported - self._syntactic_node = syntactic_node - init_model_object(self) - - @property - def name(self): - """Name of the interface.""" - return self._name - - @property - def native_name(self): - """Native generic name used by the source-language bridge.""" - return self._native_name - - @property - def native_names(self): - """Native generic name for each concrete overload candidate.""" - return self._native_names - - def native_name_for(self, function): - """Return the native generic name associated with one candidate.""" - return self._native_names[self._functions.index(function)] - - @property - def functions(self): - """ "Functions of the interface.""" - return self._functions - - @property - def arguments(self): - """Arguments shared by every overload as seen by the generated wrapper.""" - return self._functions[0].arguments - - @staticmethod - def _dispatch_arguments(function, *, include_bound=False): - arguments = list(function.arguments) - if not include_bound and arguments and arguments[0].bound_argument: - return arguments[1:] - return arguments - - @classmethod - def _validate_dispatch_signatures(cls, name, functions): - call_shapes = [] - dispatch_keys = [] - for function in functions: - arguments = cls._dispatch_arguments(function, include_bound=name.startswith("__")) - call_shapes.append( - tuple( - ( - argument.has_default, - argument.is_kwonly, - argument.is_vararg, - argument.is_kwarg, - ) - for argument in arguments - ) - ) - dispatch_keys.append(tuple((argument.var.class_type, argument.var.rank) for argument in arguments)) - - if any(shape != call_shapes[0] for shape in call_shapes[1:]): - raise ValueError(f"Function overload set {name!r} has incompatible Python call signatures") - seen = set() - for function, key in zip(functions, dispatch_keys, strict=True): - if key in seen: - raise ValueError(f"Function overload set {name!r} has indistinguishable overload {function.name!s}") - seen.add(key) - - @property - def is_argument(self): - """True if the interface is used for a function argument.""" - return self._is_argument - - @property - def is_imported(self): - """ - Indicates if the function was imported from another file. - - Indicates if the function was imported from another file. - """ - return self._is_imported - - @property - def syntactic_node(self): - """ - The syntactic node that is not annotated. - - The syntactic node that is not annotated. - """ - return self._syntactic_node - - @property - def docstring(self): - """ - The docstring of the function. - - The docstring of the interface function. - """ - return self._functions[0].docstring - - @property - def is_semantic(self): - """ - Flag to check if the node is annotated. - - Flag to check if the node has been annotated with type descriptors - in the semantic stage. - """ - return self._functions[0].is_semantic - - @property - def is_inline(self): - """ - Flag to check if the node is inlined. - - Flag to check if the node is inlined. - """ - return self._functions[0].is_inline - - @property - def is_private(self): - """ - Indicates if the interface function is private. - - Indicates if the interface function is private. - """ - return self._functions[0].is_private - - def rename(self, newname): - """ - Rename the FunctionOverloadSet name to a newname. - - Rename the FunctionOverloadSet name to a newname. - - Parameters - ---------- - newname : str - New name for the FunctionOverloadSet. - """ - - self._name = newname - - def clone(self, newname, **new_kwargs): - """ - Create an almost identical FunctionOverloadSet with name `newname`. - - Create an almost identical FunctionOverloadSet with name `newname`. - Additional parameters can be passed to alter the resulting - FunctionDef. - - Parameters - ---------- - newname : str - New name for the FunctionOverloadSet. - - **new_kwargs : dict - Any new keyword arguments to be passed to the new FunctionOverloadSet. - - Returns - ------- - FunctionOverloadSet - The clone of the interface. - """ - - args, kwargs = self.__getnewargs_ex__() - kwargs.update(new_kwargs) - cls = type(self) - new_func = cls(*args, **kwargs) - new_func.rename(newname) - return new_func - - def __getnewargs_ex__(self): - """ - This method returns the positional and keyword arguments used to create - an instance of this class. This is used by clone and can be used for pickling. - See https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__ - """ - args = (self._name, self._functions) - - kwargs = { - "is_argument": self._is_argument, - "is_imported": self._is_imported, - "native_name": self._native_name, - "native_names": self._native_names, - "syntactic_node": self._syntactic_node, - } - return args, kwargs - - def point(self, args): - """ - Return the actual function that will be called, depending on the passed arguments. - - From the arguments passed in the function call, determine which of the FunctionDef - objects in the FunctionOverloadSet is actually called. - - Parameters - ---------- - args : tuple[model object] - The arguments passed in the function call. - - Returns - ------- - FunctionDef - The function definition which corresponds with the arguments. - """ - - def type_match(call_arg, func_arg): - """ - Check that the types of the arguments in the function and the call match. - """ - return call_arg.class_type == func_arg.class_type and (call_arg.rank == func_arg.rank) - - matches = [] - for function in self._functions: - function_args = list(function.arguments) - if len(args) != len(function_args): - continue - if all( - type_match(call_arg.value, func_arg.var) for call_arg, func_arg in zip(args, function_args, strict=True) - ): - matches.append(function) - - if not matches: - raise TypeError(f"Arguments types provided to {self.name} are incompatible") - if len(matches) > 1: - names = ", ".join(str(function.name) for function in matches) - raise TypeError(f"Arguments provided to {self.name} match multiple overloads: {names}") - return matches[0] - - @staticmethod - def native_arguments(function, args): - """Restore the native argument order after Python method binding.""" - native_args = list(args) - if not function.arguments or not function.arguments[0].bound_argument: - return native_args - position = function.arguments[0].bound_argument_position - if position in {None, 0}: - return native_args - bound_arg = native_args.pop(0) - native_args.insert(position, bound_arg) - return native_args - - def __call__(self, *args, **kwargs): - arguments = [a if isinstance(a, FunctionCallArgument) else FunctionCallArgument(a) for a in args] - arguments += [FunctionCallArgument(a, keyword=key) for key, a in kwargs.items()] - return FunctionCall(self, arguments) - - -class FunctionAddress(FunctionDef): - """ - Represents a function address. - - A function definition can have a FunctionAddress as an argument. - - Parameters - ---------- - name : str - The name of the function address. - - arguments : iterable - The arguments to the function address. - - results : iterable - The direct outputs of the function address. - - is_optional : bool - If object is an optional argument of a function [Default value: False]. - - is_kwonly : bool - If object is an argument which can only be specified using its keyword. - - is_argument : bool - If object is the argument of a function [Default value: False]. - - memory_handling : str - Must be 'heap', 'stack' or 'alias' [Default value: 'stack']. - - **kwargs : dict - Any keyword arguments which should be passed to the super class FunctionDef. - - See Also - -------- - FunctionDef - The super class from which this object derives. - - Examples - -------- - >>> from x2py.ast.core import Variable, FunctionAddress, FunctionDef - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> # a function definition can have a FunctionAddress as an argument - >>> FunctionDef('g', [FunctionAddress('f', [x], [y])], [], []) - """ - - __slots__ = ("_is_argument", "_is_kwonly", "_is_optional", "_memory_handling") - - def __init__( - self, - name, - arguments, - results, - is_optional=False, - is_kwonly=False, - is_argument=False, - memory_handling="stack", - **kwargs, - ): - super().__init__(name, arguments, body=[], results=results, **kwargs) - if not isinstance(is_argument, bool): - raise TypeError("Expecting a boolean for is_argument") - - if memory_handling not in ("heap", "alias", "stack"): - raise TypeError("Expecting 'heap', 'stack', 'alias' or None for memory_handling") - - if not isinstance(is_kwonly, bool): - raise TypeError("Expecting a boolean for kwonly") - - if not isinstance(is_optional, bool): - raise TypeError("is_optional must be a boolean.") - - self._is_optional = is_optional - self._is_kwonly = is_kwonly - self._is_argument = is_argument - self._memory_handling = memory_handling - - @property - def name(self): - return self._name - - @property - def memory_handling(self): - """Returns the memory handling of the instance of FunctionAddress""" - return self._memory_handling - - @property - def is_alias(self): - """Indicates if the instance of FunctionAddress is an alias""" - return self.memory_handling == "alias" - - @property - def is_argument(self): - return self._is_argument - - @property - def is_kwonly(self): - return self._is_kwonly - - @property - def is_optional(self): - return self._is_optional - - def __getnewargs_ex__(self): - """ - This method returns the positional and keyword arguments used to create - an instance of this class. This is used by clone and can be used for pickling. - See https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__ - """ - args, kwargs = super().__getnewargs_ex__() - args = args[:-1] # Remove body argument - kwargs["is_argument"] = self.is_argument - kwargs["is_kwonly"] = self.is_kwonly - kwargs["is_optional"] = self.is_optional - kwargs["memory_handling"] = self.memory_handling - return args, kwargs - - -class ClassDef: - """ - Represents a class definition. - - Class representing a class definition in the code. It holds all objects - which may be defined in a class including methods, overload_sets, attributes, - etc. It also handles inheritance. - - Parameters - ---------- - name : str - The name of the class. - - attributes : iterable - The attributes to the class. - - methods : iterable - Class methods. - - imports : list, tuple - A list of required imports. - - superclasses : iterable - The definition of all classes from which this class inherits. - - overload_sets : iterable - The interface methods. - - docstring : CommentBlock, optional - The doc string of the class. - - scope : Scope - The scope for the class contents. - - class_type : Type - The data type associated with this class. - - decorators : dict - A dictionary whose keys are the names of decorators and whose values - contain their implementation. - - Examples - -------- - >>> from x2py.ast.core import Variable, Assign - >>> from x2py.ast.core import ClassDef, FunctionDef - >>> x = Variable(NumpyFloat64Type(), 'x') - >>> y = Variable(NumpyFloat64Type(), 'y') - >>> z = Variable(NumpyFloat64Type(), 'z') - >>> t = Variable(NumpyFloat64Type(), 't') - >>> a = Variable(NumpyFloat64Type(), 'a') - >>> b = Variable(NumpyFloat64Type(), 'b') - >>> body = [Assign(y,x+a)] - >>> translate = FunctionDef('translate', [x,y,a,b], [z,t], body) - >>> attributes = [x,y] - >>> methods = [translate] - >>> ClassDef('Point', attributes, methods) - ClassDef(Point, (x, y), (FunctionDef(translate, (x, y, a, b), (z, t), [y := a + x], [], [], None, False, function),), [public]) - """ - - __slots__ = ( - "_attributes", - "_class_type", - "_decorators", - "_docstring", - "_imports", - "_methods", - "_name", - "_overload_sets", - "_superclasses", - ) - _attribute_nodes = ( - "_attributes", - "_methods", - "_imports", - "_overload_sets", - "_docstring", - ) - - def __init__( - self, - name, - attributes=(), - methods=(), - imports=(), - superclasses=(), - overload_sets=(), - docstring=None, - scope=None, - class_type=None, - decorators=(), - ): - # name - - if isinstance(name, str): - name = Symbol(name) - else: - raise TypeError("Class name must be Symbol or string") - - # attributes - - if not iterable(attributes): - raise TypeError("attributes must be an iterable") - attributes = tuple(attributes) - - # methods - - if not iterable(methods): - raise TypeError("methods must be an iterable") - - # imports - - if not iterable(imports): - raise TypeError("imports must be an iterable") - - if not iterable(superclasses): - raise TypeError("superclasses must be iterable") - - for s in superclasses: - if not isinstance(s, ClassDef): - raise TypeError("superclass item must be a ClassDef") - - if not isinstance(class_type, Type): - raise TypeError("class_type must be a Type") - - if not iterable(overload_sets): - raise TypeError("overload_sets must be iterable") - - imports = list(imports) - for i in methods: - imports += list(i.imports) - - imports = set(imports) # for unicity - imports = tuple(imports) - - methods = tuple(methods) - - # ... - self._name = name - self._attributes = attributes - self._methods = methods - self._imports = imports - self._superclasses = superclasses - self._overload_sets = overload_sets - self._docstring = docstring - self._class_type = class_type - self._decorators = decorators - - init_model_object(self, scope=scope) - - @property - def name(self): - """ - The name of the class. - - The name of the class. - """ - return self._name - - @property - def class_type(self): - """ - The Type of an object of the described class. - - The Type of an object of the described class. - """ - return self._class_type - - @property - def attributes(self): - """ - The attributes of a class. - - Returns a tuple containing the attributes of a ClassDef. - Each element within the tuple is of type Variable. - """ - return self._attributes - - @property - def methods(self): - return self._methods - - @property - def imports(self): - return self._imports - - @property - def superclasses(self): - """ - Get the superclasses. - - Get the class definitions for the classes from which this class - inherits. - """ - return self._superclasses - - @property - def overload_sets(self): - return self._overload_sets - - @property - def docstring(self): - """ - The docstring of the class. - - The docstring of the class. - """ - return self._docstring - - @property - def decorators(self): - """ - Dictionary mapping decorator names to descriptions. - - Dictionary mapping the names of decorators applied to the function - to descriptions of the decorator annotation. - """ - return self._decorators - - @property - def methods_as_dict(self): - """ - A dictionary containing all methods with Python names as keys. - - A dictionary containing all the methods in the class. The keys are the original - Python names of the methods. The values are the methods themselves. - """ - return {self.scope.get_python_name(m.name) if m.is_semantic else m.name: m for m in self.methods} - - def add_new_method(self, method): - """ - Add a new method to the current class. - - Add a new method to the current ClassDef. - - Parameters - ---------- - method : FunctionDef - The Method that will be added. - """ - - if not isinstance(method, FunctionDef): - raise TypeError("Method must be FunctionDef") - - attach_model_child(self, method) - self._methods += (method,) - - def add_new_overload_set(self, overload_set): - """ - Add a new interface to the current class. - - Add a new interface to the current ClassDef. - - Parameters - ---------- - interface : FunctionDef - The interface that will be added. - """ - - if not isinstance(overload_set, FunctionOverloadSet): - raise TypeError("Argument 'overload_set' must be of type FunctionOverloadSet") - attach_model_child(self, overload_set) - self._overload_sets += (overload_set,) - - def get_method(self, name, raise_error_from=None): - """ - Get the method `name` of the current class. - - Look through all methods and overload_sets of the current class to - find a method called `name`. If this class inherits from another - class, that class is also searched to ensure that the inherited - methods are available. - - Parameters - ---------- - name : str - The name of the attribute we are looking for. - - raise_error_from : model object, optional - If an error should be raised then this variable should contain - the node that the error should be raised from. This allows the - correct, line/column error information to be reported. - - Returns - ------- - FunctionDef - The definition of the method. - - Raises - ------ - ValueError - Raised if the method cannot be found. - """ - - if self.scope is not None: - # Collect translated name from scope - try: - name = self.scope.get_expected_name(name) - except RuntimeError as error: - if raise_error_from: - raise AttributeError(f"Can't find method {name} in class {self.name}") from error - return None - - try: - method = next(i for i in chain(self.methods, self.overload_sets) if i.name == name) - except StopIteration: - method = None - i = 0 - n_classes = len(self.superclasses) - while method is None and i < n_classes: - try: - method = self.superclasses[i].get_method(name, raise_error_from) - except StopIteration: - method = None - i += 1 - - if method is None and raise_error_from: - raise AttributeError(f"Can't find method {name} in class {self.name}") - - return method - - @property - def is_iterable(self): - """Returns True if the class has an iterator.""" - - names = [str(m.name) for m in self.methods] - if "__next__" in names and "__iter__" in names: - return True - if "__next__" in names: - raise ValueError("ClassDef does not contain __iter__ method") - if "__iter__" in names: - raise ValueError("ClassDef does not contain __next__ method") - return False - - @property - def is_with_construct(self): - """Returns True if the class is a with construct.""" - - names = [str(m.name) for m in self.methods] - if "__enter__" in names and "__exit__" in names: - return True - if "__enter__" in names: - raise ValueError("ClassDef does not contain __exit__ method") - if "__exit__" in names: - raise ValueError("ClassDef does not contain __enter__ method") - return False - - -class Import: - """ - Represents inclusion of dependencies in the code. - - Represents the importation of targets from another source code. This is - usually used to represent an import statement in the original code but - it is also used to import language/library specific dependencies. - - Parameters - ---------- - source : str, AsName - The module from which we import. - target : str, AsName, list, tuple - Targets to import. - ignore_at_print : bool - Indicates whether the import should be printed. - mod : Module - The module describing the source. - - Examples - -------- - >>> from x2py.ast.core import Import - >>> Import('foo') - import foo - - >>> Import('foo', 'bar') - from foo import bar - """ - - __slots__ = ("_ignore_at_print", "_source", "_source_mod", "_target") - _attribute_nodes = () - - def __init__(self, source, target=None, ignore_at_print=False, mod=None): - if source is not None: - source = Import._format(source) - - self._source = source - self._target = {} # Dict is used as Python doesn't have an ordered set - self._source_mod = mod - self._ignore_at_print = ignore_at_print - - if mod is None and isinstance(target, Module): - self._source_mod = target - - if target is None: - raise KeyError("Missing argument 'target'") - if not iterable(target): - target = [target] - - else: - for i in target: - assert isinstance(i, AsName | Module) - if isinstance(i, Module): - self._target[AsName(i, source)] = None - else: - self._target[i] = None - init_model_object(self) - - @staticmethod - def _format(i): - """ - Format a string passed to this file into a X2py object. - - Format a string passed to this file into a X2py object or confirm - that it is already correctly formatted. - - Parameters - ---------- - i : Any - The object to be formatted. - - Returns - ------- - Symbol | AsName - The formatted object. - - Raises - ------ - TypeError - Raised if the input is not a string or one of the acceptable - output types. - """ - if isinstance(i, str): - return Symbol(i) - if isinstance(i, AsName | Symbol) or (isinstance(i, Literal) and isinstance(i.dtype, StringType)): - return i - raise TypeError(f"Expecting a string, Symbol, given {type(i)}") - - @property - def target(self): - """ - Get the objects that are being imported. - - Get the objects that are being imported. - """ - return self._target.keys() - - @property - def source(self): - return self._source - - @property - def ignore(self): - return self._ignore_at_print - - @ignore.setter - def ignore(self, to_ignore): - if not isinstance(to_ignore, bool): - raise TypeError("to_ignore must be a boolean.") - self._ignore_at_print = to_ignore - - def __str__(self): - source = str(self.source) - if len(self.target) == 0: - return f"import {source}" - target = ", ".join([str(i) for i in self.target]) - return f"from {source} import {target}" - - def define_target(self, new_target): - """ - Add an additional target to the imports. - - Add an additional target to the imports. - I.e. if imp is an Import defined as: - >>> from numpy import ones - - and we call imp.define_target('cos') - then it becomes: - >>> from numpy import ones, cos - - Parameters - ---------- - new_target : str | AsName | iterable[str | AsName] - The new import target. - """ - - if iterable(new_target): - self._target.update(dict.fromkeys(new_target)) - else: - self._target[new_target] = None - - @property - def source_module(self): - """The module describing the Import source""" - return self._source_mod - - -# TODO: Should Declare have an optional init value for each var? - - -# ARA : issue-999 add is_external for external function exported through header files -class Declare: - """ - Represents a variable declaration in the code. - - Represents a variable declaration in the translated code. - - Parameters - ---------- - variable : Variable - A single variable which should be declared. - access : str, optional - Read/write access used by language printers for declaration attributes. - by_value : bool, default=False - True when the declaration must include the Fortran ``value`` ABI attribute. - value : model object, optional - The initialisation value of the variable. - static : bool, default=False - True for a static declaration of an array. - external : bool, default=False - True for a function declared through a header. - module_variable : bool, default=False - True for a variable which belongs to a module. - - Examples - -------- - >>> from x2py.ast.core import Declare, Variable - >>> Declare(Variable(NumpyInt64Type(), 'n')) - Declare(n) - """ - - __slots__ = ( - "_access", - "_by_value", - "_external", - "_module_variable", - "_static", - "_value", - "_variable", - ) - _attribute_nodes = ("_variable", "_value") - - def __init__( - self, - variable, - access=None, - by_value=False, - value=None, - static=False, - external=False, - module_variable=False, - ): - if not isinstance(variable, Variable): - raise TypeError(f"var must be of type Variable, given {variable}") - - if access not in (None, "read", "write", "readwrite", "unspecified"): - raise ValueError("access must be one of None, 'read', 'write', 'readwrite', or 'unspecified'") - - if not isinstance(by_value, bool): - raise TypeError("Expecting a boolean for by_value attribute") - - if not isinstance(static, bool): - raise TypeError("Expecting a boolean for static attribute") - - if not isinstance(external, bool): - raise TypeError("Expecting a boolean for external attribute") - - if not isinstance(module_variable, bool): - raise TypeError("Expecting a boolean for module_variable attribute") - - self._variable = variable - self._access = access - self._by_value = by_value - self._value = value - self._static = static - self._external = external - self._module_variable = module_variable - init_model_object(self) - - @property - def variable(self): - return self._variable - - @property - def access(self): - return self._access - - @property - def by_value(self): - return self._by_value - - @property - def value(self): - return self._value - - @property - def static(self): - return self._static - - @property - def external(self): - return self._external - - @property - def module_variable(self): - """Indicates whether the variable is scoped to - a module - """ - return self._module_variable - - def __repr__(self): - return f"Declare({self.variable!r})" - - -class EmptyNode: - """ - Represents an empty node in the abstract syntax tree (AST). - When a subtree is removed from the AST, we replace it with an EmptyNode - object that acts as a placeholder. Using an EmptyNode instead of None - is more explicit and avoids confusion. Further, finding a None in the AST - is signal of an internal bug. - - Parameters - ---------- - text : str - the comment line - - Examples - -------- - >>> from x2py.ast.core import EmptyNode - >>> EmptyNode() - - """ - - __slots__ = () - _attribute_nodes = () - - def __init__(self): - init_model_object(self) - - def __str__(self): - return "" - - -class Comment: - """ - Represents a Comment in the code. - - Represents a Comment in the code. - - Parameters - ---------- - text : str - The comment line. - - Examples - -------- - >>> from x2py.ast.core import Comment - >>> Comment('this is a comment') - # this is a comment - """ - - __slots__ = "_text" - _attribute_nodes = () - - def __init__(self, text): - self._text = text - init_model_object(self) - - @property - def text(self): - return self._text - - def __str__(self): - return f"# {self.text}" - - -class SeparatorComment(Comment): - """Represents a Separator Comment in the code. - - Parameters - ---------- - mark : str - marker - - Examples - -------- - >>> from x2py.ast.core import SeparatorComment - >>> SeparatorComment(n=40) - # ........................................ - """ - - __slots__ = () - - def __init__(self, n): - text = """.""" * n - super().__init__(text) - - -class CommentBlock: - """Represents a Block of Comments - - Parameters - ---------- - txt : str - - """ - - __slots__ = ("_comments", "_header") - _attribute_nodes = () - - def __init__(self, txt, header="CommentBlock"): - if not isinstance(txt, str): - raise TypeError("txt must be of type str") - txt = txt.replace('"', "") - txts = txt.split("\n") - - self._header = header - self._comments = txts - - init_model_object(self) - - @property - def comments(self): - return self._comments - - @property - def header(self): - return self._header - - @header.setter - def header(self, header): - self._header = header - - -class Pass: - """Basic class for pass instruction.""" - - __slots__ = () - _attribute_nodes = () - - def __init__(self): - init_model_object(self) - - -class IfSection: - """ - Represents one condition and code block in an if statement. - - Represents a condition and associated code block - in an if statement in the code. - - Parameters - ---------- - cond : model object - A boolean expression indicating whether or not the block - should be executed. - body : CodeBlock - The code to be executed if the condition is satisfied. - - Examples - -------- - >>> from x2py.ast.internals import Symbol - >>> from x2py.ast.core import Assign, IfSection, CodeBlock - >>> n = Symbol('n') - >>> IfSection((n>1), CodeBlock([Assign(n,n-1)])) - IfSection((n>1), CodeBlock([Assign(n,n-1)])) - """ - - __slots__ = ("_block", "_condition") - _attribute_nodes = ("_condition", "_block") - - def __init__(self, cond, body): - assert cond.dtype is NumpyBoolType() - - if isinstance(body, list | tuple): - body = CodeBlock(body) - elif isinstance(body, CodeBlock): - body = body - else: - raise TypeError("body is not iterable or CodeBlock") - - self._condition = cond - self._block = body - - init_model_object(self) - - @property - def condition(self): - return self._condition - - @property - def body(self): - return self._block - - def __iter__(self): - return iter((self.condition, self.body)) - - def __str__(self): - return f"IfSec({self.condition}, {self.body})" - - -class If: - """ - Represents an if statement in the code. - - Represents an if statement in the code. - - Parameters - ---------- - *args : IfSection - All arguments are sections of the complete If block. - - Examples - -------- - >>> from x2py.ast.internals import Symbol - >>> from x2py.ast.core import Assign, If - >>> n = Symbol('n') - >>> i1 = IfSection((n>1), [Assign(n,n-1)]) - >>> i2 = IfSection(True, [Assign(n,n+1)]) - >>> If(i1, i2) - If(IfSection((n>1), [Assign(n,n-1)]), IfSection(True, [Assign(n,n+1)])) - """ - - __slots__ = ("_blocks",) - _attribute_nodes = ("_blocks",) - - # TODO add type check in the semantic stage - - def __init__(self, *args): - if not all(isinstance(a, IfSection) for a in args): - raise TypeError("An If must be composed of IfSections") - - self._blocks = args - - init_model_object(self) - - @property - def blocks(self): - """ - The IfSection blocks inside this if. - - The IfSection blocks inside this if. - """ - return self._blocks - - def __str__(self): - blocks = ",".join(str(b) for b in self.blocks) - return f"If({blocks})" - - -class CaseSection: - """Represents one section in a select-case statement.""" - - __slots__ = ("_body", "_label") - _attribute_nodes = ("_label", "_body") - - def __init__(self, label, body): - if isinstance(body, list | tuple): - body = CodeBlock(body) - elif not isinstance(body, CodeBlock): - raise TypeError("body is not iterable or CodeBlock") - self._label = label - self._body = body - init_model_object(self) - - @property - def label(self): - return self._label - - @property - def body(self): - return self._body - - -class SelectCase: - """Represents a Fortran-style select-case statement.""" - - __slots__ = ("_expr", "_sections") - _attribute_nodes = ("_expr", "_sections") - - def __init__(self, expr, *sections): - if not sections or not all(isinstance(section, CaseSection) for section in sections): - raise TypeError("SelectCase must contain CaseSection objects") - self._expr = expr - self._sections = sections - init_model_object(self) - - @property - def expr(self): - return self._expr - - @property - def sections(self): - return self._sections - - -# ======================================================================================== -class Function: - """ - Abstract class for function calls translated to X2py objects. - - A subclass of this base class represents calls to a specific internal - function of X2py, which may be simplified at a later stage, or made - available in the target language when printing the generated code. - - Parameters - ---------- - *args : iterable - The arguments passed to the function call. - """ - - __slots__ = ("_args",) - _attribute_nodes = ("_args",) - name = None - - def __init__(self, *args): - self._args = tuple(args) - init_model_object(self) - - @property - def args(self): - """ - The arguments passed to the function. - - Tuple containing all the arguments passed to the function call. - """ - return self._args - - @property - def is_elemental(self): - """ - Whether the function acts elementwise on an array argument. - - Boolean indicating whether the (scalar) function should be called - elementwise on an array argument. Here we set the default to False. - """ - return False - - -class ArraySize(Function): - """ - Gets the total number of elements in an array. - - Class representing a call to a function which would return - the total number of elements in a multi-dimensional array. - - Parameters - ---------- - arg : model object - An array of unknown size. - """ - - __slots__ = () - name = "size" - - _shape = None - _class_type = NumpyInt64Type() - - def __init__(self, arg): - super().__init__(arg) - - @property - def arg(self): - """ - Object whose size is investigated. - - The argument of the function call, i.e. the object whose size is - investigated. - """ - return self._args[0] - - def __str__(self): - return f"Size({self.arg})" - - def __eq__(self, other): - if isinstance(other, ArraySize): - return self.arg == other.arg - return False - - -class ArrayShapeElement(Function): - """ - Gets the size of one array dimension. - """ - - __slots__ = () - name = "shape" - - _shape = None - _class_type = NumpyInt64Type() - - def __init__(self, arg, index): - super().__init__(arg, index) - - @property - def arg(self): - """Object whose shape is investigated.""" - return self._args[0] - - @property - def index(self): - """Zero-based dimension index.""" - return self._args[1] - - -class ArrayLowerBound(Function): - """Gets the lower bound of one array dimension.""" - - __slots__ = () - name = "lbound" - - _shape = None - _class_type = NumpyInt64Type() - - def __init__(self, arg, index): - super().__init__(arg, index) - - @property - def arg(self): - """Object whose lower bound is investigated.""" - return self._args[0] - - @property - def index(self): - """Zero-based dimension index.""" - return self._args[1] - - -class ArrayAllocated(Function): - """ - Tests whether an allocatable array is allocated. - """ - - __slots__ = () - name = "allocated" - - _shape = None - _class_type = NumpyBoolType() - - def __init__(self, arg): - super().__init__(arg) - - @property - def arg(self): - """Object whose allocation status is investigated.""" - return self._args[0] - - -class ArrayAssociated(Function): - """ - Tests whether a Fortran pointer array is associated. - """ - - __slots__ = () - name = "associated" - - _shape = None - _class_type = NumpyBoolType() - - def __init__(self, arg): - super().__init__(arg) - - @property - def arg(self): - """Object whose pointer association status is investigated.""" - return self._args[0] - - -class ArrayContiguous(Function): - """Tests whether an array occupies contiguous native storage.""" - - __slots__ = () - name = "is_contiguous" - - _shape = None - _class_type = NumpyBoolType() - - def __init__(self, arg): - super().__init__(arg) - - @property - def arg(self): - """Object whose storage contiguity is investigated.""" - return self._args[0] - - -class Slice: - """ - Represents a slice in the code. - - An object of this class represents the slicing of a Numpy array along one of - its dimensions. In most cases this corresponds to a Python slice in the user - code, where it is represented by a `python.ast.Slice` object. - - In addition, at the wrapper and code generation stages, an integer index - `i` used to create a view of a Numpy array is converted to an object - `Slice(i, i+1, 1)`. This allows using C - variadic arguments in the function `array_slicing` (in file - x2py/stdlib/ndarrays/ndarrays.c). - - Parameters - ---------- - start : Symbol or int - Starting index. - - stop : Symbol or int - Ending index. - - step : Symbol or int, default=None - The step between indices. - - Examples - -------- - >>> from x2py.ast.internals import Slice, symbols - >>> start, end, step = symbols('start, stop, step') - >>> Slice(start, stop) - start : stop - >>> Slice(None, stop) - : stop - >>> Slice(start, None) - start : - >>> Slice(start, stop, step) - start : stop : step - """ - - __slots__ = ("_start", "_step", "_stop") - _attribute_nodes = ("_start", "_stop", "_step") - - def __init__(self, start, stop, step=None): - self._start = start - self._stop = stop - self._step = step - init_model_object(self) - - assert start is None or isinstance(getattr(start.dtype, "primitive_type", None), PrimitiveIntegerType) - assert stop is None or isinstance(getattr(stop.dtype, "primitive_type", None), PrimitiveIntegerType) - assert step is None or isinstance(getattr(step.dtype, "primitive_type", None), PrimitiveIntegerType) - - @property - def start(self): - """Index where the slicing of the object starts""" - return self._start - - @property - def stop(self): - """Index until which the slicing takes place""" - return self._stop - - @property - def step(self): - """The difference between each index of the - objects in the slice - """ - return self._step - - def __str__(self): - start = "" if self.start is None else str(self.start) - stop = "" if self.stop is None else str(self.stop) - return f"{start} : {stop} : {self.step}" - - -# ======================================================================================================= -class PythonTuple: - """ - Class representing a call to Python's native (,) function which creates tuples. - - Class representing a call to Python's native (,) function - which initialises a literal tuple. - - Parameters - ---------- - *args : tuple of model object - The arguments passed to the tuple function. - class_type : Type, optional - The final type of the tuple. This is necessary to create a printable - empty tuple. Otherwise it is not used. - """ - - __slots__ = ("_args", "_class_type", "_is_homogeneous", "_shape") - _iterable = True - _attribute_nodes = ("_args",) - - def __init__(self, *args, class_type=None): - self._args = args - init_model_object(self) - - self._is_homogeneous = True - if len(args) == 0: - self._class_type = GenericType - self._shape = (convert_to_literal(0),) - return - - self._shape = (convert_to_literal(len(args)),) - self._class_type = args[0]._class_type - - def __len__(self): - return len(self._args) - - def __str__(self): - args = ", ".join(str(a) for a in self) - return f"({args})" - - def __repr__(self): - args = ", ".join(str(a) for a in self) - return f"PythonTuple({args})" - - @property - def is_homogeneous(self): - """ - Indicates whether the tuple is homogeneous or inhomogeneous. - - Indicates whether all elements of the tuple have the same dtype, - rank, etc (homogenous) or if these values can vary (inhomogeneous). - """ - return self._is_homogeneous - - @property - def args(self): - """ - Arguments of the tuple. - - The arguments that were used to initialise the tuple. - """ - return self._args - - -def get_direct_assignment(obj): - """Return the assignment that directly consumes ``obj``, if present.""" - return _find_direct_model_parent(obj, (Assign, AliasAssign)) - - -def get_direct_function_argument(obj): - """Return the function argument that directly contains ``obj``, if present.""" - return _find_direct_model_parent(obj, FunctionDefArgument) - - -def get_direct_overload_set(obj): - """Return the interface that directly contains ``obj``, if present.""" - return _find_direct_model_parent(obj, FunctionOverloadSet) - - -def get_direct_module(obj): - """Return the module that directly contains ``obj``, if present.""" - return _find_direct_model_parent(obj, Module) - - -def get_enclosing_class(obj): - """Return the first class containing ``obj``, if present.""" - return _find_model_parent(obj, ClassDef) - - -def get_enclosing_function(obj): - """Return the first function containing ``obj``, if present.""" - return _find_model_parent(obj, FunctionDef) - - -def get_enclosing_module(obj): - """Return the first module containing ``obj``, if present.""" - return _find_model_parent(obj, Module) - - -def is_in_overload_set(obj): - """Return whether ``obj`` belongs to an interface outside a function call.""" - return _find_model_parent(obj, FunctionOverloadSet, excluded_types=(FunctionCall,)) is not None - - -for _model_cls in ( - Operator, - Variable, - IndexedElement, - AsName, - Assign, - Allocate, - Deallocate, - CodeBlock, - AliasAssign, - Module, - ModuleHeader, - FunctionCallArgument, - FunctionDefArgument, - FunctionDefResult, - FunctionCall, - Return, - FunctionDef, - FunctionOverloadSet, - ClassDef, - Import, - Declare, - EmptyNode, - Comment, - CommentBlock, - Pass, - IfSection, - If, - Function, - ArrayAllocated, - ArrayAssociated, - ArrayContiguous, - ArrayLowerBound, - ArrayShapeElement, - FortranCharacterLength, - Slice, - PythonTuple, -): - register_model_class(_model_cls) - -del _model_cls diff --git a/x2py/codegen/models/datatypes.py b/x2py/codegen/models/datatypes.py deleted file mode 100644 index be6cb444a..000000000 --- a/x2py/codegen/models/datatypes.py +++ /dev/null @@ -1,1645 +0,0 @@ -# pylint: disable=no-member, protected-access - - -""" -Classes and methods that handle supported datatypes in C/Fortran. -""" - -from functools import lru_cache -from types import GeneratorType -import numpy - -from x2py.utilities.metaclasses import Singleton - - -dict_keys = type({}.keys()) -dict_values = type({}.values()) - - -def iterable(value): - """Return whether a value is a supported model collection.""" - return isinstance(value, list | tuple | dict_keys | dict_values | set | GeneratorType) - - -_MODEL_CLASSES = set() -_MODEL_STATE = {} - - -def is_model_object(value): - """Return whether ``value`` participates in codegen model relationships.""" - return id(value) in _MODEL_STATE or is_model_class(type(value)) - - -def is_model_class(value): - """Return whether ``value`` is a class for model relationship objects.""" - return isinstance(value, type) and any(c in _MODEL_CLASSES for c in value.__mro__) - - -def _model_state(obj): - return _MODEL_STATE.setdefault(id(obj), {"parents": [], "scope": None}) - - -def _ignore_model_child(value): - return value is None or isinstance(value, type) or getattr(value, "_model_immutable", False) - - -def init_model_object(obj, scope=None): - """Initialize relationship bookkeeping for one codegen model object.""" - state = _MODEL_STATE[id(obj)] = { - "parents": [], - "scope": scope, - } - - for attribute_name in getattr(type(obj), "_attribute_nodes", ()): - child = getattr(obj, attribute_name) - if _ignore_model_child(child): - continue - - if isinstance(child, int | float | complex | str | bool): - child = convert_to_literal(child) - setattr(obj, attribute_name, child) - elif iterable(child): - size = len(child) - child = tuple( - item - if not isinstance(item, int | float | complex | str | bool) or _ignore_model_child(item) - else convert_to_literal(item) - for item in child - if not iterable(item) - ) - if len(child) != size: - raise TypeError("model child cannot contain nested collections") - setattr(obj, attribute_name, child) - elif not is_model_object(child): - raise TypeError(f"model child must be a model object or collection, not {type(child)}") - - children = child if isinstance(child, tuple) else (child,) - for item in children: - if not _ignore_model_child(item) and is_model_object(item): - attach_model_child(obj, item) - - return state - - -def attach_model_child(parent, child): - """Record that ``child`` is directly contained by ``parent``.""" - _model_state(child)["parents"].append(parent) - - -def detach_model_child(parent, child): - """Remove a direct containment link from ``parent`` to ``child``.""" - _model_state(child)["parents"].remove(parent) - - -def _find_direct_model_parent(obj, parent_type): - """Return the first direct parent of ``obj`` with the requested type.""" - return next( - (parent for parent in _model_state(obj)["parents"] if isinstance(parent, parent_type)), - None, - ) - - -def _find_model_parent(obj, parent_type, excluded_types=()): - """Return the first matching parent reachable from ``obj``.""" - visited = set() - - def find(current): - current_id = id(current) - if current_id in visited: - return None - visited.add(current_id) - - parents = _model_state(current)["parents"] - direct_parent = next( - ( - parent - for parent in parents - if isinstance(parent, parent_type) and not isinstance(parent, excluded_types) - ), - None, - ) - if direct_parent is not None: - return direct_parent - - for parent in parents: - if _ignore_model_child(parent) or isinstance(parent, excluded_types) or not is_model_object(parent): - continue - result = find(parent) - if result is not None: - return result - return None - - return find(obj) - - -def _shape(obj): - return obj._shape - - -def _rank(obj): - return obj.class_type.rank - - -def _dtype(obj): - return obj.class_type.datatype - - -def _order(obj): - return obj.class_type.order - - -def _class_type(obj): - return obj._class_type - - -def _static_type(cls): - return cls._static_type - - -def _scope(obj): - return _model_state(obj)["scope"] - - -def register_model_class(cls): - """Register a codegen model class without changing its inheritance.""" - _MODEL_CLASSES.add(cls) - if "shape" not in cls.__dict__: - cls.shape = property(_shape) - if "rank" not in cls.__dict__: - cls.rank = property(_rank) - if "dtype" not in cls.__dict__: - cls.dtype = property(_dtype) - if "order" not in cls.__dict__: - cls.order = property(_order) - if "class_type" not in cls.__dict__: - cls.class_type = property(_class_type) - if "static_type" not in cls.__dict__: - cls.static_type = classmethod(_static_type) - if "scope" not in cls.__dict__: - cls.scope = property(_scope) - return cls - - -__all__ = ( - "NIL", - # ---------- Functions ------------------- - "Cast", - # ------------ Fixed size types ------------ - "CharType", - # ------------ Container types ------------ - "CustomDataType", - "DataTypeFactory", - # ------------ Modifying types ------------ - "FinalType", - "FixedSizeNumericType", - # ------------ Super classes ------------ - "FixedSizeType", - "GenericType", - # -----------------literals----------------- - "Literal", - # ---------------numpy types -------------- - "NumpyBoolType", - "NumpyComplex64Type", - "NumpyComplex128Type", - "NumpyComplex256Type", - "NumpyFloat32Type", - "NumpyFloat64Type", - "NumpyFloat128Type", - "NumpyInt8Type", - "NumpyInt16Type", - "NumpyInt32Type", - "NumpyInt64Type", - "NumpyIntType", - "NumpyNDArrayType", - "NumpyNumericType", - # ------------ Primitive types ------------ - "PrimitiveBooleanType", - "PrimitiveCharacterType", - "PrimitiveComplexType", - "PrimitiveFloatingPointType", - "PrimitiveIntegerType", - "PrimitiveType", - "StringType", - "SymbolicType", - "TupleType", - "Type", - "VoidType", - "attach_model_child", - "cast_to", - "convert_to_literal", - "detach_model_child", - "init_model_object", - "is_model_class", - "is_model_object", - "iterable", - "register_model_class", -) - - -# ============================================================================== -class PrimitiveType(metaclass=Singleton): - """ - Base class representing types of datatypes. - - The base class representing the category of datatype to which a FixedSizeType - may belong (e.g. integer, floating point). - """ - - __slots__ = () - _name = "__UNDEFINED__" - - def __init__(self): # pylint: disable=useless-parent-delegation - # This __init__ function is required so the Singleton can - # always detect a signature - super().__init__() - - def __str__(self): - return self._name - - -class PrimitiveBooleanType(PrimitiveType): - """ - Class representing a boolean datatype. - - Class representing a boolean datatype. - """ - - __slots__ = () - _name = "boolean" - - -class PrimitiveIntegerType(PrimitiveType): - """ - Class representing an integer datatype. - - Class representing an integer datatype. - """ - - __slots__ = () - _name = "integer" - - -class PrimitiveFloatingPointType(PrimitiveType): - """ - Class representing a floating point datatype. - - Class representing a floating point datatype. - """ - - __slots__ = () - _name = "floating point" - - -class PrimitiveComplexType(PrimitiveType): - """ - Class representing a complex datatype. - - Class representing a complex datatype. - """ - - __slots__ = () - _name = "complex" - - -class PrimitiveCharacterType(PrimitiveType): - """ - Class representing a character datatype. - - Class representing a character datatype. - """ - - __slots__ = () - _name = "character" - - -# ============================================================================== - - -class Type(metaclass=Singleton): - """ - Base class representing the type of an object. - - Base class representing the type of an object from which all - types must inherit. A type must contain enough information to - describe the declaration type in a low-level language. - - Types contain an addition operator. The operator indicates the type that - is expected when calling an arithmetic operator on objects of these types. - - Where applicable, types also contain an and operator. The operator indicates the type that - is expected when calling a bitwise comparison operator on objects of these types. - - A type also contains an attribute _name which can be useful to examine - the type. - """ - - __slots__ = () - - @property - def name(self): - """ - Get the name of the x2py type. - - Get the name of the x2py type. - """ - return self._name - - def __init__(self): # pylint: disable=useless-parent-delegation - # This __init__ function is required so the Singleton can - # always detect a signature - super().__init__() - - def __str__(self): - return self._name - - def switch_basic_type(self, new_type): - """ - Change the basic type to the new type. - - Change the basic type to the new type. In the case of a FixedSizeType the - switch will replace the type completely, directly returning the new type. - Array types override this method to keep the array container and switch - the element type. - - Parameters - ---------- - new_type : Type - The new basic type. - - Returns - ------- - Type - The new type. - """ - raise NotImplementedError(f"switch_basic_type not implemented for {type(self)}") - - def shape_is_compatible(self, shape): - """ - Check if the provided shape is compatible with the datatype. - - Check if the provided shape is compatible with the format expected for - this datatype. - - Parameters - ---------- - shape : Any - The proposed shape. - - Returns - ------- - bool - True if the shape is acceptable, False otherwise. - """ - return shape is None - - -# ============================================================================== -class FinalType: - """ - A class to get Type subclasses describing constant values. - - A class to get Type subclasses describing constant values. - """ - - __slots__ = () - - @classmethod - @lru_cache - def get_new(cls, underlying_type): - """ - Get the parameterised Final type. - - Get the parameterised Final type Final[underlying_type]. - - Parameters - ---------- - underlying_type : Type - The type which is characterised as final. - """ - assert isinstance(underlying_type, Type) - if isinstance(underlying_type, FinalType): - return underlying_type - - type_class = type(underlying_type) - - def __init__(self): - self._underlying_type = underlying_type - type(underlying_type).__init__(self) - - def __hash__(self): - return type_class.__hash__(underlying_type) - - def __eq__(self, other): - return type_class.__eq__(underlying_type, other) - - def get_underlying_type(self): - """ - Get the type that is indicated as const. - - Get the type that is indicated as const. - """ - return self._underlying_type - - return type( - f"Final[{type_class.__name__}]", - ( - FinalType, - type_class, - ), - { - "__init__": __init__, - "__hash__": __hash__, - "__eq__": __eq__, - "underlying_type": property(get_underlying_type), - }, - )() - - def __str__(self): - return f"Final[{self._underlying_type}]" - - -# ============================================================================== - - -class FixedSizeType(Type): - """ - Base class representing a built-in scalar datatype. - - The base class representing a built-in scalar datatype which can be - represented in memory. E.g. int32, int64. - """ - - __slots__ = () - - @property - def datatype(self): - """ - The datatype of the object. - - The datatype of the object. - """ - return self - - @property - def primitive_type(self): - """ - The datatype category of the object. - - The datatype category of the object (e.g. integer, floating point). - """ - return self._primitive_type - - @property - def rank(self): - """ - Number of dimensions of the object. - - Number of dimensions of the object. If the object is a scalar then - this is equal to 0. - """ - return 0 - - @property - def order(self): - """ - The data layout ordering in memory. - - Indicates whether the data is stored in row-major ('C') or column-major - ('F') format. This is only relevant if rank > 1. When it is not relevant - this function returns None. - """ - return None - - def switch_basic_type(self, new_type): - """ - Change the basic type to the new type. - - Change the basic type to the new type. In the case of a FixedSizeType the - switch will replace the type completely, directly returning the new type. - - Parameters - ---------- - new_type : FixedSizeType - The new basic type. - - Returns - ------- - Type - The new type. - """ - assert isinstance(new_type, FixedSizeType) - return new_type - - -class FixedSizeNumericType(FixedSizeType): - """ - Base class representing a scalar numeric datatype. - - The base class representing a scalar numeric datatype which can be - represented in memory. E.g. int32, int64. - """ - - __slots__ = () - - @property - def precision(self): - """ - Precision of the datatype of the object. - - The precision of the datatype of the object. This number is related to the - number of bytes that the datatype takes up in memory. For basic types the - number is equivalent to the number of bytes in memory (e.g. `float64` has - precision = 8 as it takes up 8 bytes), however for less simple types the - connection is less trivial. For example `complex128` has precision = 8 as - it is comprised of two `float64` objects (which have precision=8). - It should be noted that this is not the convention chosen by NumPy (in NumPy - a `complex128` is so named because `16*8=precision*bits_in_a_byte=128`). - - The precision in X2py is equivalent to the `kind` parameter in Fortran. - """ - return self._precision - - -class VoidType(FixedSizeType): - """ - Class representing a void datatype. - - Class representing a void datatype. This class is especially useful - in the C-Python wrapper when a `void*` type is needed to collect - pointers from Fortran. - """ - - __slots__ = () - _name = "void" - _primitive_type = None - - -class GenericType(FixedSizeType): - """ - Class representing a generic datatype. - - Class representing a generic datatype. This datatype is - useful for describing an argument which can accept any type (e.g. MPI arguments). - """ - - __slots__ = () - _name = "Generic" - _primitive_type = None - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __add__(self, other): - return other - - def __eq__(self, other): - return True - - def __hash__(self): - return hash(self.__class__) - - -class SymbolicType(FixedSizeType): - """ - Class representing the datatype of a placeholder symbol. - - Class representing the datatype of a placeholder symbol. This type should - be used for objects which will not appear in the generated code but are - used to identify objects (e.g. Type aliases). - """ - - __slots__ = () - _name = "Symbolic" - _primitive_type = None - - -class CharType(FixedSizeType): - """ - Class representing a char type in C/Fortran. - - Class representing a char type in C/Fortran. This datatype is - useful for describing strings. - """ - - __slots__ = () - _name = "char" - _primitive_type = PrimitiveCharacterType() - - -class TupleType: - """ - Base class representing tuple datatypes. - - The class from which tuple datatypes must inherit. - """ - - __slots__ = () - _name = "tuple" - - -# ============================================================================== - - -class StringType(Type): - """ - Class representing Python's native string type. - - Class representing Python's native string type. - """ - - __slots__ = () - _name = "str" - - @property - def datatype(self): - """ - The datatype of the object. - - The datatype of the object. - """ - return self - - def __str__(self): - return "str" - - @property - def primitive_type(self): - """ - The datatype category of elements of the object. - - The datatype category of elements of the object (e.g. integer, floating point). - """ - return self - - @property - def rank(self): - """ - Number of dimensions of the object. - - Number of dimensions of the object. If the object is a scalar then - this is equal to 0. - """ - return 1 - - @property - def container_rank(self): - """ - Number of dimensions of the container. - - Number of dimensions of the object described by the container. This is - equal to the number of values required to index an element of this container. - """ - return 1 - - def shape_is_compatible(self, shape): - """Check if the provided shape is compatible with a string.""" - return isinstance(shape, tuple) and len(shape) == self.container_rank - - @property - def order(self): - """ - The data layout ordering in memory. - - Indicates whether the data is stored in row-major ('C') or column-major - ('F') format. This is only relevant if rank > 1. When it is not relevant - this function returns None. - """ - return None - - @property - def element_type(self): - """ - The type of elements of the object. - - The Type describing an element of the container. - """ - return CharType() - - def __eq__(self, other): - return isinstance(other, self.__class__) - - def __hash__(self): - return hash(self.__class__) - - -# ============================================================================== - - -class CustomDataType(Type): - """ - Class from which user-defined types inherit. - - A general class for custom data types which is used as a - base class when a user defines their own type using classes. - """ - - __slots__ = () - - @property - def datatype(self): - """ - The datatype of the object. - - The datatype of the object. - """ - return self - - @property - def rank(self): - """ - Number of dimensions of the object. - - Number of dimensions of the object. If the object is a scalar then - this is equal to 0. - """ - return 0 - - @property - def order(self): - """ - The data layout ordering in memory. - - Indicates whether the data is stored in row-major ('C') or column-major - ('F') format. This is only relevant if rank > 1. When it is not relevant - this function returns None. - """ - return None - - -# ============================================================================== -def DataTypeFactory(ll_name, python_name, argnames=(), *, BaseClass=CustomDataType): - """ - Create a new data class. - - Create a new data class which sub-classes a DataType. This provides - a new data type which can be used, for example, for class types. - - Parameters - ---------- - ll_name : str - The low-level name of the new class. - - python_name : str - The original name of the new class matching the name used in Python. - - argnames : iterable[str] - A list of all the arguments for the new class. - This can be used to create classes which are parametrised by a type. - - BaseClass : type inheriting from DataType - The class from which the new type will be sub-classed. - - Returns - ------- - type - A new DataType class. - """ - - def class_init_func(self, **kwargs): - """ - The __init__ function for the new CustomDataType class. - """ - for key, value in kwargs.items(): - # here, the argnames variable is the one passed to the - # DataTypeFactory call - if key not in argnames: - raise TypeError(f"Argument {key} not valid for {self.__class__.__name__}") - setattr(self, key, value) - - BaseClass.__init__(self) # pylint: disable=unnecessary-dunder-call - - assert iterable(argnames) - assert all(isinstance(a, str) for a in argnames) - - def class_name_func(self): - """ - The name function for the new CustomDataType class. - """ - if argnames: - param = ", ".join(str(getattr(self, a)) for a in argnames) - return f"{self._name}[{param}]" # pylint: disable=protected-access - return self._name # pylint: disable=protected-access - - def low_level_name(self): - """ - The low_level_name function for the new CustomDataType class. - This describes the name that will be used in the low-level language. - """ - return ll_name - - return type( - python_name, - (BaseClass,), - { - "__init__": class_init_func, - "name": property(class_name_func), - "_name": python_name, - "low_level_name": property(low_level_name), - }, - ) - - -# ======================================================================================== -class NumpyNumericType(FixedSizeNumericType): - """ - Base class representing a scalar numeric datatype defined in the numpy module. - - Base class representing a scalar numeric datatype defined in the numpy module. - """ - - __slots__ = () - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __add__(self, other): - try: - return original_type_to_x2py_type[ - numpy.result_type( - x2py_type_to_original_type[self](), - x2py_type_to_original_type[other](), - ).type - ] - except KeyError: - return NotImplemented - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __radd__(self, other): - return self.__add__(other) - - def __eq__(self, other): - if other is self: - return True - if isinstance(other, NumpyNumericType): - return False - if isinstance(other, FixedSizeNumericType): - return other.primitive_type == self.primitive_type and other.precision == self.precision - return NotImplemented - - def __hash__(self): - return hash(f"numpy.{self}") - - -# ============================================================================== - - -class NumpyBoolType(NumpyNumericType): - """ - Class representing NumPy's bool_ type. - - Class representing NumPy's bool_ type. - """ - - __slots__ = () - _name = "numpy.bool_" - _primitive_type = PrimitiveBooleanType() - _precision = -1 - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __add__(self, other): - if isinstance(other, NumpyBoolType): - return NumpyInt64Type() - if isinstance(other, NumpyNumericType): - return other - return NotImplemented - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __and__(self, other): - if isinstance(other, NumpyBoolType): - return self - if isinstance(other, NumpyNumericType): - return other - return NotImplemented - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __rand__(self, other): - return self.__and__(other) - - -# ============================================================================== - - -class NumpyIntType(NumpyNumericType): - """ - Super class representing NumPy's integer types. - - Super class representing NumPy's integer types. - """ - - __slots__ = () - _primitive_type = PrimitiveIntegerType() - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __and__(self, other): - if isinstance(other, NumpyBoolType): - return self - if isinstance(other, FixedSizeNumericType): - precision = max(self.precision, other.precision) - return numpy_precision_map[(self._primitive_type, precision)] - return NotImplemented - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __rand__(self, other): - if isinstance(other, NumpyBoolType): - return self - if isinstance(other, FixedSizeNumericType): - precision = max(self.precision, other.precision) - return numpy_precision_map[(self._primitive_type, precision)] - return NotImplemented - - -class NumpyInt8Type(NumpyIntType): - """ - Class representing NumPy's int8 type. - - Class representing NumPy's int8 type. - """ - - __slots__ = () - _name = "numpy.int8" - _precision = 1 - - -class NumpyInt16Type(NumpyIntType): - """ - Class representing NumPy's int16 type. - - Class representing NumPy's int16 type. - """ - - __slots__ = () - _name = "numpy.int16" - _precision = 2 - - -class NumpyInt32Type(NumpyIntType): - """ - Class representing NumPy's int32 type. - - Class representing NumPy's int32 type. - """ - - __slots__ = () - _name = "numpy.int32" - _precision = 4 - - -class NumpyInt64Type(NumpyIntType): - """ - Class representing NumPy's int64 type. - - Class representing NumPy's int64 type. - """ - - __slots__ = () - _name = "numpy.int64" - _precision = 8 - - -# ============================================================================== - - -class NumpyFloat32Type(NumpyNumericType): - """ - Class representing NumPy's float32 type. - - Class representing NumPy's float32 type. - """ - - __slots__ = () - _name = "numpy.float32" - _primitive_type = PrimitiveFloatingPointType() - _precision = 4 - - -class NumpyFloat64Type(NumpyNumericType): - """ - Class representing NumPy's float64 type. - - Class representing NumPy's float64 type. - """ - - __slots__ = () - _name = "numpy.float64" - _primitive_type = PrimitiveFloatingPointType() - _precision = 8 - - -class NumpyFloat128Type(NumpyNumericType): - """ - Class representing NumPy's float128 type. - - Class representing NumPy's float128 type. - """ - - __slots__ = () - _name = "numpy.float128" - _primitive_type = PrimitiveFloatingPointType() - _precision = 16 - - -# ============================================================================== - - -class NumpyComplex64Type(NumpyNumericType): - """ - Class representing NumPy's complex64 type. - - Class representing NumPy's complex64 type. - """ - - __slots__ = () - _name = "numpy.complex64" - _primitive_type = PrimitiveComplexType() - _precision = 4 - - @property - def element_type(self): - """ - The type of an element of the complex. - - The type of an element of the complex. In other words, the type - of the floats which comprise the complex type. - """ - return NumpyFloat32Type() - - -class NumpyComplex128Type(NumpyNumericType): - """ - Class representing NumPy's complex128 type. - - Class representing NumPy's complex128 type. - """ - - __slots__ = () - _name = "numpy.complex128" - _primitive_type = PrimitiveComplexType() - _precision = 8 - - @property - def element_type(self): - """ - The type of an element of the complex. - - The type of an element of the complex. In other words, the type - of the floats which comprise the complex type. - """ - return NumpyFloat64Type() - - -class NumpyComplex256Type(NumpyNumericType): - """ - Class representing NumPy's complex256 type. - - Class representing NumPy's complex256 type. - """ - - __slots__ = () - _name = "numpy.complex256" - _primitive_type = PrimitiveComplexType() - _precision = 16 - - @property - def element_type(self): - """ - The type of an element of the complex. - - The type of an element of the complex. In other words, the type - of the floats which comprise the complex type. - """ - return NumpyFloat128Type() - - -# ============================================================================== - - -class NumpyNDArrayType(Type): - """ - Class representing the NumPy ND array type. - - Class representing the NumPy ND array type. - """ - - __slots__ = ( - "_allows_strides", - "_container_rank", - "_element_type", - "_order", - "_raw", - ) - _name = "numpy.ndarray" - - @classmethod - @lru_cache - def get_new(cls, dtype, rank, order, allows_strides=True, *, raw=False): - """ - Get the parametrised NumPy ND array type. - - Get the parametrised NumPy ND array type. - - Parameters - ---------- - dtype : NumpyNumericType | GenericType - The internal datatype of the object (GenericType is allowed for external - libraries, e.g. MPI). - rank : int - The rank of the new NumPy array. - order : str - The order of the memory layout for the new NumPy array. - allows_strides : bool - Whether non-contiguous strided views are valid for this array contract. - raw : bool - Whether the array is represented directly as a C array/pointer instead - of the generated ndarray wrapper structure. - """ - assert isinstance(rank, int) - assert order in (None, "C", "F") - assert rank < 2 or order is not None - assert isinstance(allows_strides, bool) - assert isinstance(raw, bool) - if raw: - assert isinstance(dtype, FixedSizeType) - else: - assert isinstance(dtype, NumpyNumericType | GenericType | CharType) - - if rank == 0: - return dtype - - def __init__(self): - self._element_type = dtype - self._container_rank = rank - self._order = order - self._allows_strides = allows_strides - self._raw = raw - super().__init__() - - representation = "Raw" if raw else "Numpy" - stride_suffix = "strided" if allows_strides else "contiguous" - name = f"{representation}{rank}DArrayType_{order}_{stride_suffix}_{type(dtype).__name__}" - return type(name, (NumpyNDArrayType,), {"__init__": __init__})() - - @property - def datatype(self): - """The scalar datatype stored in this ndarray.""" - return self.element_type.datatype - - @property - def primitive_type(self): - """The datatype category of elements in this ndarray.""" - return self.element_type.primitive_type - - @property - def precision(self): - """The precision of elements in this ndarray.""" - return self.element_type.precision - - @property - def element_type(self): - """The scalar type of elements in this ndarray.""" - return self._element_type - - @property - def container_rank(self): - """Number of indices required to select an ndarray element.""" - return self._container_rank - - def __str__(self): - name = "raw_array" if self.raw else self._name - return f"{name}[{self._element_type}]" - - def shape_is_compatible(self, shape): - """Check if the provided shape is compatible with this ndarray.""" - return isinstance(shape, tuple) and len(shape) == self.container_rank - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __add__(self, other): - test_type = numpy.zeros(1, dtype=x2py_type_to_original_type[self.element_type]) - if isinstance(other, FixedSizeNumericType): - comparison_type = x2py_type_to_original_type[other]() - elif isinstance(other, NumpyNDArrayType): - comparison_type = numpy.zeros(1, dtype=x2py_type_to_original_type[other.element_type]) - else: - return NotImplemented - result_type = original_type_to_x2py_type[numpy.result_type(test_type, comparison_type).type] - rank = max(other.rank, self.rank) - if rank < 2: - order = None - else: - other_f_contiguous = other.order in (None, "F") - self_f_contiguous = self.order in (None, "F") - order = "F" if other_f_contiguous and self_f_contiguous else "C" - allows_strides = getattr(self, "allows_strides", True) or getattr(other, "allows_strides", True) - return NumpyNDArrayType.get_new(result_type, rank, order, allows_strides) - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __radd__(self, other): - return self.__add__(other) - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __and__(self, other): - elem_type = self.element_type - if isinstance(other, FixedSizeNumericType): - return self.switch_basic_type(elem_type & other) - if isinstance(other, NumpyNDArrayType): - return self.switch_basic_type(elem_type & other.element_type) - return NotImplemented - - @lru_cache # noqa: B019 - datatype instances are interned and process-lived. - def __rand__(self, other): - return self.__and__(other) - - def switch_basic_type(self, new_type): - """ - Change the basic type to the new type. - - Change the basic type to the new type. A new NumpyNDArrayType will be - returned whose underlying elements are of the NumPy type which is - equivalent to the new type. - - Parameters - ---------- - new_type : FixedSizeNumericType - The new basic type. - - Returns - ------- - Type - The new type. - """ - assert isinstance(new_type, FixedSizeNumericType) - new_type = numpy_precision_map[(new_type.primitive_type, new_type.precision)] - cls = type(self) - return cls.get_new( - self.element_type.switch_basic_type(new_type), - self._container_rank, - self._order, - self._allows_strides, - raw=self.raw, - ) - - def switch_rank(self, new_rank, new_order=None): - """ - Get a type which is identical to this type in all aspects except the rank and/or order. - - Get a type which is identical to this type in all aspects except the rank and/or order. - The order must be provided if the rank is increased from 1. Otherwise it defaults to the - same order as the current type. - - Parameters - ---------- - new_rank : int - The rank of the new type. - - new_order : str, optional - The order of the new type. This should be provided if the rank is increased from 1. - - Returns - ------- - Type - The new type. - """ - if new_rank == 0: - return self.element_type - new_order = (new_order or self._order) if new_rank > 1 else None - return NumpyNDArrayType.get_new( - self.element_type, - new_rank, - new_order, - self._allows_strides, - raw=self.raw, - ) - - @property - def rank(self): - """ - Number of dimensions of the object. - - Number of dimensions of the object. If the object is a scalar then - this is equal to 0. - """ - return self._container_rank - - @property - def order(self): - """ - The data layout ordering in memory. - - Indicates whether the data is stored in row-major ('C') or column-major - ('F') format. This is only relevant if rank > 1. When it is not relevant - this function returns None. - """ - return self._order - - @property - def allows_strides(self): - """Whether non-contiguous strided NumPy views are accepted.""" - return self._allows_strides - - @property - def raw(self): - """Whether this array uses a direct C array/pointer representation.""" - return self._raw - - def __repr__(self): - dims = ",".join(":" * self._container_rank) - order_str = f"(order={self._order})" if self._order else "" - stride_str = "" if self._allows_strides else "(contiguous)" - return f"{self.element_type}[{dims}]{order_str}{stride_str}" - - def __hash__(self): - return hash((self.element_type, self.rank, self.order, self.allows_strides)) - - def __eq__(self, other): - return ( - isinstance(other, NumpyNDArrayType) - and self.element_type == other.element_type - and self.rank == other.rank - and self.order == other.order - and self.allows_strides == other.allows_strides - ) - - -# ============================================================================== - -numpy_precision_map = { - (PrimitiveBooleanType(), -1): NumpyBoolType(), - (PrimitiveIntegerType(), 1): NumpyInt8Type(), - (PrimitiveIntegerType(), 2): NumpyInt16Type(), - (PrimitiveIntegerType(), 4): NumpyInt32Type(), - (PrimitiveIntegerType(), 8): NumpyInt64Type(), - (PrimitiveFloatingPointType(), 4): NumpyFloat32Type(), - (PrimitiveFloatingPointType(), 8): NumpyFloat64Type(), - (PrimitiveFloatingPointType(), 16): NumpyFloat128Type(), - (PrimitiveComplexType(), 4): NumpyComplex64Type(), - (PrimitiveComplexType(), 8): NumpyComplex128Type(), - (PrimitiveComplexType(), 16): NumpyComplex256Type(), -} - -numpy_type_to_original_type = { - NumpyBoolType(): numpy.bool_, - NumpyInt8Type(): numpy.int8, - NumpyInt16Type(): numpy.int16, - NumpyInt32Type(): numpy.int32, - NumpyInt64Type(): numpy.int64, - NumpyFloat32Type(): numpy.float32, - NumpyFloat64Type(): numpy.float64, - NumpyComplex64Type(): numpy.complex64, - NumpyComplex128Type(): numpy.complex128, -} - -x2py_type_to_original_type = { - NumpyBoolType(): numpy.bool_, - NumpyInt64Type(): numpy.int64, - NumpyFloat64Type(): numpy.float64, - NumpyComplex128Type(): numpy.complex128, -} - -original_type_to_x2py_type = { - bool: NumpyBoolType(), - int: NumpyInt64Type(), - float: NumpyFloat64Type(), - complex: NumpyComplex128Type(), -} - -# Large types don't exist on all systems -if hasattr(numpy, "float128"): - numpy_type_to_original_type.update( - { - NumpyFloat128Type(): numpy.float128, - NumpyComplex256Type(): numpy.complex256, - } - ) - -x2py_type_to_original_type.update(numpy_type_to_original_type) -original_type_to_x2py_type.update({v: k for k, v in numpy_type_to_original_type.items()}) - - -# ====================================================================== -class Literal: - """A value expressed directly in generated code.""" - - __slots__ = ("_class_type", "_shape", "_value") - _attribute_nodes = () - - def __init__(self, value, datatype): - if not isinstance(datatype, Type): - raise TypeError("datatype must be a codegen Type") - - if isinstance(datatype, StringType): - if not isinstance(value, str): - raise TypeError("string literals require a str value") - self._value = value - self._shape = (None,) - elif isinstance(datatype, VoidType): - if value is not None: - raise TypeError("void literals require a None value") - self._value = None - self._shape = None - elif isinstance(datatype, FixedSizeNumericType): - primitive_type = datatype.primitive_type - if isinstance(primitive_type, PrimitiveBooleanType): - if not isinstance(value, bool | numpy.bool_): - raise TypeError("boolean literals require a bool value") - self._value = bool(value) - elif isinstance(primitive_type, PrimitiveIntegerType): - if not isinstance(value, int | numpy.integer): - raise TypeError("integer literals require an integer value") - self._value = int(value) - elif isinstance(primitive_type, PrimitiveFloatingPointType): - if not isinstance(value, int | float | numpy.integer | numpy.floating): - raise TypeError("floating-point literals require a real value") - self._value = float(value) - elif isinstance(primitive_type, PrimitiveComplexType): - if not isinstance(value, int | float | complex | numpy.number): - raise TypeError("complex literals require a numeric value") - self._value = complex(value) - else: - raise TypeError(f"Unsupported literal datatype {datatype}") - self._shape = None - else: - raise TypeError(f"Unsupported literal datatype {datatype}") - - self._class_type = datatype - init_model_object(self) - - @property - def python_value(self): - """Return the Python value represented by this literal.""" - return self._value - - def __repr__(self): - return f"Literal({self.python_value!r}, {self.class_type!r})" - - def __str__(self): - return str(self.python_value) - - def __eq__(self, other): - if is_model_object(other): - return ( - isinstance(other, Literal) - and self.class_type == other.class_type - and self.python_value == other.python_value - ) - return self.python_value == other - - def __hash__(self): - return hash((self.python_value, self.class_type)) - - def __index__(self): - if not isinstance(self.class_type.primitive_type, PrimitiveIntegerType): - raise TypeError("only integer literals can be used as indices") - return self.python_value - - def __add__(self, o): - if isinstance(self.class_type, StringType) and isinstance(o, Literal) and isinstance(o.class_type, StringType): - return Literal(self.python_value + o.python_value, StringType()) - return NotImplemented - - def __bool__(self): - return self.python_value is not None - - -NIL = Literal(None, VoidType()) - - -# ------------------------------------------------------------------------------ - - -def convert_to_literal(value, dtype=None): - """ - Convert a Python value to a x2py Literal. - - Convert a Python value to a x2py Literal. - - Parameters - ---------- - value : int/float/complex/bool/str or NumPy scalar - The Python value. - dtype : DataType - The datatype of the Python value. - Default : Matches type of 'value'. - - Returns - ------- - Literal - The Python value 'value' expressed as a literal - with the specified dtype. - """ - from .core import UnarySub # Imported here to avoid circular import - - if isinstance(value, Literal): - if dtype is None or dtype == value.dtype: - return value - value = value.python_value - - # Calculate the default datatype - if dtype is None: - if isinstance(value, numpy.generic): - numpy_type = numpy.asarray(value).dtype.type - try: - dtype = original_type_to_x2py_type[numpy_type] - except KeyError as e: - raise TypeError(f"Unknown type of object {value}") from e - elif isinstance(value, bool): - dtype = NumpyBoolType() - elif isinstance(value, int): - dtype = NumpyInt64Type() - elif isinstance(value, float): - dtype = NumpyFloat64Type() - elif isinstance(value, complex): - dtype = NumpyComplex128Type() - elif isinstance(value, str): - dtype = StringType() - else: - raise TypeError(f"Unknown type of object {value}") - - # Resolve any datatypes which don't inherit from FixedSizeType - if isinstance(dtype, StringType): - return Literal(value, dtype) - - assert isinstance(dtype, FixedSizeNumericType) - - primitive_type = dtype.primitive_type - if isinstance(primitive_type, PrimitiveIntegerType): - literal_val = Literal(value, dtype) if value >= 0 else UnarySub(Literal(-value, dtype)) - elif isinstance(primitive_type, PrimitiveFloatingPointType | PrimitiveComplexType | PrimitiveBooleanType): - literal_val = Literal(value, dtype) - else: - raise TypeError(f"Unknown type {dtype}") - - return literal_val - - -def _cast_result_type(arg, target_type): - """Return the scalar or array datatype produced by a cast.""" - if arg.rank == 0: - return target_type - return NumpyNDArrayType.get_new( - target_type, - arg.rank, - arg.order, - getattr(arg.class_type, "allows_strides", True), - ) - - -class _DataTypeFunction: - """Small call-node base for datatype casting helpers.""" - - __slots__ = ("_args",) - _attribute_nodes = ("_args",) - name = None - - def __init__(self, *args): - self._args = tuple(args) - init_model_object(self) - - @property - def args(self): - return self._args - - @property - def is_elemental(self): - return False - - -class Cast(_DataTypeFunction): - """A conversion of one model expression to a target datatype.""" - - __slots__ = ("_class_type", "_shape") - - def __init__(self, arg, datatype): - if not isinstance(datatype, Type): - raise TypeError("datatype must be a codegen Type") - if isinstance(datatype, StringType) and not isinstance(arg.class_type, StringType | CharType): - raise NotImplementedError("Support for casting non-character types to strings is not available") - self._shape = (None,) if isinstance(datatype, StringType) else arg.shape - self._class_type = _cast_result_type(arg, datatype) - super().__init__(arg) - - @property - def arg(self): - """Return the expression being converted.""" - return self._args[0] - - @property - def is_elemental(self): - return True - - def __str__(self): - return f"Cast({self.arg}, {self.dtype})" - - -def cast_to(arg, target_type): - """Return ``arg`` cast to ``target_type`` using the codegen cast node.""" - if arg.class_type == target_type: - return arg - if isinstance(target_type, NumpyNDArrayType): - target_type = target_type.element_type - - if isinstance(target_type, NumpyBoolType) and getattr(arg, "is_optional", False): - from .core import And, IsNot - - return And(IsNot(arg, NIL), Cast(arg, target_type)) - return Cast(arg, target_type) - - -for _model_cls in (Literal, _DataTypeFunction): - register_model_class(_model_cls) - -del _model_cls diff --git a/x2py/codegen/printers/__init__.py b/x2py/codegen/printers/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/x2py/codegen/printers/ccode.py b/x2py/codegen/printers/ccode.py deleted file mode 100644 index ab151d147..000000000 --- a/x2py/codegen/printers/ccode.py +++ /dev/null @@ -1,1197 +0,0 @@ -""" -Module containing the `CCodePrinter` class which converts X2py's AST to -strings of C code. -""" - -from itertools import chain -from typing import ClassVar - - -from x2py.semantics.ownership import NativeBarrierAction, ObjectKind -from x2py.semantics.native_array_handles import NativeArrayHandlePolicy - -from ..bind_c import BindCPointer -from ..bindings.c_concepts import ( - CFIDescriptorDimField, - CFIDescriptorField, - CFIDescriptorStorageType, - CFIDescriptorType, - CFIDimensionType, - CStrStr, - ObjectAddress, - PointerCast, -) -from ..models.core import ( - AsName, - Assign, - Declare, - FunctionAddress, - FunctionCall, - get_direct_assignment, - get_direct_module, - get_enclosing_function, - Import, - Module, - PythonTuple, - SeparatorComment, -) -from ..models.datatypes import ( - CharType, - CustomDataType, - FinalType, - FixedSizeNumericType, - PrimitiveBooleanType, - PrimitiveComplexType, - PrimitiveFloatingPointType, - PrimitiveIntegerType, - NumpyBoolType, - StringType, - VoidType, -) -from ..models.datatypes import ( - Literal, - NIL, -) -from ..models.datatypes import ( - NumpyNDArrayType, -) -from ..models.core import Operator -from ..models.core import IndexedElement, Variable -from .codeprinter import CodePrinter - -# TODO: add examples - -__all__ = ["CCodePrinter"] - -c_library_headers = ( - "ISO_Fortran_binding", - "complex", - "ctype", - "float", - "inttypes", - "math", - "stdarg", - "stdbool", - "stddef", - "stdint", - "stdio", - "stdlib", - "string", -) - -import_dict = {"omp_lib": "omp"} - -c_imports = { - n: Import(n, Module(n, (), ())) - for n in [ - "ISO_Fortran_binding", - "assert", - "complex", - "float", - "inttypes", - "math", - "pyc_math_c", - "stdbool", - "stdint", - "stdio", - "stdlib", - "string", - "stc/cstr", - "CSpan_extensions", - ] -} - - -class CCodePrinter(CodePrinter): - """ - A printer for printing code in C. - - A printer to convert X2py's AST to strings of c code. - As for all printers the navigation of this file is done via _visit_X - functions. - - Parameters - ---------- - filename : str - The name of the file being converted. - verbose : int - The level of verbosity. - prefix_module : str - A prefix to be added to the name of the module. - """ - - printmethod = "_ccode" - language = "C" - - _default_settings: ClassVar = { - "tabwidth": 4, - } - - dtype_registry: ClassVar = { - VoidType(): "void", - CharType(): "char", - (PrimitiveIntegerType(), None): "int", - (PrimitiveComplexType(), 8): "double complex", - (PrimitiveComplexType(), 4): "float complex", - (PrimitiveFloatingPointType(), 8): "double", - (PrimitiveFloatingPointType(), 4): "float", - (PrimitiveIntegerType(), 4): "int32_t", - (PrimitiveIntegerType(), 8): "int64_t", - (PrimitiveIntegerType(), 2): "int16_t", - (PrimitiveIntegerType(), 1): "int8_t", - (PrimitiveBooleanType(), -1): "bool", - CFIDescriptorType(): "CFI_cdesc_t", - CFIDimensionType(): "CFI_dim_t", - } - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, filename, *, verbose, prefix_module=None): - """Initialize the state used for one generation run.""" - super().__init__(verbose) - self.prefix_module = prefix_module - self._additional_imports = {"stdlib": c_imports["stdlib"]} - self._additional_code = "" - self._additional_args = [] - self._temporary_args = [] - self._in_header = False - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_Cast(self, expr): - """Render the ``Cast`` model node.""" - value = self._visit(expr.arg) - dtype = expr.dtype - - if isinstance(dtype, StringType): - if isinstance(expr.arg.class_type, StringType): - return f"cstr_clone({value})" - assert isinstance(expr.arg.class_type, CharType) and getattr(expr.arg, "is_alias", True) - return f"cstr_from({value})" - if isinstance(dtype.primitive_type, PrimitiveBooleanType): - return f"({value} != 0)" - if isinstance(dtype.primitive_type, PrimitiveIntegerType): - self.add_import(c_imports["stdint"]) - return f"({self._c_type(dtype)})({value})" - - def _visit_Literal(self, expr): - """Render the ``Literal`` model node.""" - value = expr.python_value - dtype = expr.dtype - - if expr is NIL: - return "NULL" - if isinstance(dtype, StringType): - escaped = ( - value.replace("\\", "\\\\") - .replace("\a", "\\a") - .replace("\b", "\\b") - .replace("\f", "\\f") - .replace("\n", "\\n") - .replace("\r", "\\r") - .replace("\t", "\\t") - .replace("\v", "\\v") - .replace('"', '\\"') - .replace("'", "\\'") - ) - return f'cstr_lit("{escaped}")' - - primitive_type = dtype.primitive_type - if isinstance(primitive_type, PrimitiveBooleanType): - return "1" if value else "0" - if isinstance(primitive_type, PrimitiveIntegerType) and dtype.precision == 8: - self.add_import(c_imports["stdint"]) - sign = "-" if value < 0 else "" - return f"{sign}INT64_C({abs(value)})" - if isinstance(primitive_type, PrimitiveFloatingPointType): - suffix = "f" if dtype.precision == 4 else "" - return f"{value!r}{suffix}" - if isinstance(primitive_type, PrimitiveComplexType): - self.add_import(c_imports["complex"]) - real = self._visit(Literal(value.real, dtype.element_type)) - imag = self._visit(Literal(abs(value.imag), dtype.element_type)) - if value.real == 0: - sign = "-" if value.imag < 0 else "" - return f"({sign}{imag} * _Complex_I)" - sign = "-" if value.imag < 0 else "+" - return f"({real} {sign} {imag} * _Complex_I)" - return repr(value) - - def _visit_If(self, expr): - """Render the ``If`` model node.""" - lines = [] - condition_setup = [] - for i, (c, b) in enumerate(expr.blocks): - body = self._visit(b) - if i == len(expr.blocks) - 1 and isinstance(c, Literal) and c.python_value is True: - if i == 0: - lines.append(body) - break - lines.append("else\n") - else: - # Print condition - condition = self._visit(c) - # Retrieve any additional code which cannot be executed in the line containing the condition - condition_setup.append(self._additional_code) - self._additional_code = "" - # Add the condition to the lines of code - line = f"if ({condition})\n" - if i == 0: - lines.append(line) - else: - lines.append("else " + line) - lines.append("{\n") - lines.append(body + "}\n") - return "".join(chain(condition_setup, lines)) - - def _visit_IfTernaryOperator(self, expr): - """Render the ``IfTernaryOperator`` model node.""" - cond = self._visit(expr.cond) - value_true = self._visit(expr.value_true) - value_false = self._visit(expr.value_false) - return f"({cond} ? {value_true} : {value_false})" - - def _visit_And(self, expr): - """Render the ``And`` model node.""" - args = [(f"({self._visit(a)})" if isinstance(a, Operator) else self._visit(a)) for a in expr.args] - return " && ".join(args) - - def _visit_Or(self, expr): - """Render the ``Or`` model node.""" - args = [(f"({self._visit(a)})" if isinstance(a, Operator) else self._visit(a)) for a in expr.args] - return " || ".join(args) - - def _visit_Eq(self, expr): - """Render the ``Eq`` model node.""" - lhs, rhs = expr.args - if isinstance(lhs.class_type, StringType) and isinstance(rhs.class_type, StringType): - lhs_code = self._visit(CStrStr(lhs)) - rhs_code = self._visit(CStrStr(rhs)) - return f"!strcmp({lhs_code}, {rhs_code})" - if isinstance(lhs.class_type, FixedSizeNumericType): - lhs_code = self._visit(lhs) - rhs_code = self._visit(rhs) - return f"{lhs_code} == {rhs_code}" - raise NotImplementedError(f"C equality printing is not implemented for {expr}") - - def _visit_Ne(self, expr): - """Render the ``Ne`` model node.""" - lhs, rhs = expr.args - if isinstance(lhs.class_type, StringType) and isinstance(rhs.class_type, StringType): - lhs_code = self._visit(CStrStr(lhs)) - rhs_code = self._visit(CStrStr(rhs)) - return f"strcmp({lhs_code}, {rhs_code})" - if isinstance(lhs.class_type, FixedSizeNumericType): - lhs_code = self._visit(lhs) - rhs_code = self._visit(rhs) - return f"{lhs_code} != {rhs_code}" - raise NotImplementedError(f"C inequality printing is not implemented for {expr}") - - def _visit_Lt(self, expr): - """Render the ``Lt`` model node.""" - lhs = self._visit(expr.args[0]) - rhs = self._visit(expr.args[1]) - return f"{lhs} < {rhs}" - - def _visit_Le(self, expr): - """Render the ``Le`` model node.""" - lhs = self._visit(expr.args[0]) - rhs = self._visit(expr.args[1]) - return f"{lhs} <= {rhs}" - - def _visit_Ge(self, expr): - """Render the ``Ge`` model node.""" - lhs = self._visit(expr.args[0]) - rhs = self._visit(expr.args[1]) - return f"{lhs} >= {rhs}" - - def _visit_Not(self, expr): - """Render the ``Not`` model node.""" - arg = expr.args[0] - a = self._visit(arg) - if isinstance(arg, Operator): - a = f"({a})" - return f"!{a}" - - def _visit_Import(self, expr): - """Render the ``Import`` model node.""" - if expr.ignore: - return "" - source = expr.source.name if isinstance(expr.source, AsName) else expr.source - - source = self._visit(source) - - # Get with a default value is not used here as it is - # slower and on most occasions the import will not be in the - # dictionary - if source in import_dict: # pylint: disable=consider-using-get - source = import_dict[source] - - if source is None: - return "" - if expr.source in c_library_headers: - return f"#include <{source}.h>\n" - return f'#include "{source}.h"\n' - - def _visit_Declare(self, expr): - """Render the ``Declare`` model node.""" - var = expr.variable - declaration_type = self._get_declare_type(var) - - init = f" = {self._visit(expr.value)}" if expr.value is not None else "" - - if isinstance(var.class_type, NumpyNDArrayType) and var.class_type.raw: - assert init == "" - preface = "" - if isinstance(var.alloc_shape[0], int | Literal): - init = f"[{var.alloc_shape[0]}]" - else: - declaration_type += "*" - init = "" - elif var.is_stack_array: - preface, init = self._init_stack_array(var) - else: - preface = "" - if isinstance(var.class_type, NumpyNDArrayType) and not expr.external and not var.is_alias: - init = " = {0}" - - external = "extern " if expr.external else "" - static = "static " if expr.static else "" - const = "const " if isinstance(var.class_type, FinalType) and self._is_c_pointer(var) else "" - - return f"{preface}{static}{external}{const}{declaration_type} {var.name}{init};\n" - - def _visit_DottedVariable(self, expr): - """convert dotted Variable to their C equivalent""" - - name_code = self._visit(expr.name) - if self._is_c_pointer(expr.lhs): - code = f"{self._visit(ObjectAddress(expr.lhs))}->{name_code}" - else: - lhs_code = self._visit(expr.lhs) - code = f"{lhs_code}.{name_code}" - if self._is_c_pointer(expr): - return f"(*{code})" - return code - - def _visit_Deallocate(self, expr): - """Render the ``Deallocate`` model node.""" - var = expr.variable - variable_address = self._visit(ObjectAddress(var)) - return f"free({variable_address});\n" - - def _visit_FunctionAddress(self, expr): - """Render the ``FunctionAddress`` model node.""" - return expr.name - - def _visit_FunctionOverloadSet(self, expr): - """Render the ``FunctionOverloadSet`` model node.""" - return "".join(self._visit(f) for f in expr.functions) - - def _visit_FunctionDef(self, expr): - """Render the ``FunctionDef`` model node.""" - if not expr.is_semantic: - return "" - - self._validate_c_function_results(expr) - - sep = self._visit(SeparatorComment(40)) - - inner_funcs = "".join(self._visit(f).removeprefix(sep).removesuffix(sep) + "\n" for f in expr.functions) - - self.set_scope(expr.scope) - - # Collect results filtering out NIL - results = [r for r in self.scope.collect_all_tuple_elements(expr.results.var) if isinstance(r, Variable)] - returning_tuple = False - self._push_additional_function_args(expr, results, returning_tuple) - - body = self._visit(expr.body) - decs = self._function_declarations(expr, results, returning_tuple) - - self._additional_args.pop() - for i in expr.imports: - self.add_import(i) - docstring = self._visit(expr.docstring) if expr.docstring else "" - - parts = [ - sep, - inner_funcs, - docstring, - f"{self._function_signature(expr)}\n{{\n", - decs, - body, - "}\n", - sep, - ] - - self.exit_scope() - - return "".join(p for p in parts if p) - - @staticmethod - def _validate_c_function_results(function) -> None: - """Reject stack arrays that cannot be returned from C.""" - for result in function.scope.collect_all_tuple_elements(function.results.var): - if result.rank and result.memory_handling == "stack": - raise ValueError("Can't return a stack array from C code") - - def _push_additional_function_args(self, function, results, returning_tuple) -> None: - """Track output and global variables passed as hidden arguments.""" - self._additional_args.append(results if len(results) > 1 or returning_tuple else []) - self._additional_args[-1].extend( - variable for variable in function.global_vars if get_direct_module(variable) is None - ) - - def _function_declarations(self, function, results, returning_tuple): - """Render local and result declarations for a C function.""" - declarations = [ - Declare( - variable, - value=(NIL if variable.is_alias and isinstance(variable.class_type, VoidType | BindCPointer) else None), - ) - for variable in function.local_vars - ] - if len(results) == 1 and not returning_tuple: - result = results[0] - if not result.is_temp or result.rank: - declarations.append(Declare(result)) - return "".join(self._visit(declaration) for declaration in declarations) - - def _visit_FunctionCall(self, expr): - """Render the ``FunctionCall`` model node.""" - func = expr.funcdef - if func.name in {"memcpy", "memset", "strlen"}: - self.add_import(c_imports["string"]) - parent_assign = get_direct_assignment(expr) - returns_via_output_args = self._returns_via_output_args(func) - # Ensure the correct syntax is used for pointers - args = [ - self._prepare_call_argument(argument.value, formal.var) - for argument, formal in zip(expr.args, func.arguments, strict=False) - ] - args = self._insert_after_bound_argument(func, args, self._temporary_args) - - for v in func.global_vars: - if get_direct_module(v) is None: - args.append(ObjectAddress(v)) - - if parent_assign is not None and returns_via_output_args: - output_args = self._output_call_arguments(parent_assign) - args = self._insert_after_bound_argument(func, args, output_args) - - self._temporary_args = [] - args = ", ".join(self._visit(ai) for a in args for ai in self.scope.collect_all_tuple_elements(a)) - - call_code = f"{func.name}({args})" - if parent_assign is not None and returns_via_output_args: - return f"{call_code};\n" - if func.results.var is not NIL: - return call_code - return f"{call_code};\n" - - def _prepare_call_argument(self, argument, formal): - """Adapt one call argument to the formal C pointer contract.""" - if not self._is_c_pointer(formal): - return argument - if isinstance(argument, Variable): - return ObjectAddress(argument) - if self._is_c_pointer(argument): - return argument - temporary = self.scope.get_temporary_variable(formal.dtype) - self._additional_code += self._visit(Assign(temporary, argument)) - return ObjectAddress(temporary) - - @staticmethod - def _insert_after_bound_argument(function, arguments, inserted): - """Insert hidden arguments after a bound receiver when present.""" - if function.arguments and function.arguments[0].bound_argument: - return arguments[:1] + inserted + arguments[1:] - return inserted + arguments - - def _output_call_arguments(self, parent_assign): - """Build address arguments for results returned through outputs.""" - if isinstance(parent_assign.lhs, PythonTuple): - result_args = parent_assign.lhs.args - else: - result_args = self.scope.collect_all_tuple_elements(parent_assign.lhs) - output_args = [] - for argument in result_args: - output_arg = ObjectAddress(argument) - if not isinstance(argument, ObjectAddress) and self._is_c_pointer(argument): - output_arg = ObjectAddress(output_arg) - output_args.append(output_arg) - return output_args - - def _visit_Return(self, expr): - """Render the ``Return`` model node.""" - func = get_enclosing_function(expr) - assert func is not None - code = "" - - return_obj = expr.expr - if return_obj is None: - args = [] - else: - args = [(ObjectAddress(return_obj) if self._is_c_pointer(return_obj) else return_obj)] - - if len(args) == 0: - return code + "return;\n" - - returned_value = self.scope.collect_tuple_element(args[0]) - - return code + f"return {self._visit(returned_value)};\n" - - def _visit_Pass(self, expr): - """Render the ``Pass`` model node.""" - return "// pass\n" - - def _visit_Add(self, expr): - """Render the ``Add`` model node.""" - return " + ".join(self._visit(a) for a in expr.args) - - def _visit_Minus(self, expr): - """Render the ``Minus`` model node.""" - args = [self._visit(a) for a in expr.args] - if len(args) == 1: - return f"-{args[0]}" - return " - ".join(args) - - def _visit_Mul(self, expr): - """Render the ``Mul`` model node.""" - return " * ".join(self._visit(a) for a in expr.args) - - def _visit_UnarySub(self, expr): - """Render the ``UnarySub`` model node.""" - return f"-{self._visit(expr.args[0])}" - - def _visit_AugAssign(self, expr): - """Render the ``AugAssign`` model node.""" - op = expr.op - lhs = expr.lhs - rhs = expr.rhs - - if op == "//" or (op == "%" and isinstance(lhs.dtype.primitive_type, PrimitiveFloatingPointType)): - _expr = expr.to_basic_assign() - return self._visit(_expr) - - lhs_code = self._visit(lhs) - rhs_code = self._visit(rhs) - return f"{lhs_code} {op}= {rhs_code};\n" - - def _visit_Assign(self, expr): - """Render the ``Assign`` model node.""" - lhs = expr.lhs - rhs = expr.rhs - - if isinstance(rhs, FunctionCall) and self._returns_via_output_args(rhs.funcdef): - return self._visit(rhs) - - lhs_code = self._visit(lhs) - rhs_code = self._visit(rhs) - return f"{lhs_code} = {rhs_code};\n" - - def _visit_AliasAssign(self, expr): - """Render the ``AliasAssign`` model node.""" - lhs_var = expr.lhs - rhs_var = expr.rhs - - lhs_address = ObjectAddress(lhs_var) - rhs_address = ObjectAddress(rhs_var) - - # The condition below handles the case of reassigning a pointer to an array view. - if isinstance(lhs_var, Variable) and lhs_var.is_ndarray and not lhs_var.is_optional: - lhs = self._visit(lhs_var) - - if isinstance(rhs_var, Variable) and rhs_var.is_ndarray: - lhs_ptr = self._visit(lhs_address) - rhs = self._visit(rhs_address) - rhs_type = self._c_type(rhs_var.class_type) - slicing = ", ".join(["{c_ALL}"] * lhs_var.rank) - code = f"{lhs} = cspan_slice({rhs}, {rhs_type}, {slicing});\n" - if lhs_var.order != rhs_var.order: - code += f"cspan_transpose({lhs_ptr});\n" - return code - rhs = self._visit(rhs_var) - return f"{lhs} = {rhs};\n" - lhs = self._visit(lhs_address) - rhs = self._visit(rhs_address) - - return f"{lhs} = {rhs};\n" - - def _visit_CodeBlock(self, expr): - """Render the ``CodeBlock`` model node.""" - body_exprs = expr.body - body_stmts = [] - for b in body_exprs: - code = self._visit(b) - code = self._additional_code + code - self._additional_code = "" - body_stmts.append(code) - return "".join(self._visit(b) for b in body_stmts) - - def _visit_IsNot(self, expr): - """Render the ``IsNot`` model node.""" - return self._handle_is_operator("!=", expr) - - def _visit_Is(self, expr): - """Render the ``Is`` model node.""" - return self._handle_is_operator("==", expr) - - def _visit_Variable(self, expr): - """Render the ``Variable`` model node.""" - if isinstance(expr.class_type, BindCPointer): - return expr.name - if self._is_c_pointer(expr): - return f"(*{expr.name})" - return expr.name - - def _visit_FunctionDefArgument(self, expr): - """Render the ``FunctionDefArgument`` model node.""" - return self._visit(expr.name) - - def _visit_FunctionCallArgument(self, expr): - """Render the ``FunctionCallArgument`` model node.""" - return self._visit(expr.value) - - def _visit_ObjectAddress(self, expr): - """Render the ``ObjectAddress`` model node.""" - obj_code = self._visit(expr.obj) - if isinstance(expr.obj, ObjectAddress): - return f"&{obj_code}" - if obj_code.startswith("(*") and obj_code.endswith(")"): - return f"{obj_code[2:-1]}" - if not self._is_c_pointer(expr.obj): - return f"&{obj_code}" - return obj_code - - def _visit_PointerCast(self, expr): - """Render the ``PointerCast`` model node.""" - declare_type = self._get_declare_type(expr.cast_type) - if not self._is_c_pointer(expr.cast_type): - declare_type += "*" - obj = expr.obj - if not isinstance(obj, ObjectAddress): - obj = ObjectAddress(expr.obj) - var_code = self._visit(obj) - return f"(*({declare_type})({var_code}))" - - def _visit_Comment(self, expr): - """Render the ``Comment`` model node.""" - comments = self._visit(expr.text) - - return "/*" + comments + "*/\n" - - def _visit_Symbol(self, expr): - """Render the ``Symbol`` model node.""" - return expr - - def _visit_CommentBlock(self, expr): - """Render the ``CommentBlock`` model node.""" - txts = expr.comments - header = expr.header - header_size = len(expr.header) - - ln = max(len(i) for i in txts) - if ln < max(20, header_size + 4): - ln = 20 - top = "/*" + "_" * int((ln - header_size) / 2) + header + "_" * int((ln - header_size) / 2) + "*/\n" - ln = len(top) - 4 - bottom = "/*" + "_" * ln + "*/\n" - - txts = ["/*" + t + " " * (ln - len(t)) + "*/\n" for t in txts] - - body = "".join(i for i in txts) - - return "".join([top, body, bottom]) - - def _visit_EmptyNode(self, expr): - """Render the ``EmptyNode`` model node.""" - return "" - - def _visit_CustomDataType(self, expr): - """Render the ``CustomDataType`` model node.""" - return "struct " + expr.low_level_name - - def _visit_CFIDescriptorField(self, expr): - """Render a field read from a TS 29113 descriptor pointer.""" - descriptor = self._cfi_descriptor_pointer(expr.owner) - return f"{descriptor}->{expr.field}" - - def _visit_CFIDescriptorDimField(self, expr): - """Render a dimension field read from a TS 29113 descriptor pointer.""" - descriptor = self._cfi_descriptor_pointer(expr.owner) - index = self._visit(expr.index) - return f"{descriptor}->dim[{index}].{expr.field}" - - def _visit_CFIDescriptorEstablish(self, expr): - """Establish a standard disassociated pointer descriptor.""" - self.add_import(c_imports["ISO_Fortran_binding"]) - descriptor = self._visit(ObjectAddress(expr.descriptor)) - element_type = expr.element_type - c_type = self._c_type(element_type) - cfi_type = self._cfi_element_type_code(element_type) - base_address = ( - "NULL" - if expr.base_address is None - else self._visit(ObjectAddress(expr.base_address)) - if isinstance(expr.base_address, Variable) and expr.base_address.is_alias - else self._visit(expr.base_address) - ) - element_length = f"sizeof({c_type})" if expr.element_length is None else self._visit(expr.element_length) - extents = ( - "NULL" - if not expr.extents - else f"(CFI_index_t[]){{{', '.join(self._visit(extent) for extent in expr.extents)}}}" - ) - return ( - f"CFI_establish({descriptor}, {base_address}, CFI_attribute_{expr.attribute}, {cfi_type}, " - f"{element_length}, {expr.rank}, {extents})" - ) - - def _visit_CFIDescriptorAllocate(self, expr): - """Allocate payload storage through a standard allocatable descriptor.""" - self.add_import(c_imports["ISO_Fortran_binding"]) - descriptor = self._visit(ObjectAddress(expr.descriptor)) - lower_bounds = self._cfi_index_array(expr.lower_bounds) - upper_bounds = self._cfi_index_array(expr.upper_bounds) - return f"CFI_allocate({descriptor}, {lower_bounds}, {upper_bounds}, {self._visit(expr.element_length)})" - - def _visit_CFIDescriptorDeallocate(self, expr): - """Deallocate payload storage through a standard allocatable descriptor.""" - self.add_import(c_imports["ISO_Fortran_binding"]) - return f"CFI_deallocate({self._visit(ObjectAddress(expr.descriptor))})" - - def _visit_CFIDescriptorStorageSize(self, expr): - """Render the size of rank-specific standard descriptor storage.""" - self.add_import(c_imports["ISO_Fortran_binding"]) - return f"sizeof(CFI_CDESC_T({expr.rank}))" - - def _cfi_index_array(self, values): - """Render one temporary CFI index array.""" - return f"(CFI_index_t[]){{{', '.join(self._visit(value) for value in values)}}}" - - @staticmethod - def _cfi_element_type_code(dtype): - """Return the standard CFI type code for one intrinsic element type.""" - primitive = dtype.primitive_type - if isinstance(primitive, PrimitiveBooleanType): - return "CFI_type_Bool" - if isinstance(primitive, PrimitiveIntegerType): - return f"CFI_type_int{dtype.precision * 8}_t" - if isinstance(primitive, PrimitiveFloatingPointType): - return {4: "CFI_type_float", 8: "CFI_type_double"}[dtype.precision] - if isinstance(primitive, PrimitiveComplexType): - return {4: "CFI_type_float_Complex", 8: "CFI_type_double_Complex"}[dtype.precision] - if isinstance(dtype, CharType): - return "CFI_type_char" - raise TypeError(f"Unsupported CFI descriptor element dtype: {dtype}") - - def _cfi_descriptor_pointer(self, owner): - """Render a model object as a ``CFI_cdesc_t*`` expression.""" - self.add_import(c_imports["ISO_Fortran_binding"]) - pointer = ObjectAddress(owner) if isinstance(owner, Variable) and owner.is_alias else owner - return f"((CFI_cdesc_t*){self._visit(pointer)})" - - # ================== String methods ================== - - def _visit_CStrStr(self, expr): - """Render the ``CStrStr`` model node.""" - arg = expr.args[0] - code = self._visit(ObjectAddress(arg)) - if code.startswith("&cstr_lit("): - return code[10:-1] - return f"cstr_str({code})" - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _format_code(self, lines): - """Format code.""" - return self._indent_code(lines) - - def _is_c_pointer(self, a): - """ - Indicate whether the object is a pointer in C code. - - Some objects are accessed via a C pointer so that they can be modified in - their scope and that modification can be retrieved elsewhere. This - information cannot be found trivially so this function provides that - information while avoiding easily outdated code to be repeated. - - The main reasons for this treatment are: - 1. It is the actual memory address of an object - 2. It is a reference to another object (e.g. an alias, an optional argument, or one of multiple return arguments) - - See codegen_stage.md in the developer docs for more details. - - Parameters - ---------- - a : model object - The object whose storage we are enquiring about. - - Returns - ------- - bool - True if a C pointer, False otherwise. - """ - if a is NIL or isinstance(a, ObjectAddress | PointerCast | CStrStr): - return True - if isinstance(a, CFIDescriptorField | CFIDescriptorDimField) and isinstance(a.class_type, BindCPointer): - return True - if isinstance(a, FunctionCall): - a = a.funcdef.results.var - # STC _at and _at_mut functions return pointers - if isinstance(a, IndexedElement): - return self._indexed_element_is_c_pointer(a) - if not isinstance(a, Variable): - return False - if self._is_scalar_native_value_barrier(a): - return False - additional_argument = self._is_additional_c_argument(a) - if isinstance(a.class_type, NumpyNDArrayType): - return a.is_optional or additional_argument or (a.class_type.raw and a.is_alias) - - if isinstance(a.class_type, CustomDataType) and a.is_argument and not isinstance(a.class_type, FinalType): - return True - - return a.is_alias or a.is_optional or additional_argument - - @staticmethod - def _is_scalar_native_value_barrier(variable): - """Return whether scalar address/storage policy still prints as a C value.""" - if isinstance(variable.class_type, BindCPointer): - return False - decision = getattr(variable, "ownership_decision", None) - return bool( - decision is not None - and not variable.is_optional - and not decision.mutates_native - and decision.kind is ObjectKind.SCALAR - and decision.native_barrier_action - in { - NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS, - NativeBarrierAction.PASS_RAW_ADDRESS, - NativeBarrierAction.PASS_STORAGE_ADDRESS, - } - ) - - @staticmethod - def _indexed_element_is_c_pointer(element): - """Return whether an indexed element is represented as a C pointer.""" - raw_array = isinstance(element.base.class_type, NumpyNDArrayType) and element.base.class_type.raw - return not raw_array and element.rank == 0 - - def _is_additional_c_argument(self, variable): - """Return whether a variable is tracked as a hidden C argument.""" - return any(variable is item for arguments in self._additional_args for item in arguments) - - # ============ Elements ============ # - - @staticmethod - def _x2py_malloc_helper(): - """Handle x2py malloc helper for the current generation context.""" - return ( - "void* x2py_malloc(size_t size)\n" - "{\n" - ' const char* fail_alloc = getenv("X2PY_WRAPPER_FAIL_ALLOC");\n' - " if (fail_alloc != NULL && fail_alloc[0] != '\\0' && fail_alloc[0] != '0') {\n" - " return NULL;\n" - " }\n" - " return malloc(size == 0 ? 1 : size);\n" - "}\n" - ) - - def _c_type(self, dtype): - """ - Find the corresponding C type of the Type. - - For scalar types, this function searches for the corresponding C data type - in the `dtype_registry`. - - Parameters - ---------- - dtype : Type - The data type of the expression. - - Returns - ------- - str - The code which declares the data type in C. - - Raises - ------ - TypeError - If the dtype is not found in the dtype_registry. - """ - if isinstance(dtype, CFIDescriptorType | CFIDimensionType): - self.add_import(c_imports["ISO_Fortran_binding"]) - return self.dtype_registry[dtype] - - if isinstance(dtype, FixedSizeNumericType): - primitive_type = dtype.primitive_type - if isinstance(primitive_type, PrimitiveComplexType): - self.add_import(c_imports["complex"]) - return f"{self._c_type(dtype.element_type)} complex" - if isinstance(primitive_type, PrimitiveIntegerType): - self.add_import(c_imports["stdint"]) - elif isinstance(dtype, NumpyBoolType): - self.add_import(c_imports["stdbool"]) - return "bool" - - key = (primitive_type, dtype.precision) - - elif isinstance(dtype, StringType): - self.add_import(c_imports["stc/cstr"]) - return "cstr" - - elif isinstance(dtype, CustomDataType): - return self._visit(dtype) - - else: - key = dtype - - try: - return self.dtype_registry[key] - except KeyError: - raise TypeError(f"Unsupported C dtype: {dtype}") from None - - def _get_declare_type(self, expr): - """ - Get the string which describes the type in a declaration. - - This function returns the code which describes the type - of the `expr` object such that the declaration can be written as: - `f"{self._get_declare_type(expr)} {expr.name}"` - The function takes care of reporting errors for unknown types and - importing any necessary additional imports (e.g. stdint/ndarrays). - - Parameters - ---------- - expr : Variable - The variable whose type should be described. - - Returns - ------- - str - The code describing the type. - - Raises - ------ - X2pyCodegenError - If the type is not supported in the C code. - - Examples - -------- - >>> v = Variable(NumpyInt64Type(), 'x') - >>> self._get_declare_type(v) - 'int64_t' - - For an object accessed via a pointer: - >>> v = Variable(NumpyNDArrayType.get_new(NumpyInt64Type(), 1, None), 'x', is_optional=True) - >>> self._get_declare_type(v) - 'array_int64_1d*' - """ - if isinstance(expr.class_type, CFIDescriptorStorageType): - self.add_import(c_imports["ISO_Fortran_binding"]) - return f"CFI_CDESC_T({expr.class_type.descriptor_rank})" - if self._is_native_array_descriptor_dummy(expr): - return "void*" - class_type = expr.class_type - - if isinstance(class_type, NumpyNDArrayType) and class_type.raw: - dtype = self._c_type(class_type.element_type) - elif isinstance(class_type, NumpyNDArrayType): - dtype = self._c_type(class_type) - else: - dtype = self._c_type(expr.class_type) - - if self._is_c_pointer(expr) and not (isinstance(class_type, NumpyNDArrayType) and class_type.raw): - return f"{dtype}*" - return dtype - - @staticmethod - def _is_native_array_descriptor_dummy(expr) -> bool: - """Return whether a bind(C) array dummy is passed as a TS29113 descriptor pointer in C.""" - policy = getattr(expr, "native_array_handle_policy", None) - return bool( - isinstance(policy, NativeArrayHandlePolicy) - and policy.handle_kind in {"argument_descriptor", "optional_absent_handle"} - and isinstance(expr.class_type, NumpyNDArrayType) - ) - - def _function_signature(self, expr, print_arg_names=True): - """ - Get the C representation of the function signature. - - Extract from the function definition `expr` all the - information (name, input, output) needed to create the - function signature and return a string describing the - function. - - This is not a declaration as the signature does not end - with a semi-colon. - - Parameters - ---------- - expr : FunctionDef - The function definition for which a signature is needed. - - print_arg_names : bool, default : True - Indicates whether argument names should be printed. - - Returns - ------- - str - Signature of the function. - """ - arg_vars = [a.var for a in expr.arguments] - result_vars = [v for v in expr.scope.collect_all_tuple_elements(expr.results.var) if v and not v.is_argument] - - n_results = len(result_vars) - - if n_results > 1: - ret_type = self._c_type(VoidType()) - if expr.arguments and expr.arguments[0].bound_argument: - # Place the first arg_var (the bound class object) first - arg_vars = arg_vars[:1] + result_vars + arg_vars[1:] - else: - arg_vars = result_vars + arg_vars - self._additional_args.append(result_vars) # Ensure correct result for _is_c_pointer - elif n_results == 1: - ret_type = self._get_declare_type(result_vars[0]) - self._additional_args.append([]) - else: - ret_type = self._c_type(VoidType()) - self._additional_args.append([]) - - for v in expr.global_vars: - if get_direct_module(v) is None: - self._additional_args[-1].append(v) - arg_vars.append(v) - arg_vars = [ai for a in arg_vars for ai in expr.scope.collect_all_tuple_elements(a)] - - name = expr.name - if not arg_vars: - arg_code = "void" - else: - - def get_arg_declaration(var): - """Get the code which declares the argument variable.""" - const = "const " if isinstance(var.class_type, FinalType) else "" - code = const + self._get_declare_type(var) - if print_arg_names: - code += " " + var.name - return code - - arg_code_list = [ - (self._function_signature(var, False) if isinstance(var, FunctionAddress) else get_arg_declaration(var)) - for var in arg_vars - ] - arg_code = ", ".join(arg_code_list) - - self._additional_args.pop() - - static = "static " if expr.is_static else "" - - if isinstance(expr, FunctionAddress): - return f"{static}{ret_type} (*{name})({arg_code})" - return f"{static}{ret_type} {name}({arg_code})" - - @staticmethod - def _result_vars(func): - """Handle result vars for the current generation context.""" - if func.scope is None: - return [func.results.var] if func.results.var is not NIL else [] - return [v for v in func.scope.collect_all_tuple_elements(func.results.var) if isinstance(v, Variable)] - - def _returns_via_output_args(self, func): - """Handle returns via output args for the current generation context.""" - return len(self._result_vars(func)) > 1 - - def _handle_is_operator(self, Op, expr): - """ - Get the code to print an `is` or `is not` expression. - - Get the code to print an `is` or `is not` expression. These two operators - function similarly so this helper function reduces code duplication. - - Parameters - ---------- - Op : str - The C operator representing "is" or "is not". - - expr : Is/IsNot - The expression being printed. - - Returns - ------- - str - The code describing the expression. - - Raises - ------ - X2pyError : Raised if the comparison is poorly defined. - """ - - lhs = self._visit(expr.args[0]) - rhs = self._visit(expr.args[1]) - a = expr.args[0] - b = expr.args[1] - - if NIL in expr.args: - lhs = ObjectAddress(expr.args[0]) if isinstance(expr.args[0], Variable) else expr.args[0] - rhs = ObjectAddress(expr.args[1]) if isinstance(expr.args[1], Variable) else expr.args[1] - - lhs = self._visit(lhs) - rhs = self._visit(rhs) - return f"{lhs} {Op} {rhs}" - - if a.dtype is NumpyBoolType() and b.dtype is NumpyBoolType(): - return f"{lhs} {Op} {rhs}" - raise TypeError("C is/is not printing is only supported for booleans and nil checks") - - def _indent_code(self, code): - """ - Add the necessary indentation to a string of code or a list of code lines. - - Add the necessary indentation to a string of code or a list of code lines. - - Parameters - ---------- - code : str | iterable[str] - The code which needs indenting. - - Returns - ------- - str | list[str] - The indented code. The type matches the type of the argument. - """ - - if isinstance(code, str): - code_lines = self._indent_code(code.splitlines(True)) - return "".join(code_lines) - - tab = " " * self._default_settings["tabwidth"] - - code = [line.lstrip(" \t") for line in code] - - increase = [int(line.endswith("{\n")) for line in code] - decrease = [int(any(map(line.startswith, "}\n"))) for line in code] - - pretty = [] - level = 0 - for n, line in enumerate(code): - if line == "" or line == "\n": - pretty.append(line) - continue - level -= decrease[n] - indent = tab * level - pretty.append(f"{indent}{line}") - level += increase[n] - return pretty diff --git a/x2py/codegen/printers/codeprinter.py b/x2py/codegen/printers/codeprinter.py deleted file mode 100644 index 55208ccac..000000000 --- a/x2py/codegen/printers/codeprinter.py +++ /dev/null @@ -1,150 +0,0 @@ -""" -Module containing the base class `CodePrinter` from which all code printers -inherit. The sub-classes should define a language and `_visit_X` functions. -The `CodePrinter` class also contains some general functionality which may be -used by all code printers, such as the management of imports and the current -scope. -""" - -from x2py.utilities.visitor import ClassVisitor - -from ..models.core import Module, ModuleHeader - -# TODO: add examples - -__all__ = ["CodePrinter"] - - -class CodePrinter(ClassVisitor): - """ - The base class for code-printing subclasses. - - The base class from which code printers inherit. The sub-classes should define a language - and `_visit_X` functions. - - Parameters - ---------- - verbose : int - The level of verbosity. - """ - - language = None - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, verbose): - """Initialize the state used for one generation run.""" - self._scope = None - self._additional_imports = {} - self._verbose = verbose - - def doprint(self, expr): - """ - Print the expression as code. - - Print the expression as code. - - Parameters - ---------- - expr : Expression - The expression to be printed. - - Returns - ------- - str - The generated code. - """ - assert isinstance(expr, Module | ModuleHeader) - - # Do the actual printing - lines = self._visit(expr).splitlines(True) - - # Format the output - return "".join(self._format_code(lines)) - - def get_additional_imports(self): - """ - Get any additional imports collected during the printing stage. - - Get any additional imports collected during the printing stage. - This is necessary to correctly compile the files. - - Returns - ------- - dict[str, Import] - A dictionary mapping the include strings to the import module. - """ - return self._additional_imports - - def add_import(self, import_obj): - """ - Add a new import to the current context. - - Add a new import to the current context. This allows the import to be recognised - at the compiling/linking stage. If the source of the import is not new then any - new targets are added to the Import object. - - Parameters - ---------- - import_obj : Import - The AST node describing the import. - """ - source = str(import_obj.source) - if source not in self._additional_imports: - self._additional_imports[source] = import_obj - elif import_obj.target: - self._additional_imports[source].define_target(import_obj.target) - - @property - def scope(self): - """Return the scope associated with the object being printed""" - return self._scope - - def set_scope(self, scope): - """Change the current scope""" - assert scope is not None - self._scope = scope - - def exit_scope(self): - """Exit the current scope and return to the enclosing scope""" - self._scope = self._scope.parent_scope - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_NumberSymbol(self, expr): - """Print sympy symbols used for constants""" - return str(expr) - - def _visit_str(self, expr): - """Basic print functionality for strings""" - return expr - - def _visit_not_supported(self, expr): - """Raise an error when no visitor supports the model type.""" - msg = f"_visit_{type(expr).__name__} is not yet implemented for language : {self.language}\n" - raise NotImplementedError(msg) - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _declare_number_const(self, name, value): - """Declare a numeric constant at the top of a function""" - raise NotImplementedError("This function must be implemented by subclass of CodePrinter.") - - def _format_code(self, lines): - """Take in a list of lines of code, and format them accordingly. - - This may include indenting, wrapping long lines, etc...""" - raise NotImplementedError("This function must be implemented by subclass of CodePrinter.") - - # Number constants - _visit_Catalan = _visit_NumberSymbol - _visit_EulerGamma = _visit_NumberSymbol - _visit_GoldenRatio = _visit_NumberSymbol - _visit_Exp1 = _visit_NumberSymbol - _visit_Pi = _visit_NumberSymbol diff --git a/x2py/codegen/printers/cpythoncode.py b/x2py/codegen/printers/cpythoncode.py deleted file mode 100644 index da0eb1a37..000000000 --- a/x2py/codegen/printers/cpythoncode.py +++ /dev/null @@ -1,1318 +0,0 @@ -""" -Module containing the `CWrapperCodePrinter` class which is responsible for -printing the C-Python interface. -""" - -from typing import ClassVar - -from x2py.semantics.models import ( - INTERNAL_MODULE_VARIABLE_NAME_METADATA, - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA, -) - -from ..bind_c import BindCFunctionDef, BindCModule, BindCPointer -from ..bindings.c_concepts import CStrStr, ObjectAddress -from ..models.core import Declare, FunctionAddress, Import, Module, SeparatorComment -from ..bindings.cpython_api import ( - Py_None, - Py_ssize_t, - PyBuildValueNode, - PyCallbackContextPush, - PyCapsule_Import, - PyCapsule_New, - PyFunctionOverloadSet, - PythonObjectType, - PythonTypeObjectType, - PyModule_Create, - PyTuple_Pack, - PythonClassType, - WrapperCustomDataType, - c_to_py_registry, - py_to_c_registry, -) -from ..models.datatypes import ( - FinalType, - Literal, - NIL, - NumpyNDArrayType, - PrimitiveBooleanType, - PrimitiveComplexType, - PrimitiveFloatingPointType, - PrimitiveIntegerType, - convert_to_literal, -) -from ..bindings.numpy_cpython_api import NumpyArrayObjectType -from .ccode import CCodePrinter - -__all__ = ("CPythonCodePrinter",) - -module_imports = [ - Import("numpy_version", Module("numpy_version", (), ())), - Import("numpy/arrayobject", Module("numpy/arrayobject", (), ())), - Import("x2py_runtime/python_runtime", Module("x2py_runtime", (), ())), -] - - -class CPythonCodePrinter(CCodePrinter): - """ - A printer for printing the C-Python interface. - - A printer to convert X2py's AST describing a translated module, - to strings of C code which provide an interface between the module - and Python code. - As for all printers the navigation of this file is done via _visit_X - functions. - - Parameters - ---------- - filename : str - The name of the file being converted. - **settings : dict - Any additional arguments which are necessary for CCodePrinter. - """ - - dtype_registry: ClassVar = { - **CCodePrinter.dtype_registry, - PythonObjectType(): "PyObject", - NumpyArrayObjectType(): "PyArrayObject", - PythonTypeObjectType(): "PyTypeObject", - BindCPointer(): "void", - } - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, filename, **settings): - """Initialize the state used for one generation run.""" - CCodePrinter.__init__(self, filename, **settings) - self._to_free_PyObject_list = [] - self._function_wrapper_names = {} - self._module_name = None - - # -------------------------------------------------------------------- - # Helper functions - # -------------------------------------------------------------------- - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_PyCallbackValidate(self, expr): - """Render the ``PyCallbackValidate`` model node.""" - metadata = expr.callback.decorators.get("x2py_callback_abi", {}) - callback_name = str(getattr(metadata.get("native"), "name", expr.callback.name)) - python_object = self._visit(ObjectAddress(expr.python_object)) - return ( - f"if (!PyCallable_Check({python_object})) {{\n" - f' PyErr_SetString(PyExc_TypeError, "callback {callback_name} must be callable");\n' - f" return {self._visit(expr.error_exit)};\n" - "}\n" - ) - - def _visit_PyCallbackContextPush(self, expr): - """Render the ``PyCallbackContextPush`` model node.""" - context_type, current_name, _ = self._callback_context_names(expr.callback) - context_name = f"{self._callback_identifier(expr.callback)}_context" - python_object = self._visit(ObjectAddress(expr.python_object)) - return ( - f"{context_type} {context_name} = " - f"{{{python_object}, PyThread_get_thread_ident(), {current_name}, NULL}};\n" - f"Py_INCREF({python_object});\n" - f"{current_name} = &{context_name};\n" - ) - - def _visit_PyCallbackContextPop(self, expr): - """Render the ``PyCallbackContextPop`` model node.""" - _, current_name, _ = self._callback_context_names(expr.callback) - context_name = f"{self._callback_identifier(expr.callback)}_context" - return ( - f"{current_name} = {context_name}.previous;\n" - f"Py_XDECREF({context_name}.last_result);\n" - f"Py_DECREF({context_name}.callable);\n" - ) - - def _visit_PyAllowThreadsBegin(self, expr): - """Render the ``PyAllowThreadsBegin`` model node.""" - return "Py_BEGIN_ALLOW_THREADS\n" - - def _visit_PyAllowThreadsEnd(self, expr): - """Render the ``PyAllowThreadsEnd`` model node.""" - return "Py_END_ALLOW_THREADS\n" - - def _visit_PyFunctionDef(self, expr): - """Render the ``PyFunctionDef`` model node.""" - callbacks = [item.callback for item in expr.body.body if isinstance(item, PyCallbackContextPush)] - support = "".join(self._callback_support_code(callback) for callback in callbacks) - return support + CCodePrinter._visit_FunctionDef(self, expr) - - def _visit_PyFunctionOverloadSet(self, expr): - """Render the ``PyFunctionOverloadSet`` model node.""" - funcs_to_visit = (*expr.functions, expr.type_check_func, expr.dispatcher_func) - return "\n".join(self._visit(f) for f in funcs_to_visit) - - def _visit_PyArg_ParseTupleNode(self, expr): - """Render the ``PyArg_ParseTupleNode`` model node.""" - name = "PyArg_ParseTupleAndKeywords" - pyarg = expr.pyarg - pykwarg = expr.pykwarg - flags = expr.flags - # All args are modified so even pointers are passed by address - args = ", ".join(f"&{a.name}" for a in expr.args) - - if expr.args: - code = f'{name}({pyarg}, {pykwarg}, "{flags}", {expr.arg_names.name}, {args})' - else: - code = f'{name}({pyarg}, {pykwarg}, "", {expr.arg_names.name})' - - return code - - def _visit_PyBuildValueNode(self, expr): - """Render the ``PyBuildValueNode`` model node.""" - name = "Py_BuildValue" - flags = expr.flags - args = ", ".join(self._visit(a) for a in expr.args) - # to change for args rank 1 + - return f'(*{name}("{flags}", {args}))' if expr.args else f'(*{name}(""))' - - def _visit_PyArgKeywords(self, expr): - """Render the ``PyArgKeywords`` model node.""" - arg_names = ",\n".join([f'(char*)"{a}"' for a in expr.arg_names] + [self._visit(NIL)]) - return f"static char *{expr.name}[] = {{\n{arg_names}\n}};\n" - - def _visit_PyModule_AddObject(self, expr): - """Render the ``PyModule_AddObject`` model node.""" - name = self._visit(expr.name) - var = self._visit(expr.variable) - if expr.variable.dtype is not PythonObjectType(): - var = f"(PyObject*) {var}" - return f"PyModule_AddObject({expr.mod_name}, {name}, {var})" - - def _visit_PyCapsule_New(self, expr): - """Render the ``PyCapsule_New`` model node.""" - name = expr.capsule_name - var = self._visit(ObjectAddress(expr.API_var)) - return f'PyCapsule_New((void *){var}, "{name}", NULL)' - - def _visit_PyCapsule_Import(self, expr): - """Render the ``PyCapsule_Import`` model node.""" - name = expr.capsule_name - return f'(void**)PyCapsule_Import("{name}", 0)' - - def _visit_PyModule_Create(self, expr): - """Render the ``PyModule_Create`` model node.""" - return f"PyModule_Create(&{expr.module_def_name})" - - def _visit_PyModule_SetPropertyType(self, expr): - """Render one generated module-property type setup call.""" - return f"{expr.setup_name}({self._visit(ObjectAddress(expr.module))})" - - def _visit_ModuleHeader(self, expr): - """Render the ``ModuleHeader`` model node.""" - mod = expr.module - self.set_scope(mod.scope) - name = mod.name - - # Print imports last to be sure that all additional_imports have been collected - imports = [*module_imports, *mod.imports] - for i in imports: - self.add_import(i) - imports = "".join(self._visit(i) for i in imports) - - function_signatures = "".join( - self._function_signature(f, print_arg_names=False) + ";\n" for f in mod.external_funcs - ) - - API_var = mod.variables[0] - - macro_defs = "" - type_declarations = "" - classes = [] - for i, c in enumerate(mod.classes): - struct_name = c.struct_name - type_name = c.type_name - attributes = "".join(self._visit(Declare(a)) for a in c.attributes) - classes.append(f"struct {struct_name} {{\n PyObject_HEAD\n" + attributes + "};\n") - type_declarations += f"static PyTypeObject {c.type_name};\n" - sig_methods = ( - *c.methods, - c.new_func, - *tuple(f for i in c.overload_sets for f in i.functions), - *tuple(i.dispatcher_func for i in c.overload_sets), - *tuple(getset for p in c.properties for getset in (p.getter, p.setter) if getset), - *tuple( - method.dispatcher_func if isinstance(method, PyFunctionOverloadSet) else method - for method in c.magic_methods - ), - ) - function_signatures += "\n" + "".join(self._function_signature(f) + ";\n" for f in sig_methods) - macro_defs += f"#define {type_name} (*(PyTypeObject*){API_var.name}[{i}])\n" - - class_code = "\n".join(classes) - - static_import_decs = self._visit(Declare(API_var, static=True)) - import_func = self._visit(mod.import_func) - - self.exit_scope() - header_id = f"{name.upper()}_WRAPPER" - header_guard = f"{header_id}_H" - start = f"#ifndef {header_guard}\n#define {header_guard}\n" - end = f"#endif\n#endif // {header_guard}\n" - parts = ( - start, - imports, - class_code, - f"#ifdef {header_id}\n", - type_declarations, - function_signatures, - "#else\n", - static_import_decs, - macro_defs, - import_func, - end, - ) - return "\n".join(p for p in parts if p) - - def _visit_PyModule(self, expr): - """Render the ``PyModule`` model node.""" - scope = expr.scope - self.set_scope(scope) - - # Insert declared objects into scope - variables = expr.original_module.variables if isinstance(expr, BindCModule) else expr.variables - for f in expr.funcs: - scope.insert_symbol(f.name.lower()) - for v in variables: - if not v.is_private: - scope.insert_symbol(v.name.lower()) - - funcs = [] - - self._module_name = expr.name - sep = self._visit(SeparatorComment(40)) - - dispatcher_funcs = [f.name for i in expr.overload_sets for f in i.functions] - funcs += [ - *expr.overload_sets, - *(f for f in expr.funcs if f.name not in dispatcher_funcs), - ] - - self._in_header = True - decs = "".join(self._visit(d) for d in expr.declarations) - self._in_header = False - - function_defs = "\n".join(self._visit(f) for f in funcs) - - class_defs = f"\n{sep}\n".join(self._visit(c) for c in expr.classes) - - namespace_defs, namespace_functions, namespace_classes = self._module_namespace_exports(expr, funcs) - method_defs, module_defs = self._module_definition_blocks( - expr, - namespace_defs, - namespace_functions, - namespace_classes, - ) - property_defs = self._module_property_blocks(expr) - - init_func = self._visit(expr.init_func) - rendered_body_parts = [ - decs, - class_defs, - function_defs, - *method_defs, - *property_defs, - *module_defs, - init_func, - ] - numpy_bytes_helper = ( - self._x2py_numpy_bytes_array_helper() - if any("x2py_to_numpy_bytes_array(" in part for part in rendered_body_parts) - else "" - ) - - pymod_name = f"{expr.name}_wrapper" - imports = [ - Import(pymod_name, Module(pymod_name, (), ())), - *self._additional_imports.values(), - ] - imports = "".join(self._visit(i) for i in imports) - - self.exit_scope() - - return "\n".join( - [ - "#define PY_ARRAY_UNIQUE_SYMBOL CWRAPPER_ARRAY_API", - f"#define {pymod_name.upper()}\n", - imports, - self._x2py_malloc_helper(), - numpy_bytes_helper, - decs, - sep, - class_defs, - sep, - function_defs, - sep, - *method_defs, - sep, - *property_defs, - sep, - *module_defs, - sep, - init_func, - ] - ) - - @staticmethod - def _x2py_numpy_bytes_array_helper(): - """Create a fixed-width NumPy bytes array from native-owned data.""" - return ( - "#include \n" - "#include \n" - "static PyObject* x2py_to_numpy_bytes_array(int nd, void* data, int32_t* shape,\n" - " int64_t itemsize, bool c_order,\n" - " bool release_memory)\n" - "{\n" - " if (itemsize < 0) {\n" - " if (release_memory) free(data);\n" - ' PyErr_SetString(PyExc_ValueError, "bytes array itemsize must be non-negative");\n' - " return NULL;\n" - " }\n" - " if (nd < 0 || nd > NPY_MAXDIMS) {\n" - " if (release_memory) free(data);\n" - ' PyErr_SetString(PyExc_ValueError, "unsupported array rank");\n' - " return NULL;\n" - " }\n" - " npy_intp dims[NPY_MAXDIMS];\n" - " for (int i = 0; i < nd; ++i) {\n" - " int source = c_order ? nd - i - 1 : i;\n" - " dims[i] = (npy_intp)shape[source];\n" - " }\n" - " PyArray_Descr* descr = PyArray_DescrNewFromType(NPY_STRING);\n" - " if (descr == NULL) {\n" - " if (release_memory) free(data);\n" - " return NULL;\n" - " }\n" - "#if defined(PyDataType_SET_ELSIZE)\n" - " PyDataType_SET_ELSIZE(descr, (npy_intp)itemsize);\n" - "#else\n" - " descr->elsize = (int)itemsize;\n" - "#endif\n" - " int flags = NPY_ARRAY_ALIGNED;\n" - " flags |= c_order ? NPY_ARRAY_C_CONTIGUOUS : NPY_ARRAY_F_CONTIGUOUS;\n" - " PyObject* arr = PyArray_NewFromDescr(&PyArray_Type, descr, nd, dims, NULL, data, flags, NULL);\n" - " if (arr == NULL) {\n" - " if (release_memory) free(data);\n" - " return NULL;\n" - " }\n" - " if (release_memory) PyArray_ENABLEFLAGS((PyArrayObject*)arr, NPY_ARRAY_OWNDATA);\n" - " return arr;\n" - "}\n" - ) - - def _module_namespace_exports(self, expr, funcs): - """Group wrapped functions and classes by Python module namespace.""" - namespace_defs = {(): expr.module_def_name, **expr.namespace_module_defs} - namespace_functions = {namespace: [] for namespace in namespace_defs} - for function in funcs: - if getattr(function, "is_header", False): - continue - original = getattr(function, "original_function", None) - decorators = getattr(original, "decorators", {}) - if INTERNAL_MODULE_VARIABLE_NAME_METADATA in decorators and not decorators.get( - INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA - ): - continue - exports = ( - expr.get_python_exports(function) - if expr.has_explicit_python_exports - else (((), self._get_python_name(expr.scope, function.original_function)),) - ) - for namespace, export_name in exports: - namespace_functions[namespace].append((export_name, function)) - - namespace_classes = {namespace: [] for namespace in namespace_defs} - for wrapped_class in expr.classes: - exports = ( - expr.get_python_exports(wrapped_class) - if expr.has_explicit_python_exports - else (((), str(expr.scope.get_python_name(wrapped_class.name))),) - ) - for namespace, export_name in exports: - namespace_classes[namespace].append(export_name) - return namespace_defs, namespace_functions, namespace_classes - - def _module_definition_blocks(self, expr, namespace_defs, namespace_functions, namespace_classes): - """Render PyMethodDef arrays and PyModuleDef blocks for namespaces.""" - method_defs = [] - module_defs = [] - for namespace, definition_name in namespace_defs.items(): - method_entries = "".join( - ( - '{{\n"{name}",\n(PyCFunction){wrapper_name},\nMETH_VARARGS | METH_KEYWORDS,\n{docstring}\n}},\n' - ).format( - name=export_name, - wrapper_name=function.name, - docstring=( - self._visit(CStrStr(convert_to_literal("\n".join(function.docstring.comments)))) - if function.docstring - else '""' - ), - ) - for export_name, function in namespace_functions[namespace] - ) - suffix = "root" if not namespace else "_".join(namespace) - method_name = self.scope.get_new_name(f"{expr.name}_{suffix}_methods", object_type="wrapper") - method_defs.append( - f"static PyMethodDef {method_name}[] = {{\n{method_entries}{{ NULL, NULL, 0, NULL}}\n}};\n" - ) - qualified_name = ".".join((str(self._module_name), *namespace)) - exported_names = [name for name, _ in namespace_functions[namespace]] - module_docstring = self._visit( - CStrStr( - convert_to_literal( - "\n".join( - ( - qualified_name, - "", - "Functions", - "---------", - *exported_names, - "", - "Classes", - "-------", - *namespace_classes[namespace], - ) - ) - ) - ) - ) - module_defs.append( - f"static struct PyModuleDef {definition_name} = {{\n" - "PyModuleDef_HEAD_INIT,\n" - "/* name of module */\n" - f'"{qualified_name}",\n' - "/* module documentation, may be NULL */\n" - f"{module_docstring},\n" - "/* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */\n" - "0,\n" - f"{method_name},\n" - "};\n" - ) - return method_defs, module_defs - - def _module_property_blocks(self, expr): - """Render custom module types that route attributes through native accessors.""" - return [ - self._module_property_block(namespace, descriptor) - for namespace, descriptor in expr.module_properties.items() - ] - - def _module_property_block(self, namespace, descriptor): - """Render a custom module type for native-backed attributes.""" - setup_name = descriptor["setup_name"] - items = descriptor["items"] - get_name = f"{setup_name}_getattro" - set_name = f"{setup_name}_setattro" - slots_name = f"{setup_name}_slots" - spec_name = f"{setup_name}_spec" - qualified_name = ".".join((str(self._module_name), *namespace, "__x2py_module_type")) - - get_cases = "".join(self._module_property_get_case(name, accessors["get"]) for name, accessors in items.items()) - set_cases = "".join(self._module_property_set_case(name, accessors["set"]) for name, accessors in items.items()) - return ( - f"static PyObject *{get_name}(PyObject *self, PyObject *name)\n" - "{\n" - " if (PyUnicode_Check(name)) {\n" - f"{get_cases}" - " }\n" - " return PyModule_Type.tp_getattro(self, name);\n" - "}\n\n" - f"static int {set_name}(PyObject *self, PyObject *name, PyObject *value)\n" - "{\n" - " if (PyUnicode_Check(name)) {\n" - f"{set_cases}" - " }\n" - " return PyModule_Type.tp_setattro(self, name, value);\n" - "}\n\n" - f"static PyType_Slot {slots_name}[] = {{\n" - f" {{Py_tp_getattro, (void *){get_name}}},\n" - f" {{Py_tp_setattro, (void *){set_name}}},\n" - " {0, NULL}\n" - "};\n" - f"static PyType_Spec {spec_name} = {{\n" - f' "{qualified_name}",\n' - " 0,\n" - " 0,\n" - " Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,\n" - f" {slots_name}\n" - "};\n\n" - f"static int {setup_name}(PyObject *module)\n" - "{\n" - " PyObject *bases = PyTuple_Pack(1, (PyObject *)&PyModule_Type);\n" - " if (bases == NULL) {\n" - " return -1;\n" - " }\n" - f" PyObject *module_type = PyType_FromSpecWithBases(&{spec_name}, bases);\n" - " Py_DECREF(bases);\n" - " if (module_type == NULL) {\n" - " return -1;\n" - " }\n" - ' int status = PyObject_SetAttrString(module, "__class__", module_type);\n' - " Py_DECREF(module_type);\n" - " return status;\n" - "}\n" - ) - - @staticmethod - def _module_property_get_case(name, getter): - """Render one custom module-attribute getter branch.""" - if getter is None: - return "" - return ( - " {\n" - f' int comparison = PyUnicode_CompareWithASCIIString(name, "{name}");\n' - " if (comparison == -1 && PyErr_Occurred()) {\n" - " return NULL;\n" - " }\n" - " if (comparison == 0) {\n" - " PyObject *args = PyTuple_New(0);\n" - " if (args == NULL) {\n" - " return NULL;\n" - " }\n" - f" PyObject *result = (PyObject *){getter.name}(self, args, NULL);\n" - " Py_DECREF(args);\n" - " return result;\n" - " }\n" - " }\n" - ) - - @staticmethod - def _module_property_set_case(name, setter): - """Render one custom module-attribute setter branch.""" - prefix = ( - " {\n" - f' int comparison = PyUnicode_CompareWithASCIIString(name, "{name}");\n' - " if (comparison == -1 && PyErr_Occurred()) {\n" - " return -1;\n" - " }\n" - " if (comparison == 0) {\n" - ) - if setter is None: - return ( - prefix - + f' PyErr_SetString(PyExc_AttributeError, "module variable {name} is read-only");\n' - + " return -1;\n" - + " }\n" - + " }\n" - ) - return ( - prefix - + " if (value == NULL) {\n" - + f' PyErr_SetString(PyExc_AttributeError, "module variable {name} cannot be deleted");\n' - + " return -1;\n" - + " }\n" - + " PyObject *args = PyTuple_Pack(1, value);\n" - + " if (args == NULL) {\n" - + " return -1;\n" - + " }\n" - + f" PyObject *result = {setter.name}(self, args, NULL);\n" - + " Py_DECREF(args);\n" - + " if (result == NULL) {\n" - + " return -1;\n" - + " }\n" - + " Py_DECREF(result);\n" - + " return 0;\n" - + " }\n" - + " }\n" - ) - - def _visit_PyClassDef(self, expr): - """Render the ``PyClassDef`` model node.""" - struct_name = expr.struct_name - type_name = expr.type_name - name = self.scope.get_python_name(expr.name) - class_docstring = ( - self._visit(CStrStr(convert_to_literal("\n".join(expr.docstring.comments)))) if expr.docstring else '""' - ) - - original_scope = expr.original_class.scope - getters = tuple(p.getter for p in expr.properties) - setters = tuple(p.setter for p in expr.properties if p.setter) - print_methods = (*expr.methods, expr.new_func, *expr.overload_sets, *expr.magic_methods, *getters, *setters) - functions = "\n".join(self._visit(f) for f in print_methods) - init_string, del_string, funcs = self._class_method_metadata(expr, original_scope) - property_definitions = self._property_definitions(expr.properties) - - method_def_funcs = "".join( - (f'{{\n"{name}",\n(PyCFunction){wrapper_name},\n{flags},\n{doc_string}\n}},\n') - for name, (wrapper_name, doc_string, flags) in funcs.items() - ) - - magic_methods = {self._get_python_name(original_scope, f.original_function): f for f in expr.magic_methods} - magic_names, magic_definitions = self._magic_method_definitions(expr, magic_methods) - number_magic_method_name, seq_magic_method_name, map_magic_method_name = magic_names - number_magic_methods_def, seq_magic_methods_def, map_magic_methods_def = magic_definitions - method_def_name = self.scope.get_new_name(f"{expr.name}_methods", object_type="wrapper") - method_def = f"static PyMethodDef {method_def_name}[] = {{\n{method_def_funcs}{{ NULL, NULL, 0, NULL}}\n}};\n" - - property_def_name = self.scope.get_new_name(f"{expr.name}_properties", object_type="wrapper") - property_def = f"static PyGetSetDef {property_def_name}[] = {{\n{property_definitions}}};\n" - - richcompare_def, richcompare_slot = self._richcompare_definition(expr, magic_methods) - base_slot = self._base_type_slot(expr) - - type_code = ( - f"static PyTypeObject {type_name} = {{\n" - " PyVarObject_HEAD_INIT(NULL, 0)\n" - f' .tp_name = "{self._module_name}.{name}",\n' - f" .tp_as_number = &{number_magic_method_name},\n" - f" .tp_as_sequence = &{seq_magic_method_name},\n" - f" .tp_as_mapping = &{map_magic_method_name},\n" - f" .tp_doc = PyDoc_STR({class_docstring}),\n" - f" .tp_basicsize = sizeof(struct {struct_name}),\n" - " .tp_itemsize = 0,\n" - " .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,\n" - f" .tp_new = {expr.new_func.name},\n" - f"{base_slot}" - f"{init_string}{del_string}" - f"{richcompare_slot}" - f" .tp_methods = {method_def_name},\n" - f" .tp_getset = {property_def_name},\n" - "};\n" - ) - - return "\n".join( - ( - method_def, - number_magic_methods_def, - seq_magic_methods_def, - map_magic_methods_def, - property_def, - richcompare_def, - type_code, - functions, - ) - ) - - def _method_docstring(self, function): - """Render a method docstring as a C string.""" - if not function.docstring: - return '""' - return self._visit(CStrStr(convert_to_literal("\n".join(function.docstring.comments)))) - - def _class_method_metadata(self, expr, original_scope): - """Collect class slots and Python method-table metadata.""" - init_string = "" - del_string = "" - functions = {} - for function in expr.methods: - python_name = self._get_python_name(original_scope, function.original_function) - if python_name == "__init__": - init_string = f" .tp_init = (initproc) {function.name},\n" - continue - if python_name == "__del__": - del_string = f" .tp_dealloc = (destructor) {function.name},\n" - continue - original_args = function.original_function.arguments - flags = "METH_VARARGS | METH_KEYWORDS" - if not original_args or not original_args[0].bound_argument: - flags += " | METH_STATIC" - functions[python_name] = (function.name, self._method_docstring(function), flags) - for function in expr.overload_sets: - python_name = self._get_python_name(original_scope, function.original_function) - functions[python_name] = ( - function.name, - self._method_docstring(function), - "METH_VARARGS | METH_KEYWORDS", - ) - return init_string, del_string, functions - - def _property_definitions(self, properties): - """Render property entries for a CPython get-set table.""" - definitions = "".join( - "".join( - ( - "{\n", - f'"{prop.python_name}",\n', - f"(getter) {prop.getter.name},\n", - f"(setter) {prop.setter.name},\n" if prop.setter else "(setter) NULL,\n", - f"{self._visit(prop.docstring)},\n", - "NULL\n", - "},\n", - ) - ) - for prop in properties - ) - return definitions + "{ NULL }\n" - - def _magic_method_definitions(self, expr, magic_methods): - """Render CPython number, sequence, and mapping slot tables.""" - number_name = self.scope.get_new_name(f"{expr.name}_number_methods", object_type="wrapper") - number_slots = ( - ("__add__", "nb_add", "binaryfunc"), - ("__sub__", "nb_subtract", "binaryfunc"), - ("__mul__", "nb_multiply", "binaryfunc"), - ("__truediv__", "nb_true_divide", "binaryfunc"), - ("__pow__", "nb_power", "ternaryfunc"), - ("__neg__", "nb_negative", "unaryfunc"), - ("__pos__", "nb_positive", "unaryfunc"), - ("__invert__", "nb_invert", "unaryfunc"), - ("__lshift__", "nb_lshift", "binaryfunc"), - ("__rshift__", "nb_rshift", "binaryfunc"), - ("__and__", "nb_and", "binaryfunc"), - ("__or__", "nb_or", "binaryfunc"), - ("__iadd__", "nb_inplace_add", "binaryfunc"), - ("__isub__", "nb_inplace_subtract", "binaryfunc"), - ("__imul__", "nb_inplace_multiply", "binaryfunc"), - ("__itruediv__", "nb_inplace_true_divide", "binaryfunc"), - ("__ilshift__", "nb_inplace_lshift", "binaryfunc"), - ("__irshift__", "nb_inplace_rshift", "binaryfunc"), - ("__iand__", "nb_inplace_and", "binaryfunc"), - ("__ior__", "nb_inplace_or", "binaryfunc"), - ) - number_body = "".join( - f" .{slot} = ({cast}){magic_methods[python_name].name},\n" - for python_name, slot, cast in number_slots - if python_name in magic_methods - ) - number_def = f"static PyNumberMethods {number_name} = {{\n{number_body}}};\n" - - sequence_name = self.scope.get_new_name(f"{expr.name}_sequence_methods", object_type="wrapper") - sequence_body = self._optional_magic_slot(magic_methods, "__len__", "sq_length", "lenfunc", spaces=4) - sequence_def = f"static PySequenceMethods {sequence_name} = {{\n{sequence_body}}};\n" - - mapping_name = self.scope.get_new_name(f"{expr.name}_mapping_methods", object_type="wrapper") - mapping_body = self._optional_magic_slot(magic_methods, "__len__", "mp_length", "lenfunc", spaces=4) - mapping_body += self._optional_magic_slot(magic_methods, "__getitem__", "mp_subscript", "binaryfunc", spaces=5) - mapping_def = f"static PyMappingMethods {mapping_name} = {{\n{mapping_body}}};\n" - return (number_name, sequence_name, mapping_name), (number_def, sequence_def, mapping_def) - - @staticmethod - def _optional_magic_slot(magic_methods, python_name, slot, cast, *, spaces): - """Render one optional CPython magic-method slot.""" - method = magic_methods.get(python_name) - if method is None: - return "" - return f"{' ' * spaces}.{slot} = ({cast}){method.name},\n" - - def _richcompare_definition(self, expr, magic_methods): - """Render rich-comparison dispatch and its type slot.""" - comparison_ops = { - "__eq__": "Py_EQ", - "__ne__": "Py_NE", - "__lt__": "Py_LT", - "__le__": "Py_LE", - "__gt__": "Py_GT", - "__ge__": "Py_GE", - } - richcompare_methods = {name: magic_methods[name] for name in comparison_ops if name in magic_methods} - if not richcompare_methods: - return "", "" - richcompare_name = self.scope.get_new_name(f"{expr.name}_richcompare", object_type="wrapper") - cases = "".join( - f" case {comparison_ops[name]}:\n return {method.name}(lhs, rhs);\n" - for name, method in richcompare_methods.items() - ) - definition = ( - f"static PyObject *{richcompare_name}(PyObject *lhs, PyObject *rhs, int op)\n" - "{\n" - " switch (op) {\n" - f"{cases}" - " default:\n" - " Py_INCREF(Py_NotImplemented);\n" - " return Py_NotImplemented;\n" - " }\n" - "}\n" - ) - return definition, f" .tp_richcompare = {richcompare_name},\n" - - def _base_type_slot(self, expr): - """Render the CPython base-type slot for derived classes.""" - if not expr.original_class.superclasses: - return "" - base_class = expr.original_class.superclasses[0] - base_python_name = base_class.scope.get_python_name(base_class.name) - wrapped_base = self.scope.find(base_python_name, "classes", raise_if_missing=True) - return f" .tp_base = &{wrapped_base.type_name},\n" - - def _visit_PyModInitFunc(self, expr): - """Render the ``PyModInitFunc`` model node.""" - decs = "".join(self._visit(d) for d in expr.declarations) - body = self._visit(expr.body) - return "".join([f"PyMODINIT_FUNC {expr.name}(void)\n{{\n", decs, body, "}\n"]) - - def _visit_Allocate(self, expr): - """Render the ``Allocate`` model node.""" - variable = expr.variable - cls_base = variable.cls_base.original_class - class_def = self.scope.find(cls_base.scope.get_python_name(cls_base.name), "classes") - - type_name = class_def.type_name - var_code = self._visit(ObjectAddress(variable)) - decl_type = self._get_declare_type(variable) - return f"{var_code} = ({decl_type}){type_name}.tp_alloc(&{type_name}, 0);\n" - - def _visit_Deallocate(self, expr): - """Render the ``Deallocate`` model node.""" - variable = expr.variable - if isinstance(variable.dtype, WrapperCustomDataType): - cls_base = variable.cls_base.original_class - class_def = self.scope.find(cls_base.scope.get_python_name(cls_base.name), "classes") - - type_name = class_def.type_name - var_code = self._visit(ObjectAddress(variable)) - return f"{type_name}.tp_free({var_code});\n" - return CCodePrinter._visit_Deallocate(self, expr) - - def _visit_Declare(self, expr): - """Render the ``Declare`` model node.""" - var = expr.variable - if isinstance(var.dtype, BindCPointer): - declaration_type = "void*" - - static = "static " if expr.static else "" - external = "extern " if expr.external else "" - - variable = self._visit(expr.variable.name) - - init = f" = {self._visit(expr.value)}" if expr.value is not None else "" - if var.rank == 0: - return f"{static}{external}{declaration_type} {variable}{init};\n" - - size = var.shape[0] - if isinstance(size, Literal): - return f"{static}{external}{declaration_type} {variable}[{size}];\n" - return f"{static}{external}{declaration_type}* {variable}{init};\n" - return CCodePrinter._visit_Declare(self, expr) - - def _visit_IndexedElement(self, expr): - """Render the ``IndexedElement`` model node.""" - if isinstance(expr.base.class_type, NumpyNDArrayType) and expr.base.class_type.raw: - base = self._visit(expr.base.name) - idxs = "".join(f"[{self._visit(a)}]" for a in expr.indices) - return f"{base}{idxs}" - return CCodePrinter._visit_IndexedElement(self, expr) - - def _visit_Cast(self, expr): - """Render the ``Cast`` model node.""" - if expr.dtype is Py_ssize_t(): - return f"(Py_ssize_t){self._visit(expr.arg)}" - return super()._visit_Cast(expr) - - def _visit_PyTuple_Pack(self, expr): - """Render the ``PyTuple_Pack`` model node.""" - args = expr.args - n = len(args) - if n: - args_code = ", ".join(self._visit(a) for a in args) - return f"(*PyTuple_Pack( {n}, {args_code} ))" - return f"(*PyTuple_Pack( {n} ))" - - def _visit_PyArgumentError(self, expr): - """Render the ``PyArgumentError`` model node.""" - args = ", ".join( - [f'"{self._visit(expr.error_msg)}"'] - + [f"PyObject_Str((PyObject*)Py_TYPE({self._visit(a)}))" for a in expr.args] - ) - return f"PyErr_SetObject({self._visit(expr.error_type)}, PyUnicode_FromFormat({args}));\n" - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _is_c_pointer(self, a): - """ - Indicate whether the object is a pointer in C code. - - This function extends `CCodePrinter._is_c_pointer` to specify more objects - which are always accessed via a C pointer. - - Parameters - ---------- - a : model object - The object whose storage we are enquiring about. - - Returns - ------- - bool - True if a C pointer, False otherwise. - - See Also - -------- - CCodePrinter._is_c_pointer : The extended function. - """ - if isinstance(a, FunctionAddress): - return False - if ( - isinstance(a.class_type, WrapperCustomDataType | BindCPointer | PyTuple_Pack) - or (isinstance(a.class_type, NumpyNDArrayType) and a.class_type.raw) - ) or isinstance(a, PyBuildValueNode | PyCapsule_New | PyCapsule_Import | PyModule_Create): - return True - return CCodePrinter._is_c_pointer(self, a) - - def _get_python_name(self, scope, obj): - """ - Get the name of object as defined in the original python code. - - Get the name of the object as it was originally defined in the - Python code being translated. This name may have changed before - the printing stage in the case of name clashes or language interfaces. - - Parameters - ---------- - scope : x2py.parser.scope.Scope - The scope where the object was defined. - - obj : codegen model object - The object whose name we wish to identify. - - Returns - ------- - str - The original name of the object. - """ - if isinstance(obj, BindCFunctionDef): - return scope.get_python_name(obj.original_function.name) - if isinstance(obj, BindCModule): - return obj.original_module.name - return scope.get_python_name(obj.name) - - def _function_signature(self, expr, print_arg_names=True): - """Handle function signature for the current generation context.""" - args = list(expr.arguments) - if any(isinstance(a.var, FunctionAddress) and not a.var.decorators.get("x2py_callback_abi") for a in args): - return "" - return CCodePrinter._function_signature(self, expr, print_arg_names) - - def _get_declare_type(self, expr): - """ - Get the string which describes the type in a declaration. - - This function extends `CCodePrinter._get_declare_type` to specify types - which are only relevant in the C-Python interface. - - Parameters - ---------- - expr : Variable - The variable whose type should be described. - - Returns - ------- - str - The code describing the type. - - Raises - ------ - X2pyCodegenError - If the type is not supported in the C code or the rank is too large. - - See Also - -------- - CCodePrinter._get_declare_type : The extended function. - """ - if expr.dtype is BindCPointer(): - if isinstance(expr.class_type, FinalType): - return "const void*" - return "void*" - if expr.dtype is Py_ssize_t(): - dtype = "Py_ssize_t*" if self._is_c_pointer(expr) else "Py_ssize_t" - if isinstance(expr.class_type, FinalType): - return f"const {dtype}" - return dtype - return CCodePrinter._get_declare_type(self, expr) - - @staticmethod - def _callback_identifier(callback): - """Handle callback identifier for the current generation context.""" - return str(callback.name).replace("-", "_") - - def _callback_context_names(self, callback): - """Handle callback context names for the current generation context.""" - identifier = self._callback_identifier(callback) - return ( - f"x2py_callback_context_{identifier}", - f"x2py_callback_current_{identifier}", - f"x2py_callback_abort_{identifier}", - ) - - @staticmethod - def _callback_numpy_typenum(dtype): - """Handle callback numpy typenum for the current generation context.""" - primitive = dtype.primitive_type - precision = dtype.precision - mapping = { - (PrimitiveBooleanType(), -1): "NPY_BOOL", - (PrimitiveIntegerType(), 1): "NPY_INT8", - (PrimitiveIntegerType(), 2): "NPY_INT16", - (PrimitiveIntegerType(), 4): "NPY_INT32", - (PrimitiveIntegerType(), 8): "NPY_INT64", - (PrimitiveFloatingPointType(), 4): "NPY_FLOAT32", - (PrimitiveFloatingPointType(), 8): "NPY_FLOAT64", - (PrimitiveComplexType(), 4): "NPY_COMPLEX64", - (PrimitiveComplexType(), 8): "NPY_COMPLEX128", - } - try: - return mapping[(primitive, precision)] - except KeyError: - raise TypeError(f"Unsupported callback NumPy dtype {dtype}") from None - - def _callback_scalar_to_python(self, var, value): - """Handle callback scalar to python for the current generation context.""" - try: - cast_function = c_to_py_registry[var.dtype] - except KeyError: - raise TypeError(f"Unsupported callback scalar type {var.class_type}") from None - return f"{cast_function}(&{value})" - - def _callback_scalar_from_python(self, var, value): - """Handle callback scalar from python for the current generation context.""" - primitive = var.dtype.primitive_type - c_type = self._get_declare_type(var) - try: - cast_function = py_to_c_registry[(primitive, var.dtype.precision)] - except KeyError: - raise TypeError(f"Unsupported callback scalar type {var.class_type}") from None - return f"({c_type}){cast_function}({value})" - - def _callback_wrapped_class(self, native_var, callback): - """Handle callback wrapped class for the current generation context.""" - wrapped = self.scope.find(native_var.dtype.name, "classes") - if wrapped is None: - raise TypeError(f"Callback derived type {native_var.dtype.name} has no generated Python wrapper") - return wrapped - - def _callback_argument_code(self, callback, mapping, index, abort_name): - """Handle callback argument code for the current generation context.""" - native = mapping["native"] - abi = mapping["abi"] - py_name = f"callback_arg_{index}" - if mapping["kind"] == "scalar": - expression = self._callback_scalar_to_python(native, str(abi[0].name)) - setup = f"PyObject *{py_name} = {expression};\n" - elif mapping["kind"] == "scalar_storage": - data = abi[0] - flags = "NPY_ARRAY_ALIGNED" - decision = getattr(native, "ownership_decision", None) - if bool(getattr(native, "projected_output", False) or getattr(decision, "mutates_native", False)): - flags += " | NPY_ARRAY_WRITEABLE" - setup = ( - f"PyObject *{py_name} = PyArray_New(&PyArray_Type, 0, NULL, " - f"{self._callback_numpy_typenum(native.dtype)}, NULL, {data.name}, 0, {flags}, NULL);\n" - ) - elif mapping["kind"] == "string": - data, length = abi - setup = ( - f"PyObject *{py_name} = PyUnicode_FromStringAndSize(" - f"(const char *){data.name}, (Py_ssize_t){length.name});\n" - ) - elif mapping["kind"] == "string_storage": - data, length = abi - flags = "NPY_ARRAY_ALIGNED" - decision = getattr(native, "ownership_decision", None) - if bool(getattr(native, "projected_output", False) or getattr(decision, "mutates_native", False)): - flags += " | NPY_ARRAY_WRITEABLE" - setup = ( - f"PyArray_Descr *callback_descr_{index} = PyArray_DescrNewFromType(NPY_STRING);\n" - f'if (callback_descr_{index} == NULL) {abort_name}("failed to create callback bytes dtype");\n' - "#if defined(PyDataType_SET_ELSIZE)\n" - f"PyDataType_SET_ELSIZE(callback_descr_{index}, (npy_intp){length.name});\n" - "#else\n" - f"callback_descr_{index}->elsize = (int){length.name};\n" - "#endif\n" - f"PyObject *{py_name} = PyArray_NewFromDescr(&PyArray_Type, callback_descr_{index}, " - f"0, NULL, NULL, {data.name}, {flags}, NULL);\n" - ) - elif mapping["kind"] == "array": - data, *shape = abi - dims_name = f"callback_dims_{index}" - strides_name = f"callback_strides_{index}" - dimensions = ", ".join(f"(npy_intp){item.name}" for item in shape) - stride_lines = [f"{strides_name}[0] = (npy_intp)sizeof({self._c_type(native.dtype)});"] - stride_lines.extend( - f"{strides_name}[{i}] = {strides_name}[{i - 1}] * {dims_name}[{i - 1}];" for i in range(1, native.rank) - ) - flags = "NPY_ARRAY_F_CONTIGUOUS | NPY_ARRAY_ALIGNED" - decision = getattr(native, "ownership_decision", None) - if bool(getattr(native, "projected_output", False) or getattr(decision, "mutates_native", False)): - flags += " | NPY_ARRAY_WRITEABLE" - setup = ( - f"npy_intp {dims_name}[{native.rank}] = {{{dimensions}}};\n" - f"npy_intp {strides_name}[{native.rank}];\n" + "\n".join(stride_lines) + "\n" - f"PyObject *{py_name} = PyArray_New(&PyArray_Type, {native.rank}, {dims_name}, " - f"{self._callback_numpy_typenum(native.dtype)}, {strides_name}, {data.name}, 0, {flags}, NULL);\n" - ) - elif mapping["kind"] == "derived": - wrapped = self._callback_wrapped_class(native, callback) - setup = ( - f"struct {wrapped.struct_name} *{py_name}_value = " - f"(struct {wrapped.struct_name} *){wrapped.type_name}.tp_alloc(&{wrapped.type_name}, 0);\n" - f"PyObject *{py_name} = (PyObject *){py_name}_value;\n" - f"if ({py_name} != NULL) {{\n" - f" {py_name}_value->instance = {abi[0].name};\n" - f" {py_name}_value->referenced_objects = PyList_New(0);\n" - f" {py_name}_value->is_alias = 1;\n" - "}\n" - ) - else: - raise TypeError(f"Unsupported callback ABI argument kind {mapping['kind']}") - return ( - setup - + f'if ({py_name} == NULL) {abort_name}("failed to convert callback argument");\n' - + f"PyTuple_SET_ITEM(callback_args, {index}, {py_name});\n" - ) - - def _callback_result_code(self, callback, result, context_name, abort_name): - """Handle callback result code for the current generation context.""" - kind = result["kind"] - native = result["native"] - if kind == "none": - return ( - "if (callback_result != Py_None) {\n" - ' PyErr_SetString(PyExc_TypeError, "callback subroutine must return None");\n' - f' {abort_name}("invalid callback return value");\n' - "}\n" - "Py_DECREF(callback_result);\n" - "PyGILState_Release(callback_gil);\n" - "return;\n" - ) - if kind == "scalar": - c_type = self._get_declare_type(native) - conversion = self._callback_scalar_from_python(native, "callback_result") - return ( - f"{c_type} callback_value = {conversion};\n" - f'if (PyErr_Occurred()) {abort_name}("invalid callback return value");\n' - "Py_DECREF(callback_result);\n" - "PyGILState_Release(callback_gil);\n" - "return callback_value;\n" - ) - if kind == "array": - shape_checks = [] - for index, item in enumerate(native.alloc_shape): - if item is not None: - shape_checks.append( - f"PyArray_DIM((PyArrayObject *)callback_result, {index}) != {self._visit(item)}" - ) - conditions = [ - "!PyArray_Check(callback_result)", - f"PyArray_TYPE((PyArrayObject *)callback_result) != {self._callback_numpy_typenum(native.dtype)}", - f"PyArray_NDIM((PyArrayObject *)callback_result) != {native.rank}", - "!PyArray_CHKFLAGS((PyArrayObject *)callback_result, NPY_ARRAY_F_CONTIGUOUS | NPY_ARRAY_ALIGNED)", - *shape_checks, - ] - condition = " ||\n ".join(conditions) - validation = ( - f"if ({condition}) {{\n" - ' PyErr_SetString(PyExc_TypeError, "callback returned an incompatible array");\n' - f' {abort_name}("invalid callback return value");\n' - "}\n" - ) - elif kind == "derived": - wrapped = self._callback_wrapped_class(native, callback) - validation = ( - f"if (!PyObject_TypeCheck(callback_result, &{wrapped.type_name})) {{\n" - f' PyErr_SetString(PyExc_TypeError, "callback must return {native.dtype.name}");\n' - f' {abort_name}("invalid callback return value");\n' - "}\n" - ) - else: - raise TypeError(f"Unsupported callback ABI result kind {kind}") - - pointer = ( - "PyArray_DATA((PyArrayObject *)callback_result)" - if kind == "array" - else f"((struct {self._callback_wrapped_class(native, callback).struct_name} *)callback_result)->instance" - ) - return ( - validation - + f"Py_XDECREF({context_name}->last_result);\n" - + f"{context_name}->last_result = callback_result;\n" - + f"void *callback_value = {pointer};\n" - + "PyGILState_Release(callback_gil);\n" - + "return callback_value;\n" - ) - - def _callback_support_code(self, callback): - """Handle callback support code for the current generation context.""" - metadata = callback.decorators["x2py_callback_abi"] - context_type, current_name, abort_name = self._callback_context_names(callback) - signature = self._function_signature(callback) - signature = signature.replace(f"(*{callback.name})", str(callback.name)) - argument_code = "".join( - self._callback_argument_code(callback, mapping, index, abort_name) - for index, mapping in enumerate(metadata["arguments"]) - ) - result_code = self._callback_result_code(callback, metadata["result"], "callback_context", abort_name) - return ( - f"typedef struct {context_type} {{\n" - " PyObject *callable;\n" - " unsigned long thread_id;\n" - f" struct {context_type} *previous;\n" - " PyObject *last_result;\n" - f"}} {context_type};\n" - f"static _Thread_local {context_type} *{current_name} = NULL;\n" - f"static void {abort_name}(const char *message)\n{{\n" - " if (!PyErr_Occurred()) PyErr_SetString(PyExc_RuntimeError, message);\n" - " PyErr_PrintEx(0);\n" - " abort();\n" - "}\n" - f"static {signature}\n{{\n" - f" {context_type} *callback_context = {current_name};\n" - " if (callback_context == NULL || callback_context->thread_id != PyThread_get_thread_ident()) {\n" - " PyGILState_STATE callback_thread_gil = PyGILState_Ensure();\n" - ' PyErr_SetString(PyExc_RuntimeError, "callback invoked outside its entering Python thread");\n' - f' {abort_name}("callback thread violation");\n' - " PyGILState_Release(callback_thread_gil);\n" - " }\n" - " PyGILState_STATE callback_gil = PyGILState_Ensure();\n" - f" PyObject *callback_args = PyTuple_New({len(metadata['arguments'])});\n" - f' if (callback_args == NULL) {abort_name}("failed to allocate callback arguments");\n' - + "".join(f" {line}\n" for line in argument_code.splitlines()) - + " PyObject *callback_result = PyObject_CallObject(callback_context->callable, callback_args);\n" - " Py_DECREF(callback_args);\n" - f' if (callback_result == NULL) {abort_name}("Python callback raised an exception");\n' - + "".join(f" {line}\n" for line in result_code.splitlines()) - + "}\n" - ) - - def _handle_is_operator(self, Op, expr): - """ - Get the code to print an `is` or `is not` expression. - - Get the code to print an `is` or `is not` expression. These two operators - function similarly so this helper function reduces code duplication. - This function overrides CCodePrinter._handle_is_operator to add the - handling of `Py_None`. - - Parameters - ---------- - Op : str - The C operator representing "is" or "is not". - - expr : Is/IsNot - The expression being printed. - - Returns - ------- - str - The code describing the expression. - - Raises - ------ - X2pyError : Raised if the comparison is poorly defined. - """ - if expr.args[1] is Py_None: - lhs = ObjectAddress(expr.args[0]) - rhs = ObjectAddress(expr.args[1]) - lhs = self._visit(lhs) - rhs = self._visit(rhs) - return f"{lhs} {Op} {rhs}" - python_object_types = (PythonObjectType, PythonClassType, WrapperCustomDataType, NumpyArrayObjectType) - if all(isinstance(arg.dtype, python_object_types) for arg in expr.args): - lhs = self._visit(ObjectAddress(expr.args[0])) - rhs = self._visit(ObjectAddress(expr.args[1])) - return f"(PyObject *){lhs} {Op} (PyObject *){rhs}" - return super()._handle_is_operator(Op, expr) - - # -------------------------------------------------------------------- - # _visit_ClassName functions - # -------------------------------------------------------------------- diff --git a/x2py/codegen/printers/fcode.py b/x2py/codegen/printers/fcode.py deleted file mode 100644 index 497d14d17..000000000 --- a/x2py/codegen/printers/fcode.py +++ /dev/null @@ -1,1870 +0,0 @@ -"""Print to F90 standard. Trying to follow the information provided at -www.fortran90.org as much as possible.""" - -import re -import string -from collections import OrderedDict -from itertools import chain -from typing import ClassVar - - -from ..bind_c import ( - BindCNativeArrayHandleProperty, - BindCFunctionDef, - BindCModule, - BindCModuleConstant, - BindCPointer, - BindCVariable, - FortranTransfer, -) - -from ..models.datatypes import cast_to, is_model_object -from ..models.core import ( - AliasAssign, - Assign, - Declare, - FunctionAddress, - FunctionCall, - FunctionCallArgument, - FunctionDef, - get_direct_assignment, - get_direct_function_argument, - Module, - SeparatorComment, - Slice, -) -from ..models.datatypes import ( - CustomDataType, - FixedSizeNumericType, - FixedSizeType, - PrimitiveBooleanType, - PrimitiveCharacterType, - PrimitiveComplexType, - PrimitiveFloatingPointType, - PrimitiveIntegerType, - Type, - StringType, - SymbolicType, - TupleType, -) -from ..models.datatypes import ( - Literal, - NIL, - convert_to_literal, -) - -from ..models.datatypes import ( - NumpyInt64Type, - NumpyNDArrayType, -) -from ..models.core import ( - Add, - Minus, -) - -from ..models.core import Variable -from .codeprinter import CodePrinter -from x2py.semantics.ownership import CodegenAction, ownership_decision_for_codegen_variable - -# TODO: add examples - -__all__ = ["FCodePrinter"] - - -_FORTRAN_ACCESS_BY_CODEGEN_ACTION = { - CodegenAction.DIRECT_VALUE: "read", - CodegenAction.CALL_LOCAL_INPUT: "read", - CodegenAction.IN_PLACE_ARGUMENT: "readwrite", - CodegenAction.IDENTITY_OUTPUT: "write", - CodegenAction.COPY_IN_OUT: "readwrite", - CodegenAction.COPY_OUT: "write", - CodegenAction.SNAPSHOT_COPY: "read", - CodegenAction.BORROWED_VIEW: "read", - CodegenAction.WRAPPER_INSTANCE: "write", -} - - -# ============================================================================== -iso_c_binding = { - PrimitiveIntegerType(): { - 1: "C_INT8_T", - 2: "C_INT16_T", - 4: "C_INT32_T", - 8: "C_INT64_T", - 16: "C_INT128_T", - }, # not supported yet - PrimitiveFloatingPointType(): { - 4: "C_FLOAT", - 8: "C_DOUBLE", - 16: "C_LONG_DOUBLE", - }, # not supported yet - PrimitiveComplexType(): { - 4: "C_FLOAT_COMPLEX", - 8: "C_DOUBLE_COMPLEX", - 16: "C_LONG_DOUBLE_COMPLEX", - }, # not supported yet - PrimitiveBooleanType(): {-1: "C_BOOL"}, - PrimitiveCharacterType(): {-1: "C_CHAR"}, -} - -iso_c_binding_shortcut_mapping = { - "C_INT8_T": "i8", - "C_INT16_T": "i16", - "C_INT32_T": "i32", - "C_INT64_T": "i64", - "C_INT128_T": "i128", - "C_FLOAT": "f32", - "C_DOUBLE": "f64", - "C_LONG_DOUBLE": "f128", - "C_FLOAT_COMPLEX": "c32", - "C_DOUBLE_COMPLEX": "c64", - "C_LONG_DOUBLE_COMPLEX": "c128", - "C_BOOL": "x2py_b1", -} - -inc_keyword = ( - r"do\b", - r"if \(.*?\) then$", - r"else\b", - r"type\b\s*[^\(]", - r"(elemental )?(pure )?(recursive )?((subroutine)|(function))\b", - r"interface\b", - r"module\b(?! *procedure)", - r"program\b", -) -inc_regex = re.compile("|".join(f"({i})" for i in inc_keyword)) - -end_keyword = ( - "do", - "if", - "type", - "function", - "subroutine", - "interface", - "module", - "program", -) -end_regex_str = "(end ?({}))|(else)".format("|".join(f"({k})" for k in end_keyword)) -dec_regex = re.compile(end_regex_str) - - -class FCodePrinter(CodePrinter): - """ - A printer for printing code in Fortran. - - A printer to convert X2py's AST to strings of Fortran code. - As for all printers the navigation of this file is done via _visit_X - functions. - - Parameters - ---------- - filename : str - The name of the file being converted. - verbose : int - The level of verbosity. - prefix_module : str - A prefix to be added to the name of the module. - """ - - printmethod = "_fcode" - language = "Fortran" - - _default_settings: ClassVar = { - "tabwidth": 2, - } - - # ------------------------------------------------------------------ - # Public entrypoints and state - # ------------------------------------------------------------------ - - def __init__(self, filename, *, verbose, prefix_module=None): - """Initialize the state used for one generation run.""" - super().__init__(verbose) - self._constantImports = [] - - self._additional_code = "" - - self.prefix_module = prefix_module - - # ------------------------------------------------------------------ - # Model visitors - # ------------------------------------------------------------------ - - def _visit_Symbol(self, expr): - """Render the ``Symbol`` model node.""" - return expr - - def _visit_Module(self, expr): - """Render the ``Module`` model node.""" - self.set_scope(expr.scope) - self._constantImports.append({}) - name = self._fortran_module_name(expr.name) - - imports = "".join(self._visit(i) for i in expr.imports) - - # Define declarations - decs, class_decs_and_methods = self._module_declarations(expr) - funcs_to_visit = self._module_functions(expr) - - # ... - public_decs = self._module_public_declarations(expr, funcs_to_visit) - - # ... - sep = self._visit(SeparatorComment(40)) - interfaces, interface_public_decs = self._module_interfaces(expr) - public_decs += interface_public_decs - - body = self._module_body(expr, class_decs_and_methods, funcs_to_visit, sep) - # ... - - has_routines = bool(funcs_to_visit or expr.classes or expr.overload_sets) - private = "" if isinstance(expr, BindCModule) else "private\n" if has_routines else "" - contains = "contains\n" if has_routines else "" - imports += "".join(self._visit(i) for i in self._additional_imports.values()) - imports = self._constant_imports() + imports - implicit_none = "" if expr.is_external else "implicit none\n" - - parts = [ - f"module {name}\n", - imports, - implicit_none, - public_decs, - private, - decs, - interfaces, - contains, - body, - f"end module {name}\n", - ] - - self.exit_scope() - self._constantImports.pop() - - return "\n".join([a for a in parts if a]) - - def _fortran_module_name(self, name): - """Return the emitted Fortran module name.""" - name = self._visit(name).replace(".", "_") - if not name.startswith("mod_") and self.prefix_module: - return f"{self.prefix_module}_{name}" - return name - - def _module_declarations(self, module): - """Render module declarations and collect class method bodies.""" - class_parts = [self._visit(class_def) for class_def in module.classes] - declarations = [ - declaration - for declaration in module.declarations - if not isinstance(declaration.variable, BindCModuleConstant) - ] - self._get_external_declarations(declarations) - code = "\n".join(part[0] for part in class_parts) - code += "".join(self._visit(declaration) for declaration in declarations) - return code, class_parts - - @staticmethod - def _module_functions(module): - """Collect non-header procedures emitted in a module body.""" - candidates = [ - *module.funcs, - *(function for interface in module.overload_sets for function in interface.functions), - ] - return [function for function in candidates if not function.is_header] - - @staticmethod - def _module_public_declarations(module, functions): - """Render public declarations for module-visible symbols.""" - if isinstance(module, BindCModule): - return "private :: c_malloc\n" - names = chain( - (class_def.name for class_def in module.classes), - (function.name for function in functions if not function.is_private and function.is_semantic), - ( - variable.name - for variable in module.variables - if not variable.is_private and not isinstance(variable, BindCModuleConstant) - ), - ) - return "".join(f"public :: {name}\n" for name in names) - - def _module_interfaces(self, module): - """Render module interfaces and their public declarations.""" - if isinstance(module, BindCModule): - external_interfaces = self._bind_c_external_interfaces(module) - code = ( - "interface\n" - 'function c_malloc(size) bind(C,name="x2py_malloc") result(ptr)\n' - "use iso_c_binding\n" - "integer(c_size_t), value :: size\n" - "type(c_ptr) :: ptr\n" - "end function c_malloc\n" - f"{external_interfaces}" - "end interface\n" - ) - return code, "" - code = "\n".join(self._visit(interface) for interface in module.overload_sets) - public = "".join( - f"public :: {interface.name}\n" - for interface in module.overload_sets - if interface.is_semantic and not interface.is_private - ) - return code, public - - def _module_body(self, module, class_parts, functions, separator): - """Render class, procedure, and variable-wrapper bodies.""" - blocks = [part[1] for part in class_parts] - blocks.extend("".join((separator, self._visit(function), separator)) for function in functions) - if isinstance(module, BindCModule): - blocks.extend("".join((separator, self._visit(wrapper), separator)) for wrapper in module.variable_wrappers) - return "\n".join(blocks) - - def _visit_BindCNativeArrayHandleVariable(self, expr): - """Render generated native-array-handle operation functions.""" - return "\n".join(self._visit(function) for _name, function in expr.operation_function_items) - - def _visit_Import(self, expr): - """Render the ``Import`` model node.""" - source = "" - if expr.ignore: - return "" - - source = expr.source - if isinstance(source, Literal) and isinstance(source.dtype, StringType): - source = source.python_value - else: - source = self._visit(source) - - if source.endswith(".inc"): - return f"#include <{source}>\n" - - if expr.source_module: - source = expr.source_module.name - - if str(getattr(expr.source, "name", expr.source)) == "mpi4py": - return "use mpi\n" + "use mpiext\n" - - targets = [t for t in expr.target if not isinstance(t.object, Module)] - - if len(targets) == 0: - if isinstance(expr.source_module, FunctionDef) and expr.source_module.is_external: - if expr.source_module.results: - out_args = list(expr.source_module.scope.collect_all_tuple_elements(expr.source_module.results.var)) - return self._visit(Declare(out_args[0].clone(source), external=True)) - return f"external :: {source}\n" - - return f"use {source}\n" - - targets = [t for t in targets if not getattr(t.object, "is_inline", False)] - if len(targets) == 0: - return "" - - prefix = f"use {source}, only:" - - code = "" - for i in targets: - old_name = i.name - new_name = i.local_alias - if old_name != new_name: - target = f"{new_name} => {old_name}" - line = f"{prefix} {target}" - elif isinstance(new_name, str): - line = f"{prefix} {new_name}" - - else: - raise TypeError(f"Expecting str, Symbol or AsName, given {type(i)}") - - code = (code + "\n" + line) if code else line - - # in some cases, the source is given as a string (when using metavar) - code = code.replace("'", "") - return code + "\n" - - def _visit_Comment(self, expr): - """Render the ``Comment`` model node.""" - comments = self._visit(expr.text) - return "!" + comments + "\n" - - def _visit_EmptyNode(self, expr): - """Render the ``EmptyNode`` model node.""" - return "" - - def _visit_Variable(self, expr): - """Render the ``Variable`` model node.""" - return self._visit(expr.name) - - def _visit_FunctionDefArgument(self, expr): - """Render the ``FunctionDefArgument`` model node.""" - var = expr.var - return ", ".join(self._visit(v) for v in self.scope.collect_all_tuple_elements(var)) - - def _visit_FunctionCallArgument(self, expr): - """Render the ``FunctionCallArgument`` model node.""" - if expr.keyword and expr.keyword != "*args": - keyword = expr.keyword.lstrip("*") - return f"{keyword} = {self._visit(expr.value)}" - return self._visit(expr.value) - - def _visit_DottedVariable(self, expr): - """Render the ``DottedVariable`` model node.""" - if isinstance(expr.lhs, FunctionCall): - base = expr.lhs.funcdef.results.var - var_name = self.scope.get_new_name() - var = base.clone(var_name) - - self.scope.insert_variable(var) - - self._additional_code += self._visit(Assign(var, expr.lhs)) + "\n" - return self._visit(var) + "%" + self._visit(expr.name) - return self._visit(expr.lhs) + "%" + self._visit(expr.name) - - def _visit_Cast(self, expr): - """Render the ``Cast`` model node.""" - value = self._visit(expr.arg) - dtype = expr.dtype - - if isinstance(dtype, StringType): - return value - primitive_type = dtype.primitive_type - if isinstance(primitive_type, PrimitiveBooleanType): - return value if isinstance(expr.arg.dtype.primitive_type, PrimitiveBooleanType) else f"({value} /= 0)" - - kind = self._kind(dtype) - if isinstance(primitive_type, PrimitiveIntegerType): - return f"int({value}, kind={kind})" - if isinstance(primitive_type, PrimitiveFloatingPointType): - return f"real({value}, kind={kind})" - if isinstance(primitive_type, PrimitiveComplexType): - return f"cmplx({value}, kind={kind})" - raise TypeError(f"Unsupported Fortran cast datatype {dtype}") - - # ======================================================================= # - def _visit_ArraySize(self, expr): - """Render the ``ArraySize`` model node.""" - init_value = self._visit(expr.arg) - prec = self._kind(expr) - if isinstance(expr.arg.class_type, StringType): - return f"len({init_value}, kind={prec})" - return f"size({init_value}, kind={prec})" - - def _visit_ArrayShapeElement(self, expr): - """Render the ``ArrayShapeElement`` model node.""" - arg = expr.arg - arg_code = self._visit(arg) - prec = self._kind(expr) - - if isinstance(arg.class_type, NumpyNDArrayType): - if arg.rank == 1: - return f"size({arg_code}, kind={prec})" - - if arg.order == "C": - index = Minus(convert_to_literal(arg.rank), expr.index) - index = self._visit(index) - else: - index = Add(expr.index, convert_to_literal(1)) - index = self._visit(index) - - return f"size({arg_code}, {index}, {prec})" - - if isinstance(arg.class_type, StringType): - return f"len({arg_code})" - raise NotImplementedError(f"Don't know how to represent shape of object of type {arg.class_type}") - - def _visit_ArrayLowerBound(self, expr): - """Render the ``ArrayLowerBound`` model node.""" - arg = expr.arg - if not isinstance(arg.class_type, NumpyNDArrayType): - raise NotImplementedError(f"Don't know how to represent lower bound of object of type {arg.class_type}") - index = ( - Minus(convert_to_literal(arg.rank), expr.index) - if arg.order == "C" - else Add(expr.index, convert_to_literal(1)) - ) - return f"lbound({self._visit(arg)}, {self._visit(index)}, kind={self._kind(expr)})" - - def _visit_ArrayAllocated(self, expr): - """Render the ``ArrayAllocated`` model node.""" - return f"allocated({self._visit(expr.arg)})" - - def _visit_ArrayAssociated(self, expr): - """Render the ``ArrayAssociated`` model node.""" - return f"associated({self._visit(expr.arg)})" - - def _visit_ArrayContiguous(self, expr): - """Render the ``ArrayContiguous`` model node.""" - return f"is_contiguous({self._visit(expr.arg)})" - - def _visit_Declare(self, expr): - # ... ignored declarations - """Render the ``Declare`` model node.""" - var = expr.variable - expr_type = var.class_type - if isinstance(expr_type, SymbolicType): - return "" - - # meta-variables - if isinstance(expr.variable, Variable) and expr.variable.name.startswith("__"): - return "" - # ... - - dtype = var.dtype - rank = var.rank - shape = var.alloc_shape - is_optional = var.is_optional - is_private = var.is_private - is_alias = var.is_alias and not isinstance(dtype, BindCPointer) - on_heap = var.on_heap - on_stack = var.on_stack - is_static = expr.static - is_external = expr.external - is_target = var.is_target and not var.is_alias - by_value = expr.by_value - accepts_assumed_length = var.is_argument and on_stack and not getattr(var, "projected_output", False) - deferred_string = ( - isinstance(dtype, StringType) and not accepts_assumed_length and (not shape or shape[0] is None) - ) or self._is_deferred_character_array(var) - # ... - - dtype_str, rankstr = self._fortran_declaration_type( - var, - expr_type, - dtype, - rank, - shape, - is_alias=is_alias, - on_heap=on_heap, - on_stack=on_stack, - is_static=is_static, - accepts_assumed_length=accepts_assumed_length, - ) - - code_value = "" - if expr.value: - code_value = f" = {self._visit(expr.value)}" - - vstr = self._visit(expr.variable.name) - - # Default empty strings - intentstr = self._fortran_intent_attribute(expr.access, by_value) - valuestr = self._fortran_value_attribute(by_value, rank, is_optional, expr_type) - allocatablestr = self._fortran_allocation_attributes( - is_static, - is_alias, - on_heap, - expr_type, - deferred_string, - is_target, - ) - optionalstr = "" - privatestr = "" - externalstr = "" - - # Compute optional string - if is_optional: - optionalstr = ", optional" - - # Compute private string - if is_private: - privatestr = ", private" - - # Compute external string - if is_external: - externalstr = ", external" - - mod_str = "" - if expr.module_variable and not is_private and isinstance(expr.variable.class_type, FixedSizeNumericType): - mod_str = ", bind(c)" - - # Construct declaration - left = dtype_str + allocatablestr + optionalstr + intentstr + privatestr + externalstr + mod_str + valuestr - right = vstr + rankstr + code_value - return f"{left} :: {right}\n" - - def _fortran_declaration_type( - self, - var, - expr_type, - dtype, - rank, - shape, - *, - is_alias, - on_heap, - on_stack, - is_static, - accepts_assumed_length, - ): - """Render a variable datatype and rank declaration.""" - if isinstance(expr_type, CustomDataType): - return self._custom_declaration_type(var, expr_type), "" - if isinstance(dtype, BindCPointer): - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("c_ptr") - return "type(c_ptr)", "" - if isinstance(dtype, FixedSizeType) and isinstance(expr_type, NumpyNDArrayType | FixedSizeType): - if self._is_character_array(var): - type_code = self._fortran_character_array_type(var, dtype, accepts_assumed_length) - else: - type_code = self._visit(dtype.primitive_type) - if isinstance(dtype, FixedSizeNumericType): - type_code += f"({self._kind(var)})" - rank_code = self._fortran_rank_code( - var, - rank, - shape, - is_alias=is_alias, - on_heap=on_heap, - on_stack=on_stack, - is_static=is_static, - accepts_assumed_length=accepts_assumed_length, - ) - return type_code, rank_code - if isinstance(dtype, StringType): - return self._fortran_string_type(dtype, shape, accepts_assumed_length), "" - raise TypeError(f"Don't know how to print type {expr_type} in Fortran") - - def _custom_declaration_type(self, var, expr_type): - """Render a derived-type declaration, including bound receivers.""" - signature = "type" - if var.is_argument: - argument = get_direct_function_argument(var) - assert argument is not None - if argument.bound_argument: - signature = "class" - return f"{signature}({self._visit(expr_type)})" - - def _fortran_rank_code( - self, - var, - rank, - shape, - *, - is_alias, - on_heap, - on_stack, - is_static, - accepts_assumed_length, - ): - """Render Fortran bounds for an array declaration.""" - if rank == 0: - return "" - start = self._visit(convert_to_literal(0)) - if is_alias or on_heap: - dimensions = [":"] * rank - elif accepts_assumed_length: - dimensions = [f"{start}:"] * rank - elif is_static or on_stack: - ordered_shape = shape[::-1] if var.order == "C" else shape - upper_bounds = [Minus(item, convert_to_literal(1)) for item in ordered_shape] - dimensions = [f"{start}:{self._visit(bound)}" for bound in upper_bounds] - else: - raise NotImplementedError("Fortran rank string undetermined") - return f"({', '.join(dimensions)})" - - def _fortran_string_type(self, dtype, shape, accepts_assumed_length): - """Render a Fortran character type and length contract.""" - type_code = self._visit(dtype) - if shape and shape[0] is not None: - return f"{type_code}(len = {self._visit(shape[0])})" - if accepts_assumed_length: - return f"{type_code}(len = *)" - return f"{type_code}(len = :)" - - @staticmethod - def _is_character_array(var): - """Return whether ``var`` stores fixed-width Fortran character elements.""" - return isinstance(var.class_type, NumpyNDArrayType) and isinstance( - var.dtype.primitive_type, PrimitiveCharacterType - ) - - def _is_deferred_character_array(self, var): - """Return whether ``var`` needs an allocatable deferred character length.""" - return self._is_character_array(var) and var.fortran_character_length == ":" - - def _fortran_character_array_type(self, var, dtype, accepts_assumed_length): - """Render a Fortran character array element type and length contract.""" - type_code = self._visit(dtype.primitive_type) - length = var.fortran_character_length - if length == ":": - return f"{type_code}(len = :)" - if length is None: - return f"{type_code}(len = *)" if accepts_assumed_length else type_code - length_code = self._visit(length) if is_model_object(length) else self._visit(convert_to_literal(length)) - return f"{type_code}(len = {length_code})" - - @staticmethod - def _fortran_value_attribute(by_value, rank, is_optional, expr_type): - """Render the Fortran value ABI attribute for a declaration.""" - if by_value and rank == 0 and not is_optional and not isinstance(expr_type, CustomDataType): - return ", value" - return "" - - @staticmethod - def _fortran_intent_attribute(access, by_value): - """Render a Fortran INTENT attribute from declaration access metadata.""" - if by_value or access in (None, "unspecified"): - return "" - return { - "read": ", intent(in)", - "write": ", intent(out)", - "readwrite": ", intent(inout)", - }[access] - - @staticmethod - def _fortran_allocation_attributes(is_static, is_alias, on_heap, expr_type, deferred_string, is_target): - """Render pointer, allocatable, and target attributes.""" - if is_static: - return "" - if is_alias: - attributes = ", pointer" - elif (on_heap and isinstance(expr_type, NumpyNDArrayType | FixedSizeNumericType)) or deferred_string: - attributes = ", allocatable" - else: - attributes = "" - return f"{attributes}, target" if is_target else attributes - - def _visit_AliasAssign(self, expr): - """Render the ``AliasAssign`` model node.""" - code = "" - lhs = expr.lhs - rhs = expr.rhs - - if isinstance(rhs, FunctionCall): - return self._visit(rhs) - - # TODO improve - op = "=>" - shape_code = "" - if isinstance(lhs.class_type, (NumpyNDArrayType)): - shape_code = ", ".join("0:" for i in range(lhs.rank)) - shape_code = f"({shape_code})" - - code += f"{self._visit(expr.lhs)}{shape_code} {op} {self._visit(expr.rhs)}" - - return code + "\n" - - def _visit_CodeBlock(self, expr): - """Render the ``CodeBlock`` model node.""" - body_exprs = expr.body - body_stmts = [] - for b in body_exprs: - line = self._visit(b) - if self._additional_code: - body_stmts.append(self._additional_code) - self._additional_code = "" - body_stmts.append(line) - return "".join(body_stmts) - - def _visit_Assign(self, expr): - """Render the ``Assign`` model node.""" - lhs = expr.lhs - rhs = expr.rhs - - if isinstance(rhs, FunctionCall): - return self._visit(rhs) - - lhs_code = self._visit(lhs) - - # Right-hand side code - rhs_code = self._visit(rhs) - - code = "" - code += f"{lhs_code} = {rhs_code}" - - return code + "\n" - - # ------------------------------------------------------------------------------ - def _visit_Allocate(self, expr): - """Render the ``Allocate`` model node.""" - class_type = expr.variable.class_type - if expr.alloc_type == "function" and isinstance(class_type, NumpyNDArrayType | CustomDataType): - if expr.status == "unallocated": - return "" - if expr.status == "unknown": - var_code = self._visit(expr.variable) - return f"if (allocated({var_code})) then\n deallocate({var_code})\nend if\n" - - if expr.status == "allocated": - var_code = self._visit(expr.variable) - return f"deallocate({var_code})\n" - - if isinstance(class_type, NumpyNDArrayType | CustomDataType): - # Transpose indices because of Fortran column-major ordering - shape = () if expr.variable.rank == 0 else expr.shape if expr.order == "F" else expr.shape[::-1] - - var_code = self._visit(expr.variable) - size_code = ", ".join(self._visit(i) for i in shape) - shape_code = ", ".join("0:" + self._visit(Minus(i, convert_to_literal(1))) for i in shape) - if shape: - shape_code = f"({shape_code})" - type_spec = self._allocate_type_spec(expr.variable) - code = "" - - if expr.status == "unallocated": - code += f"allocate({type_spec}{var_code}{shape_code})\n" - - elif expr.status == "unknown": - code += f"if (allocated({var_code})) then\n" - code += f" if (any(size({var_code}) /= [{size_code}])) then\n" - code += f" deallocate({var_code})\n" - code += f" allocate({type_spec}{var_code}{shape_code})\n" - code += " end if\n" - code += "else\n" - code += f" allocate({type_spec}{var_code}{shape_code})\n" - code += "end if\n" - - elif expr.status == "allocated": - code += f"if (any(size({var_code}) /= [{size_code}])) then\n" - code += f" deallocate({var_code})\n" - code += f" allocate({type_spec}{var_code}{shape_code})\n" - code += "end if\n" - - return code - - if isinstance(class_type, NumpyNDArrayType | StringType): - return "" - - return self._visit_not_supported(expr) - - def _allocate_type_spec(self, var): - """Render an allocation type spec for fixed-length character arrays.""" - if not self._is_character_array(var): - return "" - length = var.fortran_character_length - if length in (None, ":"): - return "" - length_code = self._visit(length) if is_model_object(length) else self._visit(convert_to_literal(length)) - return f"character(len = {length_code}) :: " - - # ----------------------------------------------------------------------------- - def _visit_Deallocate(self, expr): - """Render the ``Deallocate`` model node.""" - var = expr.variable - class_type = var.class_type - - if isinstance(class_type, CustomDataType): - x2py__del = expr.variable.cls_base.scope.find("__del__") - if x2py__del: - x2py_del_args = [FunctionCallArgument(var)] - return self._visit(FunctionCall(x2py__del, x2py_del_args)) - return "" - - if var.is_alias: - return "" - if isinstance(class_type, NumpyNDArrayType | StringType): - var_code = self._visit(var) - return f"if (allocated({var_code})) deallocate({var_code})\n" - raise NotImplementedError(f"Deallocate not implemented for {class_type}") - - def _visit_DeallocatePointer(self, expr): - """Render the ``DeallocatePointer`` model node.""" - var_code = self._visit(expr.variable) - return f"deallocate({var_code})\n" - - def _visit_Nullify(self, expr): - """Render the ``Nullify`` model node.""" - return f"nullify({self._visit(expr.variable)})\n" - - # ------------------------------------------------------------------------------ - - def _visit_PrimitiveBooleanType(self, expr): - """Render the ``PrimitiveBooleanType`` model node.""" - return "logical" - - def _visit_PrimitiveIntegerType(self, expr): - """Render the ``PrimitiveIntegerType`` model node.""" - return "integer" - - def _visit_PrimitiveFloatingPointType(self, expr): - """Render the ``PrimitiveFloatingPointType`` model node.""" - return "real" - - def _visit_PrimitiveComplexType(self, expr): - """Render the ``PrimitiveComplexType`` model node.""" - return "complex" - - def _visit_PrimitiveCharacterType(self, expr): - """Render the ``PrimitiveCharacterType`` model node.""" - return "character" - - def _visit_StringType(self, expr): - """Render the ``StringType`` model node.""" - return "character" - - def _visit_FortranCharacterLength(self, expr): - """Render the Fortran element length intrinsic.""" - return f"len({self._visit(expr.arg)})" - - def _visit_CustomDataType(self, expr): - """Render the ``CustomDataType`` model node.""" - while hasattr(expr, "underlying_type"): - expr = expr.underlying_type - try: - name = self.scope.get_import_alias(expr, "cls_constructs") - except RuntimeError: - name = expr.low_level_name - return name - - def _visit_FunctionAddress(self, expr): - """Render the ``FunctionAddress`` model node.""" - return expr.name - - def _visit_FunctionDef(self, expr): - """Render the ``FunctionDef`` model node.""" - if not expr.is_semantic: - return "" - self.set_scope(expr.scope) - - self._validate_fortran_function_results(expr) - - name = expr.cls_name or expr.name - - sig_parts = self._function_signature(expr, name) - bind_c = " bind(c)" if isinstance(expr, BindCFunctionDef) else "" - prelude = sig_parts.pop("arg_decs") - functions = [f for f in expr.functions if f.is_semantic] - func_interfaces = "\n".join(self._visit(i) for i in expr.overload_sets) - body_code = self._visit(expr.body) - docstring = self._visit(expr.docstring) if expr.docstring else "" - - decs = [Declare(v) for v in expr.local_vars if not v.is_argument] - self._get_external_declarations(decs) - - prelude += "".join(self._visit(i) for i in decs) - body_code = self._function_body_with_nested(body_code, functions) - imports, external_imports = self._split_function_imports(expr.imports) - - parts = [ - docstring, - f"{sig_parts['sig']}({sig_parts['arg_code']}){bind_c} {sig_parts['func_end']}\n", - imports, - "implicit none\n", - external_imports, - prelude, - func_interfaces, - body_code, - "end {} {}\n".format(sig_parts["func_type"], name), - ] - - self.exit_scope() - - return "\n".join(a for a in parts if a) - - @staticmethod - def _validate_fortran_function_results(function) -> None: - """Reject unknown-size stack arrays returned from Fortran.""" - if function.decorators.get("x2py_callback_adapter"): - return - for result in function.scope.collect_all_tuple_elements(function.results.var): - unknown_stack_array = ( - result.rank - and result.memory_handling == "stack" - and any(not isinstance(shape, Literal) for shape in result.alloc_shape) - ) - if unknown_stack_array: - raise ValueError("Can't return a stack array of unknown size") - - def _function_body_with_nested(self, body_code, functions): - """Append nested procedures to a function body.""" - if not functions: - return body_code - functions_code = "\n".join(self._visit(function) for function in functions) - return body_code + "\ncontains\n" + functions_code - - def _split_function_imports(self, imports): - """Render regular and external procedure imports separately.""" - external = [ - item for item in imports if isinstance(item.source_module, FunctionDef) and item.source_module.is_external - ] - regular = [item for item in imports if item not in external] - return "".join(self._visit(item) for item in regular), "".join(self._visit(item) for item in external) - - def _visit_Return(self, expr): - """Render the ``Return`` model node.""" - code = "" - if expr.stmt: - code += self._visit(expr.stmt) - code += "return\n" - return code - - def _visit_IsNot(self, expr): - """Render the ``IsNot`` model node.""" - lhs, rhs = expr.args - if rhs is NIL: - return self._handle_not_none(self._visit(lhs), lhs) - if lhs is NIL: - return self._handle_not_none(self._visit(rhs), rhs) - raise NotImplementedError(f"Fortran is-not printing is not implemented for {expr}") - - def _visit_If(self, expr): - # ... - - """Render the ``If`` model node.""" - lines = [] - - for i, (c, e) in enumerate(expr.blocks): - if i == len(expr.blocks) - 1 and isinstance(c, Literal) and c.python_value is True: - lines.append("else\n") - elif i == 0: - lines.append(f"if ({self._visit(c)}) then\n") - else: - lines.append(f"else if ({self._visit(c)}) then\n") - - if isinstance(e, list | tuple): - lines.extend(self._visit(ee) for ee in e) - else: - lines.append(self._visit(e)) - - if len(lines) == 0: - return "" - if lines[0] == "else\n": - lines = lines[1:] - else: - lines.append("end if\n") - - return "".join(lines) - - def _visit_SelectCase(self, expr): - """Render the ``SelectCase`` model node.""" - lines = [f"select case ({self._visit(expr.expr)})\n"] - for section in expr.sections: - if section.label is None: - lines.append("case default\n") - else: - lines.append(f"case ({self._visit(section.label)})\n") - lines.append(self._visit(section.body)) - lines.append("end select\n") - return "".join(lines) - - def _visit_Add(self, expr): - """Render the ``Add`` model node.""" - if isinstance(expr.dtype, StringType): - return " // ".join(self._visit(a) for a in expr.args) - args = [ - (cast_to(a, NumpyInt64Type()) if isinstance(a.dtype.primitive_type, PrimitiveBooleanType) else a) - for a in expr.args - ] - return " + ".join(self._visit(a) for a in args) - - def _visit_Minus(self, expr): - """Render the ``Minus`` model node.""" - args = [ - (cast_to(a, NumpyInt64Type()) if isinstance(a.dtype.primitive_type, PrimitiveBooleanType) else a) - for a in expr.args - ] - args_code = [self._visit(a) for a in args] - - return " - ".join(args_code) - - def _visit_Mul(self, expr): - """Render the ``Mul`` model node.""" - args = [ - (cast_to(a, NumpyInt64Type()) if isinstance(a.dtype.primitive_type, PrimitiveBooleanType) else a) - for a in expr.args - ] - args_code = [self._visit(a) for a in args] - return " * ".join(a for a in args_code) - - def _visit_Literal(self, expr): - """Render the ``Literal`` model node.""" - value = expr.python_value - dtype = expr.dtype - - if expr is NIL: - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("c_null_ptr") - return "c_null_ptr" - if isinstance(dtype, StringType): - if value == "": - return "''" - special_characters = {"\a", "\b", "\f", "\r", "\t", "\v", "'", "\n"} - substring = "" - parts = [] - for character in value: - if character in special_characters: - if substring: - parts.append(f"'{substring}'") - substring = "" - parts.append(f"ACHAR({ord(character)})") - else: - substring += character - if substring: - parts.append(f"'{substring}'") - return " // ".join(parts) - - primitive_type = dtype.primitive_type - if isinstance(primitive_type, PrimitiveBooleanType): - value_code = ".True." if value else ".False." - return f"{value_code}_{self._kind(expr)}" - if isinstance(primitive_type, PrimitiveComplexType): - real = self._visit(Literal(value.real, dtype.element_type)) - imag = self._visit(Literal(value.imag, dtype.element_type)) - return f"({real}, {imag})" - return f"{value!r}_{self._kind(expr)}" - - def _visit_IndexedElement(self, expr): - """Render the ``IndexedElement`` model node.""" - base = expr.base - if isinstance(base.class_type, TupleType): - return self._visit(self.scope.collect_tuple_element(expr)) - if isinstance(base.class_type, StringType): - if len(expr.indices) != 1 or isinstance(expr.indices[0], Slice): - raise NotImplementedError("Fortran string indexing requires one index") - index = self._visit(expr.indices[0]) - return f"{self._visit(base)}({index}:{index})" - if not isinstance(base.class_type, NumpyNDArrayType): - raise NotImplementedError(f"Fortran indexing is not implemented for {base.class_type}") - - indices = list(expr.indices) - if base.order != "F": - indices.reverse() - - indices = [ - Slice(index.start, Minus(index.stop, convert_to_literal(1)), index.step) - if isinstance(index, Slice) and index.stop is not None and index.stop is not NIL - else index - for index in indices - ] - return f"{self._visit(base)}({', '.join(self._visit(i) for i in indices)})" - - def _visit_Slice(self, expr): - """Render the ``Slice`` model node.""" - start = "" if expr.start is None or expr.start is NIL else self._visit(expr.start) - stop = "" if expr.stop is None or expr.stop is NIL else self._visit(expr.stop) - if expr.step is not None: - return f"{start}:{stop}:{self._visit(expr.step)}" - return f"{start}:{stop}" - - # ======================================================================================= - - def _visit_FunctionCall(self, expr): - """Render the ``FunctionCall`` model node.""" - func = expr.funcdef - - native_name = expr.overload_set.native_name_for(func) if expr.overload_set else "" - if expr.overload_set and self._is_defined_operator(native_name): - return self._defined_operator_call(expr, func, native_name) - - f_name = self._fortran_call_name(expr, func) - - args = expr.args - func_result_variables = ( - func.scope.collect_all_tuple_elements(func.results.var) if func.scope else [func.results.var] - ) - out_results = [v for v in func_result_variables if v and not v.is_argument] - parent_assign = get_direct_assignment(expr) - is_function = self._call_is_fortran_function(func, out_results, parent_assign) - - if func.arguments and func.arguments[0].bound_argument: - f_name, args = self._bound_fortran_call(expr, func, args) - - if parent_assign: - args, results, results_strs = self._assigned_fortran_call_arguments( - args, out_results, parent_assign, is_function - ) - - else: - results_strs = [] - results = None - - args_strs = [self._visit(a) for a in args if a.value is not NIL] - args_code = ", ".join(results_strs + args_strs) - code = f"{f_name}({args_code})" - if not is_function: - code = f"call {code}\n" - - return self._finalize_fortran_call(code, parent_assign, is_function, out_results, results) - - def _defined_operator_call(self, expr, function, native_name): - """Render a defined operator call or its assignment.""" - arguments = expr.overload_set.native_arguments(function, expr.args) - values = [self._visit(argument.value) for argument in arguments] - token = self._defined_operator_token(native_name) - if len(values) == 1: - code = f".not. {values[0]}" if token == ".not." else f"{token}{values[0]}" - else: - code = f"{values[0]} {token} {values[1]}" - parent_assign = get_direct_assignment(expr) - if not parent_assign: - return code - assignment = "=>" if isinstance(parent_assign, AliasAssign) else "=" - return f"{self._visit(parent_assign.lhs)} {assignment} {code}\n" - - def _fortran_call_name(self, expr, function): - """Resolve the emitted name for a Fortran call.""" - name = self._visit(expr.func_name if not expr.overload_set else expr.overload_set_name) - if function.is_imported: - return self.scope.get_import_alias(function, "functions") - if expr.overload_set and expr.overload_set.is_imported: - return self.scope.get_import_alias(expr.overload_set, "functions") - return name - - @staticmethod - def _call_is_fortran_function(function, out_results, parent_assign): - """Return whether a call is emitted as a function expression.""" - is_function = len(out_results) == 1 and ( - function.results.var.rank == 0 or isinstance(function.results.var.class_type, StringType) - ) - if len(out_results) == 1 and isinstance(function.results.var.class_type, NumpyNDArrayType): - return parent_assign is not None or function.results.var.memory_handling in {"alias", "heap"} - return is_function - - def _bound_fortran_call(self, expr, function, arguments): - """Render a type-bound call receiver and remaining arguments.""" - bound_name = ( - expr.overload_set_name - if expr.overload_set - else (function.type_bound_name or function.scope.get_python_name(function.name)) - ) - function_name = self._visit(bound_name) - class_variable = arguments[0].value - remaining_arguments = arguments[1:] - if not isinstance(class_variable, FunctionCall): - return f"{self._visit(class_variable)} % {function_name}", remaining_arguments - base = class_variable.funcdef.results.var - variable = self.scope.get_temporary_variable(base) - self._additional_code += self._visit(Assign(variable, class_variable)) + "\n" - return f"{self._visit(variable)} % {function_name}", remaining_arguments - - def _assigned_fortran_call_arguments(self, arguments, out_results, parent_assign, is_function): - """Prepare call arguments and result targets under assignment.""" - lhs = parent_assign.lhs - lhs_vars = {out_results[0]: lhs} if len(out_results) == 1 else dict(zip(out_results, lhs, strict=False)) - assigned_arguments = [] - for argument in arguments: - value = argument.value - if value in lhs_vars.values(): - replacement = value.clone(self.scope.get_new_name()) - self.scope.insert_variable(replacement) - self._additional_code += self._visit(Assign(replacement, value)) - value = replacement - assigned_arguments.append(FunctionCallArgument(value, argument.keyword)) - results = list(lhs_vars.values()) - result_strings = [] if is_function else [self._visit(result) for result in results] - return assigned_arguments, results, result_strings - - def _finalize_fortran_call(self, code, parent_assign, is_function, out_results, results): - """Render final call or assignment syntax for a Fortran procedure.""" - if not parent_assign: - if is_function or not out_results: - return code - self._additional_code += code - if len(out_results) == 1: - return self._visit(results[0]) - return self._visit(tuple(results)) - if not is_function: - return code - result_code = self._visit(results[0]) - assignment = "=>" if isinstance(parent_assign, AliasAssign) else "=" - return f"{result_code} {assignment} {code}\n" - - def _visit_CLocFunc(self, expr): - """Render the ``CLocFunc`` model node.""" - lhs = self._visit(expr.result) - rhs = self._visit(expr.arg) - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("c_loc") - return f"{lhs} = c_loc({rhs})\n" - - def _visit_C_NULL_CHAR(self, expr): - """Render the ``C_NULL_CHAR`` model node.""" - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("C_NULL_CHAR") - return "C_NULL_CHAR" - - def _visit_C_F_Pointer(self, expr): - """Render the ``C_F_Pointer`` model node.""" - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("C_F_Pointer") - shape_tuple = expr.shape or () - shape = ", ".join(self._visit(s) for s in shape_tuple) - if shape: - return f"call C_F_Pointer({self._visit(expr.c_pointer)}, {self._visit(expr.f_array)}, [{shape}])\n" - return f"call C_F_Pointer({self._visit(expr.c_pointer)}, {self._visit(expr.f_array)})\n" - - # ======================================================================================= - - def _visit_BindCArrayVariable(self, expr): - """Render the ``BindCArrayVariable`` model node.""" - return self._visit(expr.wrapper_function) - - def _visit_BindCClassDef(self, expr): - """Render the ``BindCClassDef`` model node.""" - handle_operations = [ - function - for attribute in expr.attributes - if isinstance(attribute, BindCNativeArrayHandleProperty) - for _name, function in attribute.operation_function_items - ] - funcs = [ - expr.new_func, - *expr.methods, - *[f for i in expr.overload_sets for f in i.functions], - *[a.getter for a in expr.attributes if not isinstance(a, BindCNativeArrayHandleProperty)], - *[a.setter for a in expr.attributes if not isinstance(a, BindCNativeArrayHandleProperty) and a.setter], - *handle_operations, - ] - sep = f"\n{self._visit(SeparatorComment(40))}\n" - return "", sep.join(self._visit(f) for f in funcs) - - def _visit_BindCSizeOf(self, expr): - """Render the ``BindCSizeOf`` model node.""" - elem = self._visit(expr.args[0]) - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("c_size_t") - return f"storage_size({elem}, kind = c_size_t)" - - def _visit_FortranTransfer(self, expr: FortranTransfer): - """Render the ``FortranTransfer`` model node.""" - source = self._visit(expr.source) - mold = self._visit(expr.mold) - if expr.size is None: - return f"transfer({source}, {mold})" - size = self._visit(expr.size) - return f"transfer({source}, {mold}, {size})" - - # ------------------------------------------------------------------ - # Shared helpers - # ------------------------------------------------------------------ - - def _constant_imports(self): - """ - Print the import of constant intrinsics. - - Print the import of constants such as `C_INT` from an intrinsic module (i.e. a - module provided by Fortran) such as `iso_c_binding`. - - Returns - ------- - str - The code describing the import of the intrinsics. - """ - macros = [] - for name, imports in self._constantImports[-1].items(): - macro = f"use, intrinsic :: {name}, only : " - rename = [c if isinstance(c, str) else c[0] + " => " + c[1] for c in imports] - if len(rename) == 0: - continue - rename.sort() - macro += " , ".join(rename) - macro += "\n" - macros.append(macro) - return "".join(macros) - - def _bind_c_external_interfaces(self, expr): - """Handle explicit external interfaces for the current generation context.""" - original_module = getattr(expr, "original_module", None) - if original_module is None: - return "" - interfaces = [ - self._external_interface(func) - for func in original_module.funcs - if func.is_external and func.is_semantic and not func.is_private - ] - return "".join(interfaces) - - def _external_interface(self, func): - """Emit an explicit interface for one external native procedure.""" - args = ", ".join(self._visit(arg.name) for arg in func.arguments) - result_vars = [var for var in func.scope.collect_all_tuple_elements(func.results.var) if var] - is_function = len(result_vars) == 1 - func_type = "function" if is_function else "subroutine" - lines = [f"{func_type} {self._visit(func.name)}({args})", "import", "implicit none"] - if is_function: - lines.append(self._visit(Declare(result_vars[0].clone(str(func.name)))).rstrip()) - for arg in func.arguments: - lines.append(self._external_interface_argument_declaration(arg.var)) - lines.append(f"end {func_type} {self._visit(func.name)}") - return "\n".join(lines) + "\n" - - def _external_interface_argument_declaration(self, var): - """Declare an external native argument without changing its call ABI.""" - if isinstance(var.class_type, StringType): - return self._visit(Declare(var)).rstrip() - if isinstance(var.class_type, CustomDataType): - type_code = f"type({self._visit(var.class_type)})" - elif isinstance(var.class_type, NumpyNDArrayType | FixedSizeType): - type_code = self._visit(var.dtype.primitive_type) - if isinstance(var.dtype, FixedSizeNumericType): - type_code += f"({self._kind(var)})" - else: - raise TypeError(f"Unsupported external native argument type {var.class_type}") - - attributes = [] - if getattr(var, "is_optional", False): - attributes.append("optional") - attribute_code = f", {', '.join(attributes)}" if attributes else "" - shape_code = "" - if var.rank: - dimensions = self._external_interface_argument_dimensions(var) - shape_code = f"({', '.join(dimensions)})" - return f"{type_code}{attribute_code} :: {var.name}{shape_code}" - - def _external_interface_argument_dimensions(self, var): - """Return dimensions for an external interface without changing native ABI.""" - source_shape = tuple(getattr(var, "fortran_source_shape", ()) or ()) - if source_shape: - if var.rank > 1 and str(source_shape[0]).strip() == "*": - return ["*"] - dimensions = [] - for index, item in enumerate(var.alloc_shape): - source_dim = str(source_shape[index]).strip() if index < len(source_shape) else "" - if source_dim: - dimensions.append(source_dim) - elif item is None: - dimensions.append("*" if index == var.rank - 1 else ":") - else: - dimensions.append(self._visit(item)) - return dimensions - return [":" if item is None else self._visit(item) for item in var.alloc_shape] - - def _format_code(self, lines): - """ - Format code in order to match readable Fortran practices. - - Format code in order to match readable Fortran practices. - In particular this function indents the code. - - Parameters - ---------- - lines : list[str] - The lines of code. - - Returns - ------- - list[str] - The formatted lines of code. - """ - return self._wrap_fortran(self._indent_code(lines)) - - def _kind(self, expr): - """ - Print the kind(precision) of a literal value or its shortcut if possible. - - Print the kind(precision) of a literal value or its shortcut if possible. - - Parameters - ---------- - expr : model object | Type - The object whose precision should be investigated. - - Returns - ------- - str - The code for the kind parameter. - """ - dtype = expr if isinstance(expr, Type) else expr.dtype - - constant_name = iso_c_binding[dtype.primitive_type][dtype.precision] - - constant_shortcut = iso_c_binding_shortcut_mapping[constant_name] - if constant_shortcut not in self.scope.all_used_symbols and constant_name != constant_shortcut: - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add((constant_shortcut, constant_name)) - constant_name = constant_shortcut - else: - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add(constant_name) - return constant_name - - def _get_external_declarations(self, decs): - """ - Find external functions and declare their result type. - - Look for any external functions in the local imports from - the scope and use their definitions to create declarations - from the results. These declarations are stored in the list - passed as argument. - - Parameters - ---------- - decs : list - The list where the declarations necessary to use the external - functions will be stored. - """ - for key, f in self.scope.imports["functions"].items(): - if isinstance(f, FunctionDef) and f.is_external and f.results.var: - v = f.results.var.clone(str(key)) - decs.append(Declare(v, external=True)) - - def _function_signature(self, expr, name): - """ - Get the different parts of the signature of the function `expr`. - - A helper function to print just the signature of the function - including the declarations of the arguments and results. - - Parameters - ---------- - expr : FunctionDef - The function whose signature should be printed. - name : str - The name which should be printed as the name of the function. - (May be different from expr.name in the case of interfaces). - - Returns - ------- - dict - A dictionary with the keys : - sig - The declaration of the function/subroutine with any necessary keywords. - arg_code - A string containing a list of the arguments. - func_end - Any code to be added to the signature after the arguments (ie result). - arg_decs - The code necessary to declare the arguments of the function/subroutine. - func_type - Subroutine or function. - """ - out_args = [v for v in expr.scope.collect_all_tuple_elements(expr.results.var) if v and not v.is_argument] - arguments = expr.arguments - class_arg = next((a for a in arguments if a.bound_argument), None) - callback_adapter = bool(expr.decorators.get("x2py_callback_adapter")) - out_args, args_decs, func_type, func_end, callback_result = self._fortran_result_signature( - expr, out_args, callback_adapter - ) - callback_interfaces = self._fortran_argument_declarations(expr, arguments, callback_adapter, args_decs) - if callback_result is not None: - result, declaration = callback_result - args_decs[result] = declaration - - sig = self._fortran_signature_prefix(expr, func_type, name) - - arg_iter = chain((class_arg,), out_args, arguments[1:]) if class_arg else chain(out_args, arguments) - arg_code = ", ".join(self._visit(i) for i in arg_iter) - - arg_decs = "".join(self._visit(i) if isinstance(i, Declare) else i for i in args_decs.values()) - arg_decs = "".join(callback_interfaces) + arg_decs - - return { - "sig": sig, - "arg_code": arg_code, - "func_end": func_end, - "arg_decs": arg_decs, - "func_type": func_type, - } - - @staticmethod - def _fortran_result_signature(function, out_args, callback_adapter): - """Classify results and build their signature declarations.""" - declarations = OrderedDict() - string_result = isinstance(function.results.var.class_type, StringType) - uses_output_arguments = len(out_args) != 1 or ( - function.results.var.rank > 0 and not string_result and not callback_adapter - ) - if uses_output_arguments: - for result in out_args: - declarations[result] = Declare(result, access="write") - return out_args, declarations, "subroutine", "", None - result = out_args[0] - declaration = Declare(result) - callback_result = (result, declaration) if callback_adapter else None - if callback_result is None: - declarations[result] = declaration - return [], declarations, "function", f"result({result.name})", callback_result - - def _fortran_argument_declarations(self, function, arguments, callback_adapter, declarations): - """Populate argument declarations and callback interfaces.""" - callback_interfaces = [] - bind_c_function = isinstance(function, BindCFunctionDef) - callback_c_abi_function = bool(function.decorators.get("x2py_callback_abi")) - for argument in arguments: - variable = argument.var - if isinstance(variable, Variable): - if callback_adapter: - declarations[variable] = self._callback_native_argument_declaration(variable) - continue - is_c_abi_argument = isinstance(variable, BindCVariable) or callback_c_abi_function - for element in self.scope.collect_all_tuple_elements(variable): - by_value = self._fortran_argument_passes_by_value( - element, - bind_c_function=bind_c_function, - c_abi_argument=is_c_abi_argument, - ) - declarations[element] = Declare( - element, - access=self._fortran_argument_access(element) if bind_c_function else None, - by_value=by_value, - ) - elif isinstance(variable, FunctionAddress) and variable.decorators.get("x2py_callback_abi"): - callback_interfaces.append(self._callback_c_interface(variable)) - return callback_interfaces - - @staticmethod - def _fortran_argument_passes_by_value(var, *, bind_c_function, c_abi_argument): - """Return whether a Fortran declaration needs the C ``value`` ABI attribute.""" - if var.rank != 0 or var.is_optional or isinstance(var.class_type, CustomDataType): - return False - if c_abi_argument or getattr(var, "passes_by_value", False): - return True - return ( - bind_c_function - and var.memory_handling == "stack" - and isinstance( - var.class_type, - FixedSizeType | BindCPointer, - ) - ) - - @staticmethod - def _fortran_argument_access(var): - """Derive bridge declaration access from completed ownership policy.""" - try: - action = ownership_decision_for_codegen_variable(var).codegen_action - except ValueError: - return None - return _FORTRAN_ACCESS_BY_CODEGEN_ACTION.get(action) - - @staticmethod - def _fortran_callback_intent_attribute(access, by_value): - """Render callback adapter INTENT without hiding value input intent.""" - if by_value: - if access in {"write", "readwrite"}: - raise ValueError("Fortran VALUE callback arguments cannot have output or readwrite access") - return ", intent(in)" - if access in (None, "unspecified"): - return "" - return { - "read": ", intent(in)", - "write": ", intent(out)", - "readwrite": ", intent(inout)", - }[access] - - @staticmethod - def _fortran_signature_prefix(function, func_type, name): - """Render recursive, pure, and elemental signature prefixes.""" - signature = f"{'recursive ' if function.is_recursive else ''}{func_type} {name}" - if function.is_pure: - signature = f"pure {signature}" - if function.is_elemental: - signature = f"elemental {signature}" - return signature - - def _callback_native_argument_declaration(self, var): - """Declare an internal callback adapter argument with its native Fortran ABI.""" - if isinstance(var.class_type, StringType): - type_code = self._callback_character_type(var) - elif isinstance(var.class_type, CustomDataType): - type_code = f"type({self._visit(var.class_type)})" - elif isinstance(var.class_type, NumpyNDArrayType | FixedSizeType): - type_code = self._visit(var.dtype.primitive_type) - if isinstance(var.dtype, FixedSizeNumericType): - type_code += f"({self._kind(var)})" - else: - raise TypeError(f"Unsupported native callback argument type {var.class_type}") - - shape_code = "" - if var.rank and not isinstance(var.class_type, StringType): - dimensions = [":" if item is None else self._visit(item) for item in var.alloc_shape] - shape_code = f"({', '.join(dimensions)})" - access = getattr(var, "fortran_callback_access", None) or "read" - by_value = getattr(var, "passes_by_value", False) - intent_code = self._fortran_callback_intent_attribute(access, by_value=by_value) - value_code = self._fortran_value_attribute(by_value, var.rank, var.is_optional, var.class_type) - return f"{type_code}{intent_code}{value_code} :: {var.name}{shape_code}\n" - - def _callback_character_type(self, var): - """Render the character type for a callback adapter dummy.""" - length = var.fortran_character_length - if length in (None, "*", ":") and var.alloc_shape and var.alloc_shape[0] is not None: - length = var.alloc_shape[0] - if length in (None, "*", ":"): - return "character(len = *)" - length_code = self._visit(length) if is_model_object(length) else self._visit(convert_to_literal(length)) - return f"character(len = {length_code})" - - def _callback_c_interface(self, callback): - """Emit the interoperable interface for a C callback dummy procedure.""" - parts = self._function_signature(callback, callback.name) - signature = f"{parts['sig']}({parts['arg_code']}) bind(c) {parts['func_end']}".rstrip() - return ( - "interface\n" - f"{signature}\n" - "import\n" - f"{parts['arg_decs']}" - f"end {parts['func_type']} {callback.name}\n" - "end interface\n" - ) - - def _handle_not_none(self, lhs, lhs_var): - """ - Print code for `x is not None` statement. - - Print the code which checks if x is not None. This means different - things depending on the type of `x`. If `x` is optional it checks - if it is present, if `x` is a C pointer it checks if it points at - anything. - - Parameters - ---------- - lhs : str - The code representing `x`. - lhs_var : Variable - The Variable `x`. - - Returns - ------- - str - The code which checks if `x is not None`. - """ - if isinstance(lhs_var.dtype, BindCPointer): - self._constantImports[-1].setdefault("ISO_C_Binding", set()).add("c_associated") - return f"c_associated({lhs})" - return f"present({lhs})" - - @staticmethod - def _is_defined_operator(name): - """Return whether is defined operator.""" - return re.fullmatch(r"operator\(.+\)", re.sub(r"\s+", "", str(name)), re.IGNORECASE) is not None - - @staticmethod - def _defined_operator_token(name): - """Handle defined operator token for the current generation context.""" - compact = re.sub(r"\s+", "", str(name)) - return compact[compact.index("(") + 1 : -1] - - # ======================================================================================= - - def _wrap_fortran(self, lines): - """ - Wrap long Fortran lines. - - A comment line is split at white space. Code lines are split with a more - complex rule to give nice results. - - Parameters - ---------- - lines : list[str] - A list of lines (ending with a \\n character). - - Returns - ------- - list[str] - A list of the new lines. - """ - # routine to find split point in a code line - my_alnum = set("_+-." + string.digits + string.ascii_letters) - my_white = set(" \t()") - - def split_pos_code(line, endpos): - if len(line) <= endpos: - return len(line) - pos = endpos - - def split(pos): - return ( - (line[pos] in my_alnum and line[pos - 1] not in my_alnum) - or (line[pos] not in my_alnum and line[pos - 1] in my_alnum) - or (line[pos] in my_white and line[pos - 1] not in my_white) - or (line[pos] not in my_white and line[pos - 1] in my_white) - ) - - while not split(pos): - pos -= 1 - if pos == 0: - return endpos - return pos - - # split line by line and add the split lines to result - result = [] - trailing = " &" - # trailing with no added space characters in case splitting is within quotes - quote_trailing = "&" - - for line in lines: - if len(line) > 72: - cline = line[:72].lstrip() - if cline.startswith("!") and not cline.startswith("!$"): - result.append(line) - continue - - tab_len = line.index(cline[0]) - # code line - # set containing positions inside quotes - inside_quotes_positions = set() - inside_quotes_intervals = [ - (match.start(), match.end()) for match in re.compile("(\"[^\"]*\")|('[^']*')").finditer(line) - ] - for lidx, ridx in inside_quotes_intervals: - for idx in range(lidx, ridx): - inside_quotes_positions.add(idx) - initial_len = len(line) - pos = split_pos_code(line, 72) - - startswith_omp = cline.startswith("!$omp") - startswith_acc = cline.startswith("!$acc") - - if startswith_acc or startswith_omp: - assert pos >= 5 - - if pos not in inside_quotes_positions: - hunk = line[:pos].rstrip() - line = line[pos:].lstrip() - else: - hunk = line[:pos] - line = line[pos:] - - if line: - hunk += quote_trailing if pos in inside_quotes_positions else trailing - - last_cut_was_inside_quotes = pos in inside_quotes_positions - result.append(hunk) - while len(line) > 0: - removed = initial_len - len(line) - pos = split_pos_code(line, 65 - tab_len) - if pos + removed not in inside_quotes_positions: - hunk = line[:pos].rstrip() - line = line[pos:].lstrip() - else: - hunk = line[:pos] - line = line[pos:] - if line: - hunk += quote_trailing if (pos + removed) in inside_quotes_positions else trailing - - if last_cut_was_inside_quotes: - hunk_start = tab_len * " " + "&" - elif startswith_omp: - hunk_start = tab_len * " " + "!$omp &" - elif startswith_acc: - hunk_start = tab_len * " " + "!$acc &" - else: - hunk_start = tab_len * " " + " " - - result.append(hunk_start + hunk) - last_cut_was_inside_quotes = (pos + removed) in inside_quotes_positions - else: - result.append(line) - - # make sure that all lines end with a carriage return - return [line if line.endswith("\n") else line + "\n" for line in result] - - def _indent_code(self, code): - """ - Add the correct indentation to the code. - - Analyse the code to calculate when indentation is needed. - Add the necessary spaces at the start of each line. - - Parameters - ---------- - code : str | iterable[str] - A string of code or a list of code lines. - - Returns - ------- - list[str] - A list of indented code lines. - """ - if isinstance(code, str): - code_lines = self._indent_code(code.splitlines(True)) - return "".join(code_lines) - - code = [line.lstrip(" \t") for line in code] - - increase = [int(inc_regex.match(line) is not None) for line in code] - decrease = [int(dec_regex.match(line) is not None) for line in code] - - level = 0 - tabwidth = self._default_settings["tabwidth"] - new_code = [] - for i, line in enumerate(code): - if line in ("", "\n") or line.startswith("#"): - new_code.append(line) - continue - level -= decrease[i] - - padding = " " * (level * tabwidth) - - line = f"{padding}{line}" - - new_code.append(line) - level += increase[i] - - return new_code diff --git a/x2py/codegen/scope.py b/x2py/codegen/scope.py deleted file mode 100644 index 95f07c4f0..000000000 --- a/x2py/codegen/scope.py +++ /dev/null @@ -1,1070 +0,0 @@ -"""Module containing the Scope class""" - -from immutabledict import immutabledict - -from .bind_c import BindCVariable -from .models.datatypes import TupleType -from .models.core import ClassDef -from .models.core import Symbol -from .models.core import ( - DottedVariable, - IndexedElement, - Variable, -) -from x2py.naming import NamingPolicy -from x2py.naming import generated_symbol_rules -from x2py.utilities.strings import create_incremented_string - - -def _resolve_scope_naming_state(parent_scope, naming_policy, public_namespace, symbol_language): - """Return inherited naming policy, public namespace, and symbol language.""" - if naming_policy is None and parent_scope is not None: - naming_policy = parent_scope.naming_policy - if naming_policy is None: - naming_policy = NamingPolicy() - if public_namespace is None and parent_scope is not None: - public_namespace = parent_scope.public_namespace - if symbol_language is None and parent_scope is not None: - symbol_language = parent_scope.symbol_language - if symbol_language is None: - symbol_language = "python" - generated_symbol_rules(symbol_language) - return naming_policy, tuple(public_namespace or ()), symbol_language.casefold() - - -class Scope: - """ - Class representing all objects defined within a given scope. - - This class provides all necessary functionalities for creating new object - names without causing name clashes. It also stores all objects defined - within the scope. This allows us to search for variables only in relevant - scopes. - - Parameters - ---------- - name : str, optional - The name of the scope. The value needs to be provided when it is not a loop. - - decorators : dict, default: () - A dictionary of any decorators which operate on objects in this scope. - - is_loop : bool, default: False - Indicates if the scope represents a loop (in Python variables declared - in loops are not scoped to the loop). - - parent_scope : Scope, default: None - The enclosing scope. - - used_symbols : dict, default: None - A dictionary mapping all the names which we know will appear in the scope and which - we therefore want to avoid when creating new names to their collisionless name. - - original_symbols : dict, default: None - A dictionary which maps names used in the code to the original name used - in the Python code. - - symbolic_aliases : dict, optional - A dictionary which maps indexed tuple elements to variables representing those - elements. This argument should only be used after the semantic stage. - - naming_policy : NamingPolicy, optional - Shared policy used for public Python names and generated target-language - symbols. Child scopes inherit this from their parent. - - public_namespace : tuple[str, ...], optional - Namespace key used when reserving public Python names. - - symbol_language : str, optional - Target language used when reserving generated symbols. - - scope_type : str - The type of the scope being created [module, function, class, loop, program]. - """ - - allow_loop_scoping = False - __slots__ = ( - "_dotted_symbols", - "_dummy_counter", - "_imports", - "_is_loop", - "_locals", - "_loops", - "_name", - "_naming_policy", - "_original_symbol", - "_parent_scope", - "_public_namespace", - "_scope_type", - "_sons_scopes", - "_symbol_language", - "_symbol_prefix", - "_temporary_variables", - "_used_symbols", - ) - - categories = ( - "functions", - "variables", - "classes", - "imports", - "symbolic_aliases", - "decorators", - "cls_constructs", - ) - - def __init__( - self, - *, - name=None, - decorators=(), - is_loop=False, - parent_scope=None, - used_symbols=None, - original_symbols=None, - naming_policy=None, - public_namespace=None, - symbol_language=None, - symbolic_aliases=None, - scope_type, - ): - assert (name is None) != (not is_loop) - assert scope_type in ("module", "function", "class", "loop", "program") - - self._name = name - self._scope_type = scope_type - self._imports = {k: {} for k in self.categories} - - self._locals = {k: {} for k in self.categories} - - prefix_set = () - if parent_scope and parent_scope.symbol_prefix: - prefix_set += (parent_scope.symbol_prefix.removesuffix("__"),) - if name: - prefix_set += (name,) - - self._symbol_prefix = "__".join((*prefix_set, "")) - - self._temporary_variables = [] - - if used_symbols and not isinstance(used_symbols, dict): - raise RuntimeError("Used symbols must be a dictionary") - - self._used_symbols = used_symbols or {} - self._original_symbol = original_symbols or {} - self._naming_policy, self._public_namespace, self._symbol_language = _resolve_scope_naming_state( - parent_scope, - naming_policy, - public_namespace, - symbol_language, - ) - - self._dummy_counter = 0 - - self._locals["decorators"].update(decorators) - if symbolic_aliases: - self._locals["symbolic_aliases"].update(symbolic_aliases) - - # TODO use another name for headers - # => reserved keyword, or use __ - self._parent_scope = parent_scope - self._sons_scopes = {} - - self._is_loop = is_loop - # scoping for loops - self._loops = [] - - self._dotted_symbols = [] - - def new_child_scope(self, name, scope_type, **kwargs): - """ - Create a new child Scope object which has the current object as parent. - - The parent scope can access the child scope through the '_sons_scopes' - dictionary, using the provided name as key. Conversely, the child scope - can access the parent scope through the 'parent_scope' attribute. - - Parameters - ---------- - name : str - Name of the new scope, used as a key to retrieve the new scope. - scope_type : str - The type of the scope being created [module, function, class, loop, program]. - **kwargs : dict - Keyword arguments passed to __init__() for object initialization. - - Returns - ------- - Scope - New child scope, which has the current object as parent. - """ - ps = kwargs.pop("parent_scope", self) - if ps is not self: - raise ValueError(f"A child of {self} cannot have a parent {ps}") - - child = Scope(name=name, **kwargs, parent_scope=self, scope_type=scope_type) - - self.add_son(name, child) - - return child - - @property - def naming_policy(self): - """Policy used to reserve public names and generated symbols.""" - return self._naming_policy - - @property - def public_namespace(self): - """Namespace key used for public wrapper name reservations.""" - return self._public_namespace - - @property - def symbol_language(self): - """Target language used for generated-symbol reservations.""" - return self._symbol_language - - def child_public_namespace(self, *parts): - """Return a child public namespace below the current scope.""" - return (*self._public_namespace, *(str(part) for part in parts)) - - @property - def name(self): - """ - The name of the scope. - - The name of the scope. - """ - return self._name - - @property - def symbol_prefix(self): - """ - The prefix used for symbols. - - The prefix that may be prepended to symbols for context information. - """ - return self._symbol_prefix - - @property - def imports(self): - """A dictionary of objects imported in this scope""" - return self._imports - - @property - def variables(self): - """ - A dictionary of variables defined in this scope. - - A dictionary whose keys are the original Python names of the variables - in the scope and whose values are Variable objects. When handling an - inlined function it is possible that some of the values will not be - Variable objects but rather the value that the variable takes in this - context. - """ - return immutabledict(self._locals["variables"]) - - @property - def classes(self): - """ - A dictionary of classes defined in this scope. - - A dictionary whose keys are the original Python names of the classes - in the scope and whose variables are ClassDef objects. - """ - return immutabledict(self._locals["classes"]) - - @property - def functions(self): - """ - A dictionary of functions defined in this scope. - - A dictionary whose keys are the original Python names of the functions - in the scope and whose variables are ClassDef objects. - """ - return immutabledict(self._locals["functions"]) - - @property - def decorators(self): - """ - A dictionary of the decorators applied to the current function. - - A dictionary of the decorators which are applied to the function definition - in this scope. The keys are the name of the decorator function. The values - depend on the decorator. - """ - return immutabledict(self._locals["decorators"]) - - @property - def cls_constructs(self): - """ - A dictionary of datatypes for the classes defined in this scope. - - A dictionary whose keys are the original Python names of the classes - found in this scope and whose values are the types inheriting from - Type which identify these classes. - """ - return immutabledict(self._locals["cls_constructs"]) - - @property - def symbolic_aliases(self): - """ - A dictionary of symbolic alias defined in this scope. - - A symbolic alias is a symbol declared in the scope which is mapped - to a constant object. E.g. a symbol which represents a type. - """ - return immutabledict(self._locals["symbolic_aliases"]) - - def find(self, name, category=None, local_only=False, raise_if_missing=False): - """ - Find and return the specified object in the scope. - - Find a specified object in the scope and return it. - The object is identified by a string containing its name. - If the object cannot be found then None is returned unless - an error is requested. - - Parameters - ---------- - name : str - The Python name of the object we are searching for. - category : str, optional - The type of object we are searching for. - This must be one of the strings in Scope.categories. - If no value is provided then we look in all categories. - local_only : bool, default=False - Indicates whether we should look for variables in the - entire scope or whether we should limit ourselves to the - local scope. - raise_if_missing : bool, default=False - Indicates whether an error should be raised if the object - cannot be found. - - Returns - ------- - codegen model object - The object stored in the scope. - """ - for local_category in [category] if category else self._locals.keys(): - if name in self._locals[local_category]: - return self._locals[local_category][name] - - if name in self.imports[local_category]: - return self.imports[local_category][name] - - # Walk up the tree of Scope objects, until the root if needed - if self.parent_scope and (self.is_loop or not local_only): - return self.parent_scope.find(name, category, local_only, raise_if_missing) - if raise_if_missing: - raise RuntimeError(f"Can't find expected object {name} in scope") - return None - - def find_all(self, category): - """ - Find and return all objects from the specified category in the scope. - - Find and return all objects from the specified category in the scope. - - Parameters - ---------- - category : str - The type of object we are searching for. - This must be one of the strings in Scope.categories. - - Returns - ------- - dict - A dictionary containing all the objects of the specified category - found in the scope. - """ - result = self.parent_scope.find_all(category) if self.parent_scope else {} - - result.update(self._locals[category]) - result.update(self._imports[category]) - - return result - - @property - def is_loop(self): - """Indicates whether this scope describes a loop""" - return self._is_loop - - def create_new_loop_scope(self): - """ - Create a new Scope within the current scope describing a loop. - - Create a new Scope within the current scope describing a loop - (For/While/etc). - - Returns - ------- - Scope - The newly created loop scope. - """ - new_scope = Scope( - decorators=self.decorators, - is_loop=True, - parent_scope=self, - scope_type="loop", - ) - self.add_loop(new_scope) - return new_scope - - def insert_variable(self, var, name=None): - """ - Add a variable to the current scope. - - Add a variable to the current scope. - - Parameters - ---------- - var : Variable - The variable to be inserted into the current scope. - name : str, default=var.name - The name of the variable in the Python code. - """ - if var.name == "_": - raise ValueError("A temporary variable should have a name generated by Scope.get_new_name") - if not isinstance(var, Variable): - raise TypeError("variable must be of type Variable") - - if name is None: - name = self.get_python_name(var.name) - - if not self.allow_loop_scoping and self.is_loop: - self.parent_scope.insert_variable(var, name) - else: - if name in self._locals["variables"]: - if name in self.symbolic_aliases.values(): - # If the syntactic name is in the symbolic aliases then the link was created - # at the syntactic stage. In this case the element will be created before the - # tuple - return - raise RuntimeError(f"New variable {name} already exists in scope") - - if name == "_": - self._temporary_variables.append(var) - else: - self._locals["variables"][name] = var - - def remove_variable(self, var, name=None, remove_symbol=True): - """ - Remove a variable from anywhere in scope. - - Remove a variable from anywhere in scope. - - Parameters - ---------- - var : Variable - The variable to be removed. - name : str, optional - The name of the variable in the python code - Default : var.name. - remove_symbol : bool, default=True - Indicate if the associated symbol should also be removed. This is assumed - to be true but it may need to be set to false if the variable is removed - in order to update the definition. - """ - if name is None: - name = self.get_python_name(var.name) - - if name in self._locals["variables"]: - self._locals["variables"].pop(name) - if remove_symbol: - self._used_symbols.pop(name) - elif self.parent_scope: - self.parent_scope.remove_variable(var, name) - else: - raise RuntimeError("Variable not found in scope") - - def insert_class(self, cls, name=None): - """ - Add a class to the current scope. - - Add the definition of a class to the current scope to - make it discoverable when used. - - Parameters - ---------- - cls : ClassDef - The class to be inserted into the current scope. - - name : str, optional - The name under which the classes should be indexed in the scope. - This defaults to the name of the class in Python. - """ - if not isinstance(cls, ClassDef): - raise TypeError("class must be of type ClassDef") - - assert not self.is_loop - - if name is None: - name = cls.name - name = self.get_python_name(name) - if name in self._locals["classes"]: - raise RuntimeError(f"A class with name '{name}' already exists in the scope") - assert name in self._used_symbols - self._locals["classes"][name] = cls - - def insert_cls_construct(self, class_type): - """ - Add a class construct to the scope. - - Add a class construct to the scope. A class construct is a type inheriting from - Type which describes the type of a class. - - Parameters - ---------- - class_type : Type - The construct to be inserted. - """ - name = class_type.name - self._locals["cls_constructs"][name] = class_type - - def insert_function(self, func, name): - """ - Add a function to the scope. - - Add a function to the scope. The key will be the original name of the - function in the Python code. - - Parameters - ---------- - func : FunctionDef - The function to be inserted. - name : str | Symbol - The original name of the function in the Python code. This will be - used as the key for the function in the scope. - """ - assert name in self._used_symbols - assert name not in self._locals["functions"] - self._locals["functions"][name] = func - - def insert_symbol(self, symbol, object_type="variable"): - """ - Add a new symbol to the scope. - - Add a new symbol to the scope in the syntactic stage. This should be used to - declare symbols defined by the user. Once the symbol is declared the Scope - generates a collisionless name if necessary which can be used in the target - language without causing problems by being a keyword or being confused with - other symbols (e.g. in Fortran which is not case-sensitive). This new name - can be retrieved later using `Scope.get_expected_name`. - - Parameters - ---------- - symbol : Symbol | DottedName - The symbol to be added to the scope. - - object_type : str, default=variable - The type of the object for which a name is requested (e.g. module, function, - class, variable). - - Returns - ------- - Symbol | DottedName - The new collisionless symbol that will be used in the low-level code. - """ - - if type(symbol).__name__ == "AnnotatedSymbol": - symbol = symbol.name - - if not self.allow_loop_scoping and self.is_loop: - return self.parent_scope.insert_symbol(symbol) - if symbol not in self._used_symbols: - collisionless_name = self._naming_policy.generated_symbol( - symbol, - self.all_used_symbols, - language=self._symbol_language, - prefix=self._symbol_prefix, - context=object_type, - parent_context=self._scope_type, - ) - collisionless_symbol = Symbol(collisionless_name, is_temp=getattr(symbol, "is_temp", False)) - self._used_symbols[symbol] = collisionless_symbol - self._original_symbol[collisionless_symbol] = symbol - return collisionless_symbol - return self._used_symbols[symbol] - - def insert_low_level_symbol(self, python_symbol, low_level_symbol): - """ - Add a new symbol to the scope for which the low-level equivalent is known. - - Add a new symbol to the scope in the syntactic stage. This should be used to - declare symbols defined by the user but mapped to a low-level name (e.g. via - @low_level). - - Parameters - ---------- - python_symbol : Symbol - The symbol to be added to the scope. - low_level_symbol : Symbol - The low-level equivalent of the symbol being added to the scope. - """ - - if not self.allow_loop_scoping and self.is_loop: - self.parent_scope.insert_low_level_symbol(python_symbol, low_level_symbol) - - assert python_symbol not in self._used_symbols - - if self._naming_policy.has_generated_symbol_clash( - low_level_symbol, - self.all_used_symbols, - language=self._symbol_language, - ): - raise ValueError("Low-level name conflicts with name already in use.") - - self._used_symbols[python_symbol] = low_level_symbol - self._original_symbol[low_level_symbol] = python_symbol - - def remove_symbol(self, symbol): - """ - Remove symbol from the scope. - - Remove symbol from the scope. - - Parameters - ---------- - symbol : Symbol - The symbol to be removed from the scope. - """ - - if symbol in self._used_symbols: - collisionless_symbol = self._used_symbols.pop(symbol) - self._original_symbol.pop(collisionless_symbol) - - def insert_symbolic_alias(self, symbol, alias): - """ - Add a new symbolic alias to the scope. - - A symbolic alias is a symbol declared in the scope which is mapped - to a constant object. E.g. a symbol which represents a type. - - Parameters - ---------- - symbol : Symbol - The symbol which will represent the object in the code. - alias : object - The object which will be represented by the symbol. - """ - if not self.allow_loop_scoping and self.is_loop: - self.parent_scope.insert_symbolic_alias(symbol, alias) - else: - symbolic_aliases = self._locals["symbolic_aliases"] - if symbol in symbolic_aliases: - raise ValueError(f"{symbol} cannot represent multiple static concepts") - - symbolic_aliases[symbol] = alias - - @property - def all_used_symbols(self): - """ - Get all low-level symbols which already exist in this scope. - - Get a set containing all low-level symbols which already exist - in this scope. - """ - symbols = self.parent_scope.all_used_symbols if self.parent_scope else set() - symbols.update(self._used_symbols.values()) - return symbols - - @property - def all_python_symbols(self): - """ - Get all Python symbols which already exist in this scope. - - Get a set containing all Python symbols which already exist - in this scope. - """ - symbols = self.parent_scope.all_python_symbols if self.parent_scope else set() - symbols.update(self._used_symbols.keys()) - return symbols - - @property - def local_used_symbols(self): - """ - Get all symbols which already exist in this local scope. - - Get the dictionary describing all symbols which already exist - in the local scope. The local scope is this scope excluding - enclosing scopes. The dictionary's keys are existing symbols - (that were used in the original Python code). Its values are - the collisionless symbols that will be used in the low-level - code to describe these objects. - """ - return self._used_symbols - - def symbol_in_use(self, name): - """ - Determine if a name is already in use in this scope. - - Determine if a name is already in use in this scope. - - Parameters - ---------- - name : Symbol - The name we are searching for. - - Returns - ------- - bool - True if the name has already been inserted into this scope, False otherwise. - """ - if name in self._used_symbols: - return True - if self.parent_scope: - return self.parent_scope.symbol_in_use(name) - return False - - def get_new_name(self, current_name=None, *, is_temp=None, object_type="variable"): - """ - Get a new name which does not clash with any names in the current context. - - Creates a new name. A current_name can be provided indicating the name the - user would like to use if possible. If this name is not available then it - will be used as a prefix for the new name. - If no current_name is provided, then the standard prefix is used, and the - dummy counter is used and updated to facilitate finding the next value of - this common case. - - Parameters - ---------- - current_name : str, default: None - The name the user would like to use if possible. - - is_temp : bool, optional - Indicates if the generated symbol should be a temporary (i.e. an extra - temporary object generated by X2py). This is always the case if no - current_name is provided. - - object_type : str, default=variable - The type of the object for which a name is requested (e.g. module, function, - class, variable). - - Returns - ------- - Symbol - The new name which will be printed in the code. - """ - if current_name is not None and not self._naming_policy.has_generated_symbol_clash( - current_name, - self.all_python_symbols, - language=self._symbol_language, - ): - new_name = Symbol(current_name, is_temp=is_temp) - return self.insert_symbol(new_name, object_type=object_type) - - if current_name is None: - assert is_temp is None - is_temp = True - # Avoid confusing names by also searching in parent scopes - new_name, self._dummy_counter = create_incremented_string( - self.all_used_symbols, - prefix=current_name, - counter=self._dummy_counter, - naming_rules=generated_symbol_rules(self._symbol_language), - ) - else: - if is_temp is None: - is_temp = True - # When a name is suggested, try to stick to it - new_name, _ = create_incremented_string(self.all_used_symbols, prefix=current_name) - - collisionless_name = self._naming_policy.generated_symbol( - new_name, - self.all_used_symbols, - language=self._symbol_language, - prefix=self._symbol_prefix, - context=object_type, - parent_context=self._scope_type, - ) - collisionless_symbol = Symbol(collisionless_name, is_temp=True) - self._used_symbols[collisionless_symbol] = collisionless_symbol - self._original_symbol[collisionless_symbol] = collisionless_symbol - return self.insert_symbol(collisionless_symbol, object_type) - - def reserve_public_name(self, raw_name, *, object_type="variable", owner=None): - """Reserve a Python-visible name in this scope's public namespace.""" - return self._naming_policy.reserve_public_name( - self._public_namespace, - raw_name, - category=object_type, - owner=owner, - ) - - def get_new_public_name( - self, - current_name=None, - *, - python_name=None, - is_temp=None, - object_type="variable", - owner=None, - ): - """Create a low-level symbol and map it to a reserved Python public name.""" - raw_public_name = current_name if python_name is None else python_name - public_name = self.reserve_public_name(raw_public_name, object_type=object_type, owner=owner) - symbol_object_type = "variable" if object_type in {"argument", "field"} else object_type - symbol = self.get_new_name( - current_name if current_name is not None else public_name, - is_temp=is_temp, - object_type=symbol_object_type, - ) - self._original_symbol[symbol] = public_name - return symbol - - def get_temporary_variable(self, dtype_or_var, name=None, **kwargs): - """ - Get a temporary variable. - - Get a temporary variable. - - Parameters - ---------- - dtype_or_var : str, DataType, Variable - In the case of a string of DataType: The type of the Variable to be created - In the case of a Variable: a Variable which will be cloned to set all the Variable properties. - name : str, optional - The requested name for the new variable. - **kwargs : dict - See Variable keyword arguments. - - Returns - ------- - Variable - The temporary variable. - """ - assert isinstance(name, str | type(None)) - name = self.get_new_name(name) - if isinstance(dtype_or_var, Variable): - var = dtype_or_var.clone(name, **kwargs, is_temp=True) - else: - var = Variable(dtype_or_var, name, **kwargs, is_temp=True) - - self.insert_variable(var) - return var - - def get_expected_name(self, start_name): - """ - Get a name with no collisions. - - Get a name with no collisions, ideally the provided name. - The provided name should already exist in the symbols. - - Parameters - ---------- - start_name : str - The name which was used in the Python code. - - Returns - ------- - Symbol - The name which will be used in the generated code. - """ - if start_name == "_": - return self.get_new_name() - if start_name in self._used_symbols: - return self._used_symbols[start_name] - if self.parent_scope: - return self.parent_scope.get_expected_name(start_name) - raise RuntimeError(f"{start_name} does not exist in scope") - - def get_import_alias(self, obj, category=None): - """ - Get the name used to access an imported object in the current scope. - - Get the name used to access an imported object in the current scope. - This is different to the current name when the function was imported - with import X as Y, but only some languages are capable of renaming - methods in this way so the original object's name shouldn't be - modified. - - Parameters - ---------- - obj : model object - The object we are searching for. - category : str, optional - The type of object we are searching for. - This must be one of the strings in Scope.categories. - If no value is provided then we look in all categories. - - Returns - ------- - str - The name used to access an imported object in the current scope. - """ - for local_category in [category] if category else self._locals.keys(): - import_obj = self.imports[local_category] - name = next((n for n, o in import_obj.items() if o is obj), None) - if name: - return name - - if self.parent_scope: - return self.parent_scope.get_import_alias(obj, category) - raise RuntimeError(f"Can't find expected imported object {obj} in scope") - - def collect_all_imports(self): - """Collect the names of all modules necessary to understand this scope""" - imports = list(self._imports["imports"].keys()) - imports.extend([i for s in self._sons_scopes.values() for i in s.collect_all_imports()]) - return imports - - def collect_all_type_vars(self): - """ - Collect all TypeVar objects which are available in this scope. - - Collect all TypeVar objects which are available in this scope. This includes - TypeVars declared in parent scopes. - - Returns - ------- - list[TypeVar] - A list of TypeVars in the scope. - """ - type_vars = {n: t for n, t in self.symbolic_aliases.items() if type(t).__name__ == "TypingTypeVar"} - if self.parent_scope: - parent_type_vars = self.parent_scope.collect_all_type_vars() - parent_type_vars.update(type_vars) - return parent_type_vars - return type_vars - - def update_parent_scope(self, new_parent, is_loop, name=None): - """Change the parent scope""" - if is_loop: - if self.parent_scope: - self.parent_scope.remove_loop(self) - self._parent_scope = new_parent - self.parent_scope.add_loop(self) - else: - if self.parent_scope: - name = self.parent_scope.remove_son(self) - self._parent_scope = new_parent - self.parent_scope.add_son(name, self) - - @property - def parent_scope(self): - """Return the enclosing scope""" - return self._parent_scope - - def remove_loop(self, loop): - """Remove a loop from the scope""" - self._loops.remove(loop) - - def remove_son(self, son): - """Remove a sub-scope from the scope""" - name = [k for k, v in self._sons_scopes.items() if v is son] - assert len(name) == 1 - self._sons_scopes.pop(name[0]) - - def add_loop(self, loop): - """Make parent aware of new child loop""" - assert loop.parent_scope is self - self._loops.append(loop) - - def add_son(self, name, son): - """Make parent aware of new child""" - assert son.parent_scope is self - self._sons_scopes[name] = son - - def get_python_name(self, name): - """ - Get the name used in the original Python code. - - Get the name used in the original Python code from the name used - by the variable that was created in the parser. - - Parameters - ---------- - name : Symbol | str - The name of the Variable in the generated code. - - Returns - ------- - str - The name of the Variable in the original code. - """ - if name in self._original_symbol: - return self._original_symbol[name] - if self.parent_scope: - return self.parent_scope.get_python_name(name) - raise RuntimeError(f"Can't find {name} in scope") - - @property - def python_names(self): - """Get map of new names to original python names""" - return self._original_symbol - - def collect_tuple_element(self, tuple_elem): - """ - Get an element of a tuple. - - This function is mainly designed to handle inhomogeneous tuples. Such tuples - cannot be directly represented in low-level languages. Instead they are replaced - by multiple variables representing each of the elements of the tuple. This - function maps tuple elements (e.g. `var[0]`) to the variable representing that - element in the low-level language (e.g. `var_0`). - - Parameters - ---------- - tuple_elem : model object - The element of the tuple obtained via the `__getitem__` function. - - Returns - ------- - Variable - The variable which represents the tuple element in a low-level language. - - Raises - ------ - X2pyError - An error is raised if the tuple element has not yet been added to the scope. - """ - if isinstance(tuple_elem, IndexedElement) and isinstance(tuple_elem.base, DottedVariable): - cls_scope = tuple_elem.base.lhs.cls_base.scope - if cls_scope is not self: - return cls_scope.collect_tuple_element(tuple_elem) - - if isinstance(tuple_elem, IndexedElement) and isinstance(tuple_elem.base.class_type, TupleType): - for element, alias in self.symbolic_aliases.items(): - if ( - isinstance(element, IndexedElement) - and element.base is tuple_elem.base - and element.indices == tuple_elem.indices - ): - return alias - raise RuntimeError(f"Tuple element {tuple_elem} has no symbolic alias") - - return tuple_elem - - def collect_all_tuple_elements(self, tuple_var): - """ - Create a tuple of variables from a variable representing an inhomogeneous object. - - Create a tuple of variables that can be printed in a low-level language. An - inhomogeneous object cannot be represented as is in a low-level language so - it must be unpacked into a PythonTuple. This function is recursive so that - variables with a type such as `tuple[tuple[int,bool],float]` generate - `PythonTuple(PythonTuple(var_0_0, var_0_1), var_1)`. - - Parameters - ---------- - tuple_var : Variable | FunctionAddress - A variable which may or may not be an inhomogeneous tuple. - - Returns - ------- - list[Variable] - All variables that should be printed in a low-level language to represent - the Variable. - """ - if isinstance(tuple_var, BindCVariable): - tuple_var = tuple_var.new_var - - if isinstance(tuple_var, Variable) and isinstance(tuple_var.class_type, TupleType): - elements = [] - for i in range(len(tuple_var.class_type)): - element = self.collect_tuple_element(IndexedElement(tuple_var, i)) - elements.extend(self.collect_all_tuple_elements(element)) - return elements - - return [tuple_var] diff --git a/x2py/compiling/README.md b/x2py/compiling/README.md index 3780d1e07..281e0bd66 100644 --- a/x2py/compiling/README.md +++ b/x2py/compiling/README.md @@ -11,9 +11,12 @@ linking. | `basic.py` | Compile object model and dependency relationships. | | `compilers.py` | Compiler command execution and tool lookup helpers. | | `default_compilers.py` | Default compiler selection helpers. | -| `python_wrapper.py` | Generated bridge/binding compilation and shared-library creation. | | `runtime_support.py` | Copying and compiling x2py runtime support used by generated wrappers. | +Generated-wrapper compile-object assembly and shared-library orchestration live +in `x2py/pipeline/build.py`, where the canonical rendered wrapper artifacts are +available. The compiling package does not import or regenerate wrapper plans. + ## Pipeline Position ```text diff --git a/x2py/compiling/python_wrapper.py b/x2py/compiling/python_wrapper.py deleted file mode 100644 index ad06a79b2..000000000 --- a/x2py/compiling/python_wrapper.py +++ /dev/null @@ -1,186 +0,0 @@ -""" -Module containing the `create_shared_library` function which creates a CPython -extension module. This is a shared library which can be called from Python. It -is created from a `CodePrinter` object describing code which has been printed -in a target language. -""" - -import os -import time - -from .basic import CompileObj -from .runtime_support import install_runtime_support -from x2py.codegen.binding_pipeline import BindingPipeline - -__all__ = ["create_shared_library"] - - -def _print_verbose_timing(verbose, label, elapsed): - """Print one elapsed build-stage timing when verbose output is enabled.""" - if verbose: - print(f">> Timing :: {label}: {elapsed:.3f}s") - - -# ============================================================================== -def create_shared_library( - codegen, - main_obj, - *, - language, - wrapper_flags, - x2py_dirpath, - output_dirpath, - compiler, - sharedlib_modname=None, - dependencies=(), - verbose, -): - """ - Create a shared library which can be called from X2py. - - From a CodePrinter object describing code which has been printed - in a target language, create a shared library which can be - called from X2py. In order to do this the code must be wrapped. - First, if the code is not written in C, it must be wrapped to - make it callable from C. This intermediary code is printed - and compiled. From the C-compatible code a second (first for C) - wrapper is created which exposes the C code to Python. This - is done via the CWrapper. Finally this new code is compiled - to generate the required shared language. - - Parameters - ---------- - codegen : x2py.codegen.printing.codeprinter.CodePrinter - The printer which was used to print the translated code. - - main_obj : x2py.codegen.compiling.basic.CompileObj - The compile object which describes the translated code. - - language : str - The language which X2py translated to. - - wrapper_flags : iterable - Any additional flags which should be used to compile the wrapper. - - x2py_dirpath : str - The path to the directory where the files are created and compiled. - - output_dirpath : str - Path to the directory where the shared library should be outputted. - - compiler : x2py.codegen.compiling.compilers.Compiler - The compiler which should be used to compile the library. - - sharedlib_modname : str, default: None - The name of the shared library. The default is the name of the - module printed by the printer. - - verbose : int - Indicates the level of verbosity. - - Returns - ------- - sharedlib_filepath : str - The absolute path to the shared library which was created. - - timings : dict - The time spent in the different parts of the library creation. - """ - timings = {} - - # Get module name - module_name = codegen.name - - # Name of shared library - if sharedlib_modname is None: - sharedlib_modname = module_name - - gen = BindingPipeline(codegen, module_name, language, verbose) - - # ------------------------------------------- - # Wrap code - # ------------------------------------------- - - start_wrapper_creation = time.perf_counter() - gen.generate(os.path.dirname(x2py_dirpath)) - timings["Wrapper creation"] = time.perf_counter() - start_wrapper_creation - _print_verbose_timing(verbose, "Wrapper creation", timings["Wrapper creation"]) - - # ------------------------------------------- - # Print wrapper code - # ------------------------------------------- - - start_wrapper_printing = time.perf_counter() - wrapper_files = gen.write(x2py_dirpath) - timings["Wrapper printing"] = time.perf_counter() - start_wrapper_printing - _print_verbose_timing(verbose, "Wrapper printing", timings["Wrapper printing"]) - - printed_languages = gen.generated_languages - - # ------------------------------------------- - # Prepare the compile objects - # ------------------------------------------- - - wrapper_compile_objs = [ - CompileObj( - filepath.name, - x2py_dirpath, - flags=main_obj.flags, - dependencies=(main_obj,), - ) - for filepath in wrapper_files[:-1] - ] + [ - CompileObj( - wrapper_files[-1].name, - x2py_dirpath, - flags=wrapper_flags, - link_args=main_obj.link_args, - dependencies=(main_obj, *dependencies), - extra_compilation_tools=("python",), - ) - ] - - for i, (obj, lang, imports) in enumerate( - zip( - wrapper_compile_objs, - printed_languages, - gen.get_additional_imports(), - strict=True, - ) - ): - obj.add_dependencies(*wrapper_compile_objs[:i]) - install_runtime_support( - imports, - x2py_dirpath=x2py_dirpath, - compiler=compiler, - wrapper_obj=obj, - language=lang, - verbose=verbose, - ) - - # ------------------------------------------- - # Compile code - # ------------------------------------------- - - start_compile_wrapper = time.perf_counter() - for obj, wrapper_language in zip(wrapper_compile_objs, printed_languages, strict=True): - compiler.compile_module( - compile_obj=obj, - output_folder=x2py_dirpath, - language=wrapper_language, - verbose=verbose, - ) - - sharedlib_filepath = compiler.compile_shared_library( - wrapper_compile_objs[-1], - output_folder=output_dirpath, - sharedlib_modname=sharedlib_modname, - language=language, - verbose=verbose, - ) - - timings["Wrapper compilation"] = time.perf_counter() - start_compile_wrapper - _print_verbose_timing(verbose, "Wrapper compilation", timings["Wrapper compilation"]) - - # Return absolute path of shared library - return sharedlib_filepath, timings diff --git a/x2py/compiling/runtime_support.py b/x2py/compiling/runtime_support.py index 5d5e1eb69..c8a5cd83d 100644 --- a/x2py/compiling/runtime_support.py +++ b/x2py/compiling/runtime_support.py @@ -4,9 +4,9 @@ import shutil from filelock import FileLock +import numpy as np import x2py.stdlib as stdlib_folder -from x2py.codegen.bindings.numpy_cpython_api import get_numpy_max_acceptable_version_file from .basic import CompileObj @@ -15,6 +15,17 @@ _RUNTIME_SOURCE = Path(stdlib_folder.__file__).parent / _RUNTIME_IMPORT +def _numpy_version_header() -> str: + """Return NumPy API version guards for the bundled native runtime.""" + maximum_supported = [1, 19] + current = [int(value) for value in np.version.version.split(".")[:2]] + major, minor = min(maximum_supported, current) + header = f"#ifndef NPY_NO_DEPRECATED_API\n# define NPY_NO_DEPRECATED_API NPY_{major}_{minor}_API_VERSION\n#endif\n" + if current[0] >= 2: + header += "#ifndef NPY_TARGET_VERSION\n# define NPY_TARGET_VERSION NPY_2_0_API_VERSION\n#endif\n" + return header + + def install_runtime_support(imports, *, x2py_dirpath, compiler, wrapper_obj, language, verbose): """Copy, register, and compile runtime support imported by one wrapper.""" if not any(name == _RUNTIME_IMPORT or name.startswith(f"{_RUNTIME_IMPORT}/") for name in imports): @@ -28,7 +39,7 @@ def install_runtime_support(imports, *, x2py_dirpath, compiler, wrapper_obj, lan shutil.copytree(_RUNTIME_SOURCE, destination) (destination / "numpy_version.h").write_text( - get_numpy_max_acceptable_version_file(), + _numpy_version_header(), encoding="utf-8", ) runtime_obj = CompileObj( diff --git a/x2py/fortran_parser/cli.py b/x2py/fortran_parser/cli.py index 35a9841ff..302576015 100644 --- a/x2py/fortran_parser/cli.py +++ b/x2py/fortran_parser/cli.py @@ -81,7 +81,7 @@ def _parse_paths(paths: list[str]) -> dict[str, dict]: def _semantic_report(paths: list[str]) -> dict[str, dict]: """Generate semantic IR and pyi text per parsed file.""" from x2py.semantics.fortran2ir import fortran_module_to_semantic_module - from x2py.codegen.printers.pyi_printer import emit_module + from x2py.wrapper_codegen.printers import emit_module parsed = _parse_paths(paths) semantic_out: dict[str, dict] = {} diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 11bc05a13..61f898890 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -12,15 +12,11 @@ from filelock import FileLock -from x2py.codegen.codegen import Codegen -from x2py.codegen.scope import Scope from x2py.compiling.basic import CompileObj from x2py.compiling.compilers import Compiler, get_condaless_search_path -from x2py.compiling.python_wrapper import _print_verbose_timing, create_shared_library from x2py.compiling.runtime_support import install_runtime_support from x2py.fortran_parser.parser import parse_fortran_project from x2py.probes.fortran_types import evaluate_fortran_type_facts, evaluate_fortran_type_requirements -from x2py.naming import NamingPolicy from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source from x2py.pipeline.wrapper_artifacts import GeneratedSourceFile, RenderedGeneratedWrapperArtifacts from x2py.semantics.fortran2ir import ( @@ -28,7 +24,6 @@ collect_semantic_compile_time_requirements, fortran_project_to_semantic_modules, ) -from x2py.semantics.ir2ast import semantic_ir_to_codegen_ast from x2py.semantics.models import ( PYTHON_EXPORTS_METADATA, PYTHON_EXPORTS_PREPARED_METADATA, @@ -51,7 +46,6 @@ WrapperCodeGenerator, WrapperPlanner, WrapperPlanSupportAnalyzer, - WrapperPlanSupportReport, ) @@ -76,224 +70,12 @@ _RENDERED_WRAPPER_RUNTIME_IMPORTS = { "python_runtime": ("x2py_runtime/python_runtime",), } -_WRAPPER_PLAN_COMPLETED_LANES = frozenset( - { - # Scalar rollout lanes. - "scalar-inputs", - "scalar-storage-inputs", - "scalar-raw-address-inputs", - "scalar-direct-results", - "scalar-hidden-outputs", - "scalar-multiple-results", - "scalar-optional-inputs", - "scalar-descriptor-inputs", - "scalar-writebacks", - "scalar-module-variables", - # String rollout lanes. - "string-value-inputs", - "string-storage-inputs", - "string-raw-address-inputs", - "string-optional-inputs", - "string-writebacks", - "fixed-string-direct-results", - "fixed-string-hidden-outputs", - # Ordinary-array rollout lanes. - "array-buffer-inputs", - "array-native-handle-actuals", - "array-raw-address-inputs", - "array-optional-inputs", - "array-writebacks", - "array-direct-results", - "array-hidden-outputs", - "array-multiple-results", - "array-copy-to-fortran", - # Native-array handle and descriptor rollout lanes. - "allocatable-descriptor-inputs", - "pointer-descriptor-inputs", - "optional-native-array-handles", - "projected-native-array-handles", - "owned-allocatable-results", - "owned-allocatable-hidden-outputs", - "allocatable-module-handles", - "pointer-module-handles", - "scalar-descriptor-results", - "deferred-string-descriptor-results", - # Derived-object rollout lanes. - "derived-wrapper-inputs", - "optional-derived-inputs", - "in-place-derived-inputs", - "typed-derived-value-inputs", - "derived-direct-results", - "derived-hidden-outputs", - "plain-derived-module-proxies", - "aliased-derived-module-objects", - "derived-module-constant-values", - "derived-scalar-fields", - "derived-string-fields", - "derived-array-fields", - "derived-native-handle-fields", - "derived-borrowed-field-owners", - # Generated class orchestration lanes. - "class-registration", - "default-class-constructors", - "bound-class-constructors", - "overloaded-class-constructors", - "instance-methods", - "static-methods", - "class-overloads", - "class-finalizers", - "class-inheritance", - "scalar-polymorphic-inputs", - # Immediate callback rollout lane. - "immediate-callbacks", - "callback-context-runtime", - "callback-same-thread-reentry", - "callback-fatal-errors", - "callback-scalar-values", - "callback-scalar-storage", - "callback-fixed-strings", - "callback-arrays", - "callback-derived-values", - "callback-results", - # Cross-cutting rollout lanes. - "void-calls", - "python-namespaces", - "native-call-runtime", - "native-status-errors", - } -) -_WRAPPER_PLAN_EVIDENCE = ( - "tests/wrapper/fortran/scalars/test_verified_baseline.py::" - "test_fmath_scalar_sources_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_fixed_optional_scalar_wrapper_plan_route_matches_all_presence_states", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_allocatable_scalar_descriptor_distinguishes_omitted_none_and_value", - "tests/wrapper/fortran/function_calls/test_scalar_writeback_plan.py::" - "test_scalar_copy_in_out_returns_replacement_through_both_routes", - "tests/wrapper/fortran/module_state/test_scalar_module_variable_plan.py::" - "test_whole_scalar_module_variable_behavior_matches_legacy_route", - "tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py::" - "test_complete_general_source_preserves_namespaces_through_both_routes", - "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" - "test_compiled_runtime_policies_release_gil_and_project_native_errors", - "tests/wrapper/fortran/runtime_behavior/test_runtime_policies.py::" - "test_pyi_runtime_policies_release_gil_and_project_native_errors", - "tests/wrapper/fortran/runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_value_storage_raw_address_out_and_inout_match_both_routes", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_primitive_kinds_match_both_routes_without_array_blockers", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_multiple_scalar_results_match_both_routes_without_array_blockers", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_required_scalar_string_inputs_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_fixed_string_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_fixed_hidden_string_output_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_fixed_string_replacement_and_identity_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_edge_cases.py::" - "test_assumed_and_optional_string_replacements_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_fixed_string_storage_and_raw_address_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/scalars/test_verified_baseline.py::" - "test_required_array_buffers_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_multidimensional_arrays.py::" - "test_dense_strided_and_projected_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_ordinary_array_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_assumed_rank_arrays.py::" - "test_assumed_rank_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_array_buffers_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_output_arguments.py::" - "test_hidden_ordinary_array_output_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_raw_array_addresses_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py::" - "test_copy_f_preserves_logical_axes_through_binding_owned_temporary", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_raw_fixed_width_character_arrays_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_pointers.py::" - "test_module_native_array_handles_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/function_calls/test_optional_arguments.py::" - "test_optional_array_descriptors_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/module_state/test_allocatable_replacement.py::" - "test_projected_allocatable_descriptor_matches_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_owned_allocatable_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/arrays/test_array_results.py::" - "test_array_results_follow_data_buffer_and_descriptor_handle_contracts", - "tests/wrapper/fortran/scalars/test_scalar_boundary_plan.py::" - "test_scalar_descriptor_results_copy_values_or_none_through_wrapper_plan_route", - "tests/wrapper/fortran/module_state/test_allocatable_views.py::" - "test_scalar_descriptor_module_variables_return_copied_optional_values", - "tests/wrapper/fortran/module_state/test_allocatable_views.py::" - "test_plain_allocatable_module_array_exposes_current_live_view", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_deferred_allocatable_string_results_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/strings/test_character_arguments.py::" - "test_deferred_character_array_handles_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_scalar_derived_objects_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_value_copy_and_optional_derived_inputs_match_source_oracle", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_plain_module_derived_proxy_reads_and_writes_live_members", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_aliased_module_derived_object_uses_direct_live_field_handles", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_derived_module_constant_returns_independent_owned_values", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_fixed_string_fields_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_pointer_field_descriptor_views_match_legacy_and_wrapper_plan_routes", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_borrowed_child_retains_owner_and_finalizes_exactly_once", - "tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::" - "test_eligible_derived_contract_selects_production_plan_without_legacy_lowering", - "tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::" - "test_bound_constructor_replaces_field_initialization_and_reuses_method_plan", - "tests/wrapper/fortran/naming/test_phase9_class_overloads.py::" - "test_exact_method_overloads_match_without_trial_calls", - "tests/wrapper/fortran/naming/test_phase9_class_overloads.py::" - "test_constructor_overloads_share_owned_allocation_and_exact_matching", - "tests/wrapper/fortran/callbacks/test_all_callback_shapes.py::" - "test_immediate_callbacks_cover_all_supported_argument_shapes", - "tests/wrapper/fortran/callbacks/test_scalar_callbacks.py::" - "test_callback_exception_prints_traceback_and_aborts_host_process", -) - - -@dataclass(frozen=True) -class WrapperPlanRouteDecision: - """One inspectable route decision for a policy-completed generation unit.""" - - owner_path: str - selected_route: str - support_report: WrapperPlanSupportReport - rollout_eligible: bool - rollout_evidence: tuple[str, ...] - selection_reason: str - - @property - def covered_lanes(self) -> tuple[str, ...]: - """Return the completed lanes reported for this generation unit.""" - return self.support_report.covered_lanes - @property - def blockers(self): - """Return stable owner-path blockers from support analysis.""" - return self.support_report.blockers - @property - def uses_wrapper_plan(self) -> bool: - """Return whether this decision selects wrapper-plan generation.""" - return self.selected_route == "wrapper-plan" +def _print_verbose_timing(verbose: bool | int, label: str, elapsed: float) -> None: + """Print one elapsed build-stage timing when verbose output is enabled.""" + if verbose: + print(f">> Timing :: {label}: {elapsed:.3f}s") @dataclass(frozen=True) @@ -710,109 +492,59 @@ def _build_rendered_wrapper_extension( ) -def _select_wrapper_plan_route( - module: SemanticModule, +def _attach_build_makefile( + result: WrapperBuildResult, *, - makefile: bool, - strict_wrapper_names: bool, - force_legacy: bool = False, - force_wrapper_plan: bool = False, -) -> WrapperPlanRouteDecision: - """Select one complete wrapper route from completed support and rollout data.""" - if force_legacy and force_wrapper_plan: - raise ValueError("cannot force both legacy and wrapper-plan routes") - - support_report = WrapperPlanSupportAnalyzer().analyze(module) - support_report.freeze() - if force_legacy: - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="legacy", - support_report=support_report, - rollout_eligible=False, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="legacy route forced for migration rollback or comparison", - ) - if not support_report.supported: - if force_wrapper_plan: - details = "; ".join(f"{item.owner_path}: {item.reason}" for item in support_report.blockers) - raise ValueError( - f"cannot force wrapper-plan route for unsupported generation unit {module.name!r}: {details}" - ) - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="legacy", - support_report=support_report, - rollout_eligible=False, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="generation unit has unsupported wrapper-plan owners", - ) - if makefile or strict_wrapper_names: - mode = "makefile" if makefile else "strict-wrapper-name" - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="legacy", - support_report=support_report, - rollout_eligible=False, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason=f"{mode} mode remains on the legacy route", - ) - if force_wrapper_plan: - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="wrapper-plan", - support_report=support_report, - rollout_eligible=False, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="wrapper-plan route forced for internal migration verification", - ) - if not frozenset(support_report.covered_lanes) <= _WRAPPER_PLAN_COMPLETED_LANES: - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="legacy", - support_report=support_report, - rollout_eligible=False, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="covered lanes exceed the recorded wrapper-plan parity evidence", - ) - return WrapperPlanRouteDecision( - owner_path=module.name, - selected_route="wrapper-plan", - support_report=support_report, - rollout_eligible=True, - rollout_evidence=_WRAPPER_PLAN_EVIDENCE, - selection_reason="whole generation unit is covered by completed wrapper-plan lanes", + compiler: Compiler, + source_objects: tuple[CompileObj, ...], + extra_dependencies: tuple[Path, ...] = (), + build_manifest: Path | None = None, +) -> WrapperBuildResult: + """Attach one replayable Makefile to an unexecuted canonical build.""" + build_makefile = _write_build_makefile( + path=result.output_dir / "Makefile.x2py", + commands=compiler.command_log, + source_objects=source_objects, + working_directory=Path.cwd(), + extra_dependencies=extra_dependencies, + ) + additions = tuple(path for path in (build_manifest, build_makefile) if path is not None) + return replace( + result, + build_makefile=build_makefile, + compiled=False, + generated_files=(*result.generated_files, *additions), + build_manifest=build_manifest, ) -def _render_selected_wrapper_plan(module: SemanticModule) -> RenderedGeneratedWrapperArtifacts: - """Render one selected complete module through the wrapper-plan route.""" +def _render_wrapper_plan(module: SemanticModule) -> RenderedGeneratedWrapperArtifacts: + """Render one policy-completed module through the canonical generator.""" plan = WrapperPlanner().build(module) return WrapperCodeGenerator().generate(plan) +def _require_wrapper_plan_support(module: SemanticModule) -> None: + """Reject unsupported completed owners without retrying another generator.""" + report = WrapperPlanSupportAnalyzer().analyze(module) + report.freeze() + if report.supported: + return + details = "; ".join(f"{item.owner_path}: {item.reason}" for item in report.blockers) + raise ValueError(f"Unsupported wrapper generation unit {module.name!r}: {details}") + + def _generated_wrapper_plan_artifacts( module: SemanticModule, *, - makefile: bool, strict_wrapper_names: bool, - force_legacy: bool, - force_wrapper_plan: bool, verbose: bool | int = False, -) -> RenderedGeneratedWrapperArtifacts | None: - """Complete policy and generate selected wrapper-plan artifacts, if chosen.""" +) -> RenderedGeneratedWrapperArtifacts: + """Complete policy and generate the one production wrapper representation.""" creation_started = time.perf_counter() - complete_semantic_policies(module) - decision = _select_wrapper_plan_route( - module, - makefile=makefile, - strict_wrapper_names=strict_wrapper_names, - force_legacy=force_legacy, - force_wrapper_plan=force_wrapper_plan, - ) - if not decision.uses_wrapper_plan: - return None - rendered = _render_selected_wrapper_plan(module) + complete_semantic_policies(module, strict_wrapper_names=strict_wrapper_names) + _require_wrapper_plan_support(module) + rendered = _render_wrapper_plan(module) _print_verbose_timing(verbose, "Wrapper creation", time.perf_counter() - creation_started) return rendered @@ -1708,22 +1440,6 @@ def _merge_wrapper_modules(modules: list[SemanticModule], *, name: str | None = ) -def _wrapper_codegen_module_name(codegen_ast, requested_name: str, *, explicit_output_name: bool) -> str: - if explicit_output_name: - module_name = requested_name - else: - module_name = codegen_ast.scope.naming_policy.reserve_public_name( - (), - requested_name, - category="module", - owner=requested_name, - ) - - codegen_ast._name = module_name - codegen_ast.scope._original_symbol[module_name] = module_name - return str(module_name) - - def _wrapper_module_metadata(modules: list[SemanticModule]) -> dict[str, object]: metadata: dict[str, object] = {"wrapper_native_modules": _wrapper_native_modules(modules)} if any(module.metadata.get(PYTHON_EXPORTS_PREPARED_METADATA) for module in modules): @@ -1956,8 +1672,6 @@ def build_fortran_extension( wrapper_compiler_debug: bool = False, wrapper_fortran_flags: Iterable[str] | None = None, wrapper_c_flags: Iterable[str] | None = None, - _force_legacy_wrapper_route: bool = False, - _force_wrapper_plan_route: bool = False, ) -> WrapperBuildResult: """Build one extension, or generate its Makefile, from ordered sources.""" @@ -2005,10 +1719,7 @@ def build_fortran_extension( module = _merge_wrapper_modules(modules, name=requested_name) rendered_wrapper_plan = _generated_wrapper_plan_artifacts( module, - makefile=makefile, strict_wrapper_names=strict_wrapper_names, - force_legacy=_force_legacy_wrapper_route, - force_wrapper_plan=_force_wrapper_plan_route, verbose=verbose, ) @@ -2032,93 +1743,26 @@ def build_fortran_extension( verbose=verbose, ) - if rendered_wrapper_plan is not None: - return _build_rendered_wrapper_extension( - rendered_wrapper_plan, - output_dir=output_path, - shared_library_output_dir=shared_library_output_path, - sources=source_paths, - native_build_plan=native_build_plan, - native_dependencies=source_objects, - native_link_args=_rendered_wrapper_native_link_args(native_build_plan), - wrapper_fortran_flags=wrapper_fortran_flags, - wrapper_c_flags=wrapper_c_flags, - compiler=compiler, - verbose=verbose, - ) - - scope = Scope( - name=module.name, - scope_type="module", - naming_policy=NamingPolicy(strict_public_names=strict_wrapper_names), - public_namespace=(module.name.casefold(),), - ) - codegen_ast = semantic_ir_to_codegen_ast(module, scope) - module_name = _wrapper_codegen_module_name( - codegen_ast, - requested_name, - explicit_output_name=output_name is not None, - ) - - codegen = Codegen(module_name, codegen_ast, codegen_ast.scope) - module_obj = CompileObj( - file_name=module_name, - folder=str(output_path), - flags=wrapper_fortran_flags, - has_target_file=False, - ) - shared_library, _timings = create_shared_library( - codegen, - module_obj, - language="fortran", - wrapper_flags=wrapper_c_flags, - x2py_dirpath=str(output_path), - output_dirpath=str(shared_library_output_path), + result = _build_rendered_wrapper_extension( + rendered_wrapper_plan, + output_dir=output_path, + shared_library_output_dir=shared_library_output_path, + sources=source_paths, + native_build_plan=native_build_plan, + native_dependencies=source_objects, + native_link_args=_rendered_wrapper_native_link_args(native_build_plan), + wrapper_fortran_flags=wrapper_fortran_flags, + wrapper_c_flags=wrapper_c_flags, compiler=compiler, - sharedlib_modname=module_name, - dependencies=source_objects, verbose=verbose, ) - - shared_library_path = Path(shared_library) - build_makefile = ( - _write_build_makefile( - path=output_path / "Makefile.x2py", - commands=compiler.command_log, + if makefile: + result = _attach_build_makefile( + result, + compiler=compiler, source_objects=source_objects, - working_directory=Path.cwd(), ) - if makefile - else None - ) - generated_sources = tuple( - path - for path in ( - output_path / f"bind_c_{module_name}_wrapper.f90", - output_path / f"{module_name}_wrapper.c", - output_path / f"{module_name}_wrapper.h", - ) - if path.exists() - ) - generated_files = _expected_generated_files( - source_objects=source_objects, - output_dir=output_path, - module_name=module_name, - shared_library=shared_library_path, - ) - if build_makefile is not None: - generated_files = (*generated_files, build_makefile) - return WrapperBuildResult( - sources=source_paths, - module_name=module_name, - output_dir=output_path, - shared_library=shared_library_path, - build_makefile=build_makefile, - compiled=not makefile, - generated_sources=generated_sources, - generated_files=generated_files, - native_build_plan=native_build_plan, - ) + return result def build_pyi_extension( @@ -2140,8 +1784,6 @@ def build_pyi_extension( wrapper_compiler_debug: bool = False, wrapper_fortran_flags: Iterable[str] | None = None, wrapper_c_flags: Iterable[str] | None = None, - _force_legacy_wrapper_route: bool = False, - _force_wrapper_plan_route: bool = False, ) -> WrapperBuildResult: """Build one extension from one entry `.pyi` and native link inputs.""" @@ -2175,10 +1817,7 @@ def build_pyi_extension( module = _merge_wrapper_modules(modules, name=requested_name) rendered_wrapper_plan = _generated_wrapper_plan_artifacts( module, - makefile=makefile, strict_wrapper_names=strict_wrapper_names, - force_legacy=_force_legacy_wrapper_route, - force_wrapper_plan=_force_wrapper_plan_route, verbose=verbose, ) @@ -2211,132 +1850,45 @@ def build_pyi_extension( ) native_array_build_requirements = native_array_handle_build_requirements(module) - if rendered_wrapper_plan is not None: - result = _build_rendered_wrapper_extension( - rendered_wrapper_plan, - output_dir=output_path, - shared_library_output_dir=shared_library_output_path, - sources=bundle.paths, - native_build_plan=native_build_plan, - native_dependencies=native_source_objects, - native_link_args=_rendered_wrapper_native_link_args(native_build_plan), - wrapper_fortran_flags=wrapper_fortran_flags, - wrapper_c_flags=wrapper_c_flags, - compiler=compiler, - verbose=verbose, - ) - return _with_pyi_manifest( - result, - bundle=bundle, - strict_wrapper_names=strict_wrapper_names, - requested_output_name=output_name, - native_fortran_flags=native_inputs.source_flags, - wrapper_compiler_debug=wrapper_compiler_debug, - wrapper_fortran_flags=wrapper_fortran_flags, - wrapper_c_flags=wrapper_c_flags, - native_array_build_requirements=native_array_build_requirements, - ) - - scope = Scope( - name=module.name, - scope_type="module", - naming_policy=NamingPolicy(strict_public_names=strict_wrapper_names), - public_namespace=(module.name.casefold(),), - ) - codegen_ast = semantic_ir_to_codegen_ast(module, scope) - module_name = _wrapper_codegen_module_name( - codegen_ast, - requested_name, - explicit_output_name=output_name is not None, - ) - codegen = Codegen(module_name, codegen_ast, codegen_ast.scope) - module_obj = CompileObj( - file_name=module_name, - folder=str(output_path), - flags=wrapper_fortran_flags, - has_target_file=False, - include=include_dirs, - libdir=native_inputs.library_dirs, - link_args=_native_link_args(native_build_plan.link_items), - ) - shared_library, _timings = create_shared_library( - codegen, - module_obj, - language="fortran", - wrapper_flags=wrapper_c_flags, - x2py_dirpath=str(output_path), - output_dirpath=str(shared_library_output_path), + result = _build_rendered_wrapper_extension( + rendered_wrapper_plan, + output_dir=output_path, + shared_library_output_dir=shared_library_output_path, + sources=bundle.paths, + native_build_plan=native_build_plan, + native_dependencies=native_source_objects, + native_link_args=_rendered_wrapper_native_link_args(native_build_plan), + wrapper_fortran_flags=wrapper_fortran_flags, + wrapper_c_flags=wrapper_c_flags, compiler=compiler, - sharedlib_modname=module_name, - dependencies=(), verbose=verbose, ) - - shared_library_path = Path(shared_library) - manifest = _pyi_build_manifest( + result = _with_pyi_manifest( + result, bundle=bundle, - module_name=module_name, - output_dir=output_path, - shared_library=shared_library_path, strict_wrapper_names=strict_wrapper_names, requested_output_name=output_name, native_fortran_flags=native_inputs.source_flags, wrapper_compiler_debug=wrapper_compiler_debug, wrapper_fortran_flags=wrapper_fortran_flags, wrapper_c_flags=wrapper_c_flags, - native_build_plan=native_build_plan, native_array_build_requirements=native_array_build_requirements, - manifest_dir=output_path, ) - build_manifest = _write_build_manifest(output_path / _BUILD_MANIFEST_NAME, manifest) if makefile else None - makefile_dependencies = ( - *bundle.paths, - *_link_item_paths(native_build_plan.link_items), - *((build_manifest,) if build_manifest is not None else ()), - ) - build_makefile = ( - _write_build_makefile( - path=output_path / "Makefile.x2py", - commands=compiler.command_log, - source_objects=native_source_objects, - working_directory=Path.cwd(), - extra_dependencies=makefile_dependencies, + if makefile: + build_manifest = _write_build_manifest(output_path / _BUILD_MANIFEST_NAME, result.manifest) + dependencies = ( + *bundle.paths, + *_link_item_paths(native_build_plan.link_items), + build_manifest, ) - if makefile - else None - ) - generated_sources = tuple( - path - for path in ( - output_path / f"bind_c_{module_name}_wrapper.f90", - output_path / f"{module_name}_wrapper.c", - output_path / f"{module_name}_wrapper.h", + result = _attach_build_makefile( + result, + compiler=compiler, + source_objects=native_source_objects, + extra_dependencies=dependencies, + build_manifest=build_manifest, ) - if path.exists() - ) - generated_files = _expected_generated_files( - source_objects=native_source_objects, - output_dir=output_path, - module_name=module_name, - shared_library=shared_library_path, - ) - if build_manifest is not None: - generated_files = (*generated_files, build_manifest) - if build_makefile is not None: - generated_files = (*generated_files, build_makefile) - return WrapperBuildResult( - sources=bundle.paths, - module_name=module_name, - output_dir=output_path, - shared_library=shared_library_path, - build_makefile=build_makefile, - compiled=not makefile, - generated_sources=generated_sources, - generated_files=generated_files, - native_build_plan=native_build_plan, - build_manifest=build_manifest, - manifest=manifest, - ) + return result def build_pyi_extension_from_manifest( diff --git a/x2py/runtime/handles.py b/x2py/runtime/handles.py index c6dc76672..7534a651c 100644 --- a/x2py/runtime/handles.py +++ b/x2py/runtime/handles.py @@ -959,12 +959,17 @@ def _native_array_actual_argument_for_binding_positional( require_contiguous: bool = False, ) -> tuple[int, ...]: """Pack a normal array actual into generated Bind-C array descriptor fields.""" + strided_ndarray = include_strides and isinstance(value, np.ndarray) + if strided_ndarray: + _validate_ndarray_positive_strides(value) actual = _native_array_actual_for_binding( value, expected_dtype=expected_dtype, expected_rank=expected_rank, expected_shape=expected_shape, - expected_layout=expected_layout, + # Positive-stride validation below is the exact Fortran-order contract + # for a strided ndarray; NumPy's contiguous flag is intentionally false. + expected_layout=None if strided_ndarray else expected_layout, require_writeable=bool(require_writeable), require_native_byte_order=bool(require_native_byte_order), require_aligned=bool(require_aligned), @@ -978,11 +983,60 @@ def _native_array_actual_argument_for_binding_positional( fields.append(itemsize) fields.extend(shape) if include_strides: - fields.extend(shape) - fields.extend(1 for _axis in shape) + extents, upper_bounds, strides = _normal_array_actual_stride_facts(actual, shape, itemsize) + fields[-len(shape) :] = extents + fields.extend(upper_bounds) + fields.extend(strides) return tuple(fields) +def _normal_array_actual_stride_facts( + actual: Any, + shape: tuple[int, ...], + itemsize: int, +) -> tuple[tuple[int, ...], tuple[int, ...], tuple[int, ...]]: + """Pack the positive-stride base extent and slice facts used by the bridge.""" + if isinstance(actual, _NativeArrayHandoff): + return shape, tuple(max(extent - 1, -1) for extent in shape), (1,) * len(shape) + if not isinstance(actual, np.ndarray): + raise TypeError(f"normal array actual operation returned unsupported value {type(actual).__name__}") + if actual.size == 0: + # NumPy may report zero strides for empty dimensions. No element can + # be addressed, so use the bridge's canonical empty-array facts. + return shape, tuple(max(extent - 1, -1) for extent in shape), (1,) * len(shape) + if any(stride <= 0 for stride in actual.strides): + raise ValueError("array actual strides must be positive") + + extents = [] + upper_bounds = [] + relative_strides = [] + base_product = 1 + element_strides = tuple(stride // itemsize for stride in actual.strides) + for axis, (logical_extent, element_stride) in enumerate(zip(shape, element_strides, strict=True)): + relative_stride = element_stride // base_product + relative_strides.append(relative_stride) + upper_bound = -1 if logical_extent == 0 else (logical_extent - 1) * relative_stride + upper_bounds.append(upper_bound) + base_extent = max(element_strides[axis + 1] // base_product, 1) if axis + 1 < len(shape) else upper_bound + 1 + extents.append(base_extent) + base_product *= base_extent + return tuple(extents), tuple(upper_bounds), tuple(relative_strides) + + +def _validate_ndarray_positive_strides(value: np.ndarray) -> None: + """Match the bridge's positive non-overlapping Fortran slice contract.""" + for axis, stride in enumerate(value.strides): + invalid = stride % value.itemsize != 0 or (value.size > 0 and value.shape[axis] > 1 and stride <= 0) + if axis: + invalid |= ( + value.size > 0 + and value.shape[axis - 1] > 0 + and (stride < value.strides[axis - 1] * value.shape[axis - 1]) + ) + if invalid: + raise TypeError("NumPy array actual has incompatible layout; expected ordering (F)") + + def _normal_array_actual_abi_facts( value: Any, actual: Any, @@ -1151,11 +1205,11 @@ def _validate_ndarray_array_actual( require_contiguous: bool = False, ) -> None: _validate_ndarray_expected_rank(value, expected_rank) + _validate_ndarray_native_byte_order(value, require_native_byte_order) _validate_ndarray_expected_dtype(value, expected_dtype) _validate_ndarray_expected_shape(tuple(int(dimension) for dimension in value.shape), expected_shape) _validate_ndarray_expected_layout(value, expected_layout) _validate_ndarray_writeable(value, require_writeable) - _validate_ndarray_native_byte_order(value, require_native_byte_order) _validate_ndarray_aligned(value, require_aligned) _validate_ndarray_contiguous(value, require_contiguous) @@ -1201,7 +1255,9 @@ def _validate_ndarray_expected_shape( raise TypeError(f"NumPy array shape rank {len(shape)} does not match expected shape rank {len(expected)}") for axis, (actual, wanted) in enumerate(zip(shape, expected, strict=True)): if wanted is not None and actual != wanted: - raise TypeError(f"NumPy array shape {shape!r} does not match expected shape {expected!r} at axis {axis}") + raise TypeError( + f"NumPy array has incompatible shape at axis {axis}: received {shape!r}, expected {expected!r}" + ) def _validate_ndarray_expected_layout(value: np.ndarray, expected_layout: str | None) -> None: diff --git a/x2py/semantics/ir2ast.py b/x2py/semantics/ir2ast.py deleted file mode 100644 index 7034562e3..000000000 --- a/x2py/semantics/ir2ast.py +++ /dev/null @@ -1,1723 +0,0 @@ -"""Convert x2py semantic IR nodes into codegen AST nodes.""" - -from __future__ import annotations - -import ast -from dataclasses import replace -from itertools import product -import numpy as np -import re - -from x2py import SEMANTIC_DTYPE_TO_NUMPY_DTYPE -from x2py.codegen.models.core import ( - Add, - AsName, - ClassDef, - Div, - FunctionDef, - FunctionAddress, - FunctionDefArgument, - FunctionDefResult, - FunctionOverloadSet, - Import, - Minus, - Module, - Mul, - UnarySub, - Variable, -) -from x2py.codegen.models.datatypes import ( - CharType, - DataTypeFactory, - FinalType, - NIL, - NumpyNDArrayType, - StringType, - convert_to_literal, - original_type_to_x2py_type, -) -from x2py.semantics.metadata import ( - ADDRESS_ROLE_METADATA, - ADDRESS_ROLE_PROJECTION, - ADDRESS_ROLE_RAW, - BIND_TARGET_METADATA, - NATIVE_PROJECTION_METADATA, - PROJECTED_OUTPUT_METADATA, - SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA, -) -from x2py.semantics import models -from x2py.semantics.models import ( - FORTRAN_GENERIC_NAME_METADATA, - OVERLOAD_KIND_METADATA, - PYTHON_BOUND_POSITION_METADATA, -) -from x2py.semantics.native_array_handles import array_interop_policy, native_array_descriptor_kind -from x2py.semantics.native_contract import NATIVE_CONTRACT_PREPARED_METADATA -from x2py.semantics.pyi_metadata import PYI_LOADED_METADATA -from x2py.semantics.wrapper_policy import CallbackHandoffPolicy, FunctionWrapperPolicy, NativeStatusErrorPolicy -from x2py.utilities.visitor import ClassVisitor - - -_SEMANTIC_ORDER_TO_NUMPY_ORDER = { - "ORDER_C": "C", - "ORDER_F": "F", -} -_MAX_SUPPORTED_ARRAY_RANK = 15 -_ISO_C_KIND_TOKENS = frozenset( - { - "c_bool", - "c_char", - "c_double", - "c_double_complex", - "c_float", - "c_float_complex", - "c_int", - "c_int16_t", - "c_int32_t", - "c_int64_t", - "c_int8_t", - "c_long_double", - "c_long_double_complex", - "c_long_long", - "c_short", - "c_signed_char", - "c_size_t", - } -) -_POLYMORPHIC_DISPATCH_VARIANT_METADATA = "fortran_polymorphic_dispatch_variant" - - -def _numpy_type(dtype: str): - return getattr(np, dtype.removeprefix("numpy.")) - - -def _codegen_type(dtype: str, custom_types: dict[str, object] | None = None): - if custom_types and dtype in custom_types: - return custom_types[dtype] - if dtype == "String": - return StringType() - numpy_type = _numpy_type(SEMANTIC_DTYPE_TO_NUMPY_DTYPE[dtype]) - return original_type_to_x2py_type[numpy_type] - - -def _is_constant(semantic_type: models.SemanticType) -> bool: - return any(constraint.name == "Constant" for constraint in semantic_type.constraints) - - -def _string_shape(semantic_type: models.SemanticType): - length = semantic_type.metadata.get("fortran_character_length") - if isinstance(length, str) and length.isdigit(): - return (convert_to_literal(int(length)),) - return (None,) - - -def _fortran_character_length(semantic_type: models.SemanticType): - """Return codegen metadata for the native Fortran character element length.""" - length = semantic_type.metadata.get("fortran_character_length") - if isinstance(length, str) and length.isdigit(): - return convert_to_literal(int(length)) - return length - - -def _array_contract( - semantic_type: models.SemanticType, -) -> models.SemanticArrayContract | None: - if semantic_type.storage is None: - return None - return semantic_type.storage.array - - -def _numpy_array_order(semantic_type: models.SemanticType, rank: int) -> str | None: - if rank <= 1: - return None - contract = _array_contract(semantic_type) - order = contract.order if contract is not None else None - return _SEMANTIC_ORDER_TO_NUMPY_ORDER.get(order, "C") - - -def _array_allows_strides(semantic_type: models.SemanticType) -> bool: - contract = _array_contract(semantic_type) - return contract is None or contract.contiguous is not True - - -def _codegen_dimension_expression(text: str, scope): - try: - parsed = ast.parse(text, mode="eval").body - except SyntaxError: - return None - return _codegen_expression_node(parsed, scope) - - -def _codegen_expression_node(node: ast.AST, scope): - if isinstance(node, ast.Constant) and isinstance(node.value, int): - return convert_to_literal(node.value) - if isinstance(node, ast.Name): - return _codegen_shape_symbol(scope.find(node.id, "variables")) - if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub): - operand = _codegen_expression_node(node.operand, scope) - return None if operand is None else UnarySub(operand) - if isinstance(node, ast.BinOp): - left = _codegen_expression_node(node.left, scope) - right = _codegen_expression_node(node.right, scope) - if left is None or right is None: - return None - operators = { - ast.Add: Add, - ast.Sub: Minus, - ast.Mult: Mul, - ast.Div: Div, - } - for ast_op, codegen_op in operators.items(): - if isinstance(node.op, ast_op): - return codegen_op(left, right) - return None - - -def _codegen_shape_symbol(symbol): - if isinstance(symbol, Variable) and symbol.rank == 0: - return symbol.clone( - str(symbol.name), - new_class=Variable, - is_optional=False, - memory_handling="stack", - ) - return symbol - - -def _codegen_array_shape(semantic_type: models.SemanticType, scope) -> tuple[object | None, ...] | None: - if semantic_type.rank <= 0: - return None - shape = list(semantic_type.shape) - if not shape: - contract = _array_contract(semantic_type) - shape = list(contract.shape if contract is not None and contract.shape else []) - if not shape: - return None - - result = [] - for dimension in shape: - text = str(dimension).strip() - if text in {"", ":", "*"} or "Strided" in text: - result.append(None) - elif text.isdigit(): - result.append(convert_to_literal(int(text))) - else: - result.append(_codegen_dimension_expression(text, scope)) - return tuple(result) - - -def _class_type(semantic_class: models.SemanticClass): - return DataTypeFactory( - semantic_class.native_name or semantic_class.name, - semantic_class.name, - )() - - -def _iter_semantic_classes(classes: list[models.SemanticClass]): - for semantic_class in classes: - yield semantic_class - yield from _iter_semantic_classes(semantic_class.classes) - - -def _semantic_class_lookup(classes: list[models.SemanticClass]) -> dict[str, models.SemanticClass]: - return {semantic_class.name: semantic_class for semantic_class in _iter_semantic_classes(classes)} - - -def _semantic_class_order(classes: list[models.SemanticClass]) -> dict[str, int]: - return {semantic_class.name: index for index, semantic_class in enumerate(_iter_semantic_classes(classes))} - - -def _semantic_class_descendants(classes: list[models.SemanticClass]) -> dict[str, tuple[str, ...]]: - lookup = _semantic_class_lookup(classes) - direct: dict[str, list[str]] = {name: [] for name in lookup} - for semantic_class in lookup.values(): - for base_name in semantic_class.base_classes: - if base_name in direct: - direct[base_name].append(semantic_class.name) - - def collect(base_name: str) -> tuple[str, ...]: - names = [] - for child_name in direct.get(base_name, ()): - names.extend(collect(child_name)) - names.append(child_name) - return tuple(dict.fromkeys(names)) - - return {base_name: collect(base_name) for base_name in direct} - - -def _missing_completed_policy(owner: str) -> ValueError: - return ValueError( - f"{owner} is missing completed ownership policy; run complete_semantic_policies before ir2ast lowering" - ) - - -def _variable_ownership_decision(variable: models.SemanticVariable): - decision = variable.metadata.get(models.RESOLVED_OWNERSHIP_POLICY_METADATA) - if decision is None: - raise _missing_completed_policy(f"Variable {variable.name!r}") - return decision - - -def _function_return_ownership_decision(function: models.SemanticFunction): - decision = function.metadata.get(models.RESOLVED_RETURN_OWNERSHIP_POLICY_METADATA) - if decision is None: - raise _missing_completed_policy(f"Function {function.name!r} result") - return decision - - -def _type_ownership_decision(owner: str, semantic_type: models.SemanticType): - decision = semantic_type.metadata.get(models.RESOLVED_OWNERSHIP_POLICY_METADATA) - if decision is None: - raise _missing_completed_policy(owner) - return decision - - -def _missing_completed_native_array_handle_policy(owner: str) -> ValueError: - return ValueError( - f"{owner} is missing completed native-array-handle policy; " - "run complete_semantic_policies before ir2ast lowering" - ) - - -def _native_array_handle_policy( - owner: str, - semantic_type: models.SemanticType | None, - metadata: dict, -): - if native_array_descriptor_kind(semantic_type) is None: - return None - policy = metadata.get(models.RESOLVED_NATIVE_ARRAY_HANDLE_POLICY_METADATA) - if policy is None: - raise _missing_completed_native_array_handle_policy(owner) - if policy.is_blocked: - raise ValueError(f"{owner} cannot be wrapped safely: {policy.blocker or policy.handle_kind}") - return policy - - -def _native_array_variable_policy(variable: models.SemanticVariable): - return _native_array_handle_policy( - f"Variable {variable.name!r}", - variable.semantic_type, - variable.metadata, - ) - - -def _native_array_function_result_policy(function: models.SemanticFunction): - return _native_array_handle_policy( - f"Function {function.name!r} result", - function.return_type, - function.metadata, - ) - - -def _passes_by_value(node: models.SemanticVariable) -> bool: - return bool( - getattr(node, "origin", None) is not None - and isinstance(node.origin.metadata, dict) - and node.origin.metadata.get("value") - ) - - -def _passed_object_position(node: models.SemanticFunction) -> int | None: - if node.name == "__init__" and node.metadata.get(BIND_TARGET_METADATA): - return None - overload_kind = node.metadata.get(OVERLOAD_KIND_METADATA) - if overload_kind in {"generic", "assignment", "named_operator", "comparison"}: - position = node.metadata.get(PYTHON_BOUND_POSITION_METADATA) - return position if isinstance(position, int) else None - if not isinstance(node, models.SemanticMethod) or node.is_static: - return None - return node.passed_object_position if node.passed_object_position is not None else 0 - - -def _codegen_function_arguments(declarations: list[Variable], passed_object_position: int | None): - native_args = [ - FunctionDefArgument( - item, - value=NIL if item.is_optional else None, - bound_argument=index == passed_object_position, - bound_argument_position=index if index == passed_object_position else None, - ) - for index, item in enumerate(declarations) - ] - if passed_object_position is None: - return native_args - return [ - native_args[passed_object_position], - *native_args[:passed_object_position], - *native_args[passed_object_position + 1 :], - ] - - -def _pyi_bound_constructor_self( - node: models.SemanticFunction, - cls_base: ClassDef | None, - func_scope, -) -> Variable | None: - if cls_base is None or node.name != "__init__" or not node.metadata.get(BIND_TARGET_METADATA): - return None - self_policy = cls_base.decorators[models.RESOLVED_CLASS_SELF_POLICY_METADATA] - self_var = Variable( - cls_base.class_type, - func_scope.get_new_name("self"), - cls_base=cls_base, - memory_handling=self_policy.boundary_storage_mode.value, - ownership_decision=self_policy, - ) - func_scope.insert_variable(self_var) - return self_var - - -def _raise_for_unresolved_generic_targets(node: models.SemanticModule | models.SemanticClass) -> None: - blockers = node.metadata.get("readiness_blockers", ()) - for blocker in blockers: - if blocker.get("code") != "fortran_generic_target_unresolved": - continue - item = next(iter(blocker.get("items", ())), {}) - generic = item.get("generic", "") - missing = item.get("missing_targets", ()) - if missing: - targets = ", ".join(str(target) for target in missing) - raise ValueError(f"Generic interface {generic!r} references missing specific procedure(s): {targets}") - raise ValueError(f"Generic interface {generic!r} does not declare any specific procedures") - - -def _raise_for_unsupported_constructor_overloads(node: models.SemanticClass) -> None: - if any(overload_set.name == "__init__" for overload_set in node.overload_sets): - raise ValueError( - "Constructor overload dispatch is not mapped to Python tp_init yet; " - "use the generated field constructor until overloaded constructor lowering is implemented." - ) - - -def _raise_for_unsupported_fortran_module_features(node: models.SemanticModule) -> None: - owners = [node, *node.variables, *node.functions] - blocking_codes = { - "fortran_generic_constructor_unsupported", - models.MODULE_VARIABLE_INITIALIZER_UNSUPPORTED_BLOCKER, - } - for owner in owners: - for blocker in owner.metadata.get("readiness_blockers", ()): - if blocker.get("code") in blocking_codes: - raise ValueError(str(blocker.get("message") or "Unsupported Fortran wrapper feature.")) - - -def _is_allocatable_array(semantic_type: models.SemanticType | None) -> bool: - return bool( - semantic_type is not None - and semantic_type.storage is not None - and semantic_type.storage.array is not None - and semantic_type.storage.array.allocatable - ) - - -def _is_allocatable_scalar(semantic_type: models.SemanticType | None) -> bool: - return bool( - semantic_type is not None and semantic_type.rank == 0 and semantic_type.metadata.get("fortran_allocatable") - ) - - -def _is_pointer_array(semantic_type: models.SemanticType | None) -> bool: - return bool( - semantic_type is not None - and semantic_type.storage is not None - and semantic_type.storage.array is not None - and semantic_type.storage.array.pointer - ) - - -def _is_pointer(semantic_type: models.SemanticType | None) -> bool: - if semantic_type is None: - return False - storage = semantic_type.storage - return bool( - semantic_type.metadata.get("fortran_pointer") - or (storage is not None and storage.array is not None and storage.array.pointer) - ) - - -def _argument_uses_writable_storage(argument: models.SemanticArgument) -> bool: - storage = argument.semantic_type.storage - return bool( - argument.semantic_type.ownership.mutable - or (storage is not None and (storage.mutable or not storage.read_only)) - or argument.metadata.get(PROJECTED_OUTPUT_METADATA) - ) - - -def _array_contract_category(semantic_type: models.SemanticType | None) -> str | None: - contract = _array_contract(semantic_type) if semantic_type is not None else None - return None if contract is None else contract.category - - -def _is_assumed_rank(semantic_type: models.SemanticType | None) -> bool: - return _array_contract_category(semantic_type) == "assumed_rank" - - -def _is_assumed_type(semantic_type: models.SemanticType | None) -> bool: - if semantic_type is None: - return False - source_type = (semantic_type.origin.source_type or "").casefold().replace(" ", "") - return "type(*)" in source_type or "class(*)" in source_type - - -def _is_fortran_polymorphic(semantic_type: models.SemanticType | None) -> bool: - return bool( - semantic_type is not None - and semantic_type.metadata.get("fortran_polymorphic") - and not _is_assumed_type(semantic_type) - ) - - -def _is_supported_passed_object_polymorphic_arg( - node: models.SemanticFunction, - argument: models.SemanticArgument, - *, - cls_base: ClassDef | None = None, - passed_object_position: int | None = None, - argument_position: int | None = None, -) -> bool: - if ( - isinstance(node, models.SemanticMethod) - and not node.is_static - and str(node.passed_object_name) == str(argument.name) - ): - return True - if cls_base is not None and passed_object_position is not None and argument_position == passed_object_position: - return True - python_bound_position = node.metadata.get(PYTHON_BOUND_POSITION_METADATA) - if python_bound_position is not None and argument_position == int(python_bound_position): - return True - return bool( - node.metadata.get("fortran_type_bound_target") - and str(node.metadata.get("fortran_passed_object_name")) == str(argument.name) - ) - - -def _is_scalar_polymorphic_input_dispatch_arg( - argument: models.SemanticArgument, - class_lookup: dict[str, models.SemanticClass], -) -> bool: - semantic_type = argument.semantic_type - if not _is_fortran_polymorphic(semantic_type): - return False - if semantic_type.rank != 0 or _argument_uses_writable_storage(argument): - return False - if semantic_type.metadata.get("fortran_allocatable"): - return False - if getattr(argument.origin, "metadata", {}).get("pointer"): - return False - return semantic_type.name in class_lookup - - -def _semantic_class_depth( - name: str, - class_lookup: dict[str, models.SemanticClass], - cache: dict[str, int], -) -> int: - if name in cache: - return cache[name] - semantic_class = class_lookup.get(name) - if semantic_class is None or not semantic_class.base_classes: - cache[name] = 0 - return 0 - cache[name] = 1 + max( - (_semantic_class_depth(base_name, class_lookup, cache) for base_name in semantic_class.base_classes), - default=0, - ) - return cache[name] - - -def _polymorphic_dispatch_class_names( - semantic_type: models.SemanticType, - class_lookup: dict[str, models.SemanticClass], - class_descendants: dict[str, tuple[str, ...]], - class_order: dict[str, int], -) -> tuple[str, ...]: - base_name = semantic_type.name - if base_name not in class_lookup: - return () - depth_cache: dict[str, int] = {} - descendants = sorted( - class_descendants.get(base_name, ()), - key=lambda name: ( - -_semantic_class_depth(name, class_lookup, depth_cache), - class_order.get(name, 0), - ), - ) - return (*descendants, base_name) - - -def _polymorphic_dispatch_options( - node: models.SemanticFunction, - *, - cls_base: ClassDef | None, - passed_object_position: int | None, - class_lookup: dict[str, models.SemanticClass], - class_descendants: dict[str, tuple[str, ...]], - class_order: dict[str, int], -) -> tuple[tuple[int, tuple[str, ...]], ...]: - if node.metadata.get(_POLYMORPHIC_DISPATCH_VARIANT_METADATA): - return () - - options = [] - for index, argument in enumerate(node.arguments): - if not _is_fortran_polymorphic(argument.semantic_type): - continue - if _is_supported_passed_object_polymorphic_arg( - node, - argument, - cls_base=cls_base, - passed_object_position=passed_object_position, - argument_position=index, - ): - continue - if not _is_scalar_polymorphic_input_dispatch_arg(argument, class_lookup): - continue - class_names = _polymorphic_dispatch_class_names( - argument.semantic_type, - class_lookup, - class_descendants, - class_order, - ) - if class_names: - options.append((index, class_names)) - return tuple(options) - - -def _dispatch_argument_for_class(argument: models.SemanticArgument, class_name: str) -> models.SemanticArgument: - type_metadata = dict(argument.semantic_type.metadata) - type_metadata.pop("fortran_polymorphic", None) - type_metadata["fortran_polymorphic_dispatch_base"] = argument.semantic_type.name - type_metadata["fortran_polymorphic_dispatch_type"] = class_name - semantic_type = replace( - argument.semantic_type, - name=class_name, - dtype=class_name, - metadata=type_metadata, - ) - return models.SemanticArgument( - argument.name, - semantic_type, - optional=argument.optional, - visibility=argument.visibility, - default_value=argument.default_value, - metadata=dict(argument.metadata), - origin=argument.origin, - ) - - -def _polymorphic_dispatch_variants( - node: models.SemanticFunction, - dispatch_options: tuple[tuple[int, tuple[str, ...]], ...], -) -> tuple[models.SemanticFunction, ...]: - positions = tuple(position for position, _ in dispatch_options) - class_options = tuple(class_names for _, class_names in dispatch_options) - variants = [] - for selected_classes in product(*class_options): - arguments = list(node.arguments) - for position, class_name in zip(positions, selected_classes, strict=True): - arguments[position] = _dispatch_argument_for_class(arguments[position], class_name) - metadata = dict(node.metadata) - metadata[_POLYMORPHIC_DISPATCH_VARIANT_METADATA] = True - variants.append(replace(node, arguments=arguments, metadata=metadata)) - return tuple(variants) - - -def _is_derived_type_array(semantic_type: models.SemanticType | None) -> bool: - return bool( - semantic_type is not None - and semantic_type.rank > 0 - and semantic_type.name not in SEMANTIC_DTYPE_TO_NUMPY_DTYPE - and semantic_type.name != "String" - ) - - -def _raise_for_unsupported_array_contracts_in_type( - owner: str, - semantic_type: models.SemanticType | None, -) -> None: - if semantic_type is None or semantic_type.rank <= 0: - return - if semantic_type.rank > _MAX_SUPPORTED_ARRAY_RANK: - raise ValueError( - f"{owner} has rank {semantic_type.rank}, but wrapper generation supports ranks " - f"1 through {_MAX_SUPPORTED_ARRAY_RANK}" - ) - if _is_assumed_type(semantic_type): - raise ValueError( - f"{owner} uses assumed-type type(*), which needs an explicit dtype and descriptor policy " - "before wrapper generation" - ) - if _is_derived_type_array(semantic_type): - raise ValueError( - f"{owner} is an array of derived type values, which needs explicit layout and ownership policy" - ) - - -def _raise_for_unsupported_array_contracts_in_function(node: models.SemanticFunction) -> None: - for argument in node.arguments: - _raise_for_unsupported_array_contracts_in_type( - f"Function {node.name!r} argument {argument.name!r}", - argument.semantic_type, - ) - _raise_for_unsupported_array_contracts_in_type(f"Function {node.name!r} result", node.return_type) - - -def _raise_for_unsupported_array_contracts_in_class(node: models.SemanticClass) -> None: - for field in node.fields: - _raise_for_unsupported_array_contracts_in_type( - f"Class {node.name!r} field {field.name!r}", - field.semantic_type, - ) - for method in node.methods: - _raise_for_unsupported_array_contracts_in_function(method) - - -def _raise_for_unsupported_array_contracts(node: models.SemanticModule) -> None: - for variable in node.variables: - _raise_for_unsupported_array_contracts_in_type( - f"Module variable {variable.name!r}", - variable.semantic_type, - ) - for function in node.functions: - _raise_for_unsupported_array_contracts_in_function(function) - for overload_set in node.overload_sets: - for procedure in overload_set.procedures: - _raise_for_unsupported_array_contracts_in_function(procedure) - for semantic_class in node.classes: - _raise_for_unsupported_array_contracts_in_class(semantic_class) - - -def _raise_for_blocked_non_derived_pointer_outputs(node: models.SemanticFunction) -> None: - """Leave completed scalar-derived pointer policy to wrapper planning.""" - for argument in node.arguments: - semantic_type = argument.semantic_type - scalar_derived = ( - semantic_type.rank == 0 - and semantic_type.name not in SEMANTIC_DTYPE_TO_NUMPY_DTYPE - and semantic_type.name != "String" - ) - decision = _variable_ownership_decision(argument) - if _is_pointer(semantic_type) and decision.is_blocked and not scalar_derived: - raise ValueError( - f"Function {node.name!r} has pointer argument {argument.name!r}, " - f"which cannot be wrapped safely: {decision.blocker or decision.reason}" - ) - - -def _raise_for_blocked_ownership_policy( - owner: str, - decision, -) -> None: - if decision.is_blocked: - raise ValueError(f"{owner} cannot be wrapped safely: {decision.blocker or decision.reason}") - - -def _raise_for_unsupported_assumed_type_contracts(node: models.SemanticFunction) -> None: - for argument in node.arguments: - if _is_assumed_type(argument.semantic_type): - raise ValueError( - f"Function {node.name!r} has assumed-type argument {argument.name!r}, " - "which needs an explicit dtype and descriptor policy before wrapper generation" - ) - if _is_assumed_type(node.return_type): - raise ValueError( - f"Function {node.name!r} has an assumed-type result, " - "which needs an explicit dtype and descriptor policy before wrapper generation" - ) - - -def _raise_for_unsupported_polymorphic_contracts( - node: models.SemanticFunction, - *, - cls_base: ClassDef | None = None, - passed_object_position: int | None = None, - dispatch_positions: set[int] | None = None, -) -> None: - supported_dispatch_positions = set() if dispatch_positions is None else dispatch_positions - for index, argument in enumerate(node.arguments): - if not _is_fortran_polymorphic(argument.semantic_type): - continue - if index in supported_dispatch_positions: - continue - if not _is_supported_passed_object_polymorphic_arg( - node, - argument, - cls_base=cls_base, - passed_object_position=passed_object_position, - argument_position=index, - ): - raise ValueError( - f"Function {node.name!r} has polymorphic argument {argument.name!r}, " - "which needs explicit dynamic-type and dispatch policy" - ) - if _is_fortran_polymorphic(node.return_type): - raise ValueError( - f"Function {node.name!r} has a polymorphic result, " - "which needs explicit dynamic-type, allocation, and ownership policy" - ) - - -def _is_bind_c_derived_type( - semantic_type: models.SemanticType, - class_lookup: dict[str, models.SemanticClass], -) -> bool: - semantic_class = class_lookup.get(semantic_type.name) - return semantic_class is not None and bool(semantic_class.metadata.get("fortran_bind_c")) - - -def _raise_for_unsupported_bind_c_abi( - node: models.SemanticFunction, - class_lookup: dict[str, models.SemanticClass], -) -> None: - if not node.metadata.get("fortran_bind_c"): - return - for argument in node.arguments: - semantic_type = argument.semantic_type - if semantic_type.rank > 0: - continue - if _is_bind_c_derived_type(semantic_type, class_lookup): - continue - if semantic_type.name in class_lookup: - is_value = bool(getattr(argument.origin, "metadata", {}).get("value")) - transfer = "by-value " if is_value else "" - raise ValueError( - f"Function {node.name!r} has bind(C) {transfer}derived-type argument {argument.name!r} " - "whose type is not declared bind(C); aggregate layout is not inferred" - ) - if not _has_known_iso_c_kind(semantic_type): - raise ValueError( - f"Function {node.name!r} has bind(C) scalar argument {argument.name!r} " - "without a supported ISO C binding kind" - ) - if node.return_type is not None and node.return_type.rank == 0: - if _is_bind_c_derived_type(node.return_type, class_lookup): - return - if node.return_type.name in class_lookup: - raise ValueError( - f"Function {node.name!r} has a bind(C) derived-type result whose type is not declared bind(C); " - "aggregate layout is not inferred" - ) - if not _has_known_iso_c_kind(node.return_type): - raise ValueError( - f"Function {node.name!r} has a bind(C) scalar result without a supported ISO C binding kind" - ) - - -def _raise_for_blocked_ownership_contracts_in_function(node: models.SemanticFunction) -> None: - for argument in node.arguments: - _raise_for_blocked_ownership_policy( - f"Function {node.name!r} argument {argument.name!r}", - _variable_ownership_decision(argument), - ) - if node.return_type is not None: - _raise_for_blocked_ownership_policy( - f"Function {node.name!r} result", - _function_return_ownership_decision(node), - ) - - -def _raise_for_native_array_handle_policies_in_function(node: models.SemanticFunction) -> None: - for argument in node.arguments: - _native_array_variable_policy(argument) - if node.return_type is not None: - _native_array_function_result_policy(node) - - -def _raise_for_blocked_ownership_contracts_in_class(node: models.SemanticClass) -> None: - for field in node.fields: - _raise_for_blocked_ownership_policy( - f"Class {node.name!r} field {field.name!r}", - _variable_ownership_decision(field), - ) - - -def _raise_for_native_array_handle_policies_in_class(node: models.SemanticClass) -> None: - for field in node.fields: - _native_array_variable_policy(field) - for nested in node.classes: - _raise_for_native_array_handle_policies_in_class(nested) - - -def _raise_for_blocked_ownership_contracts(node: models.SemanticModule) -> None: - for variable in node.variables: - _raise_for_blocked_ownership_policy( - f"Module variable {variable.name!r}", - _variable_ownership_decision(variable), - ) - for semantic_class in node.classes: - _raise_for_blocked_ownership_contracts_in_class(semantic_class) - - -def _raise_for_native_array_handle_policies(node: models.SemanticModule) -> None: - for variable in node.variables: - _native_array_variable_policy(variable) - for semantic_class in node.classes: - _raise_for_native_array_handle_policies_in_class(semantic_class) - - -def _is_public(node) -> bool: - return getattr(node, "visibility", "public") != "private" - - -def _references_private_type(semantic_type: models.SemanticType | None, private_type_names: set[str]) -> bool: - return bool(semantic_type is not None and semantic_type.name in private_type_names) - - -def _raise_if_private_type_exposed( - owner: str, - semantic_type: models.SemanticType | None, - private_type_names: set[str], -) -> None: - if _references_private_type(semantic_type, private_type_names): - raise ValueError(f"{owner} exposes private derived type {semantic_type.name!r} in the Python wrapper API") - - -def _raise_for_private_type_exposure_in_function( - node: models.SemanticFunction, - private_type_names: set[str], -) -> None: - if not _is_public(node): - return - for argument in node.arguments: - _raise_if_private_type_exposed( - f"Public function {node.name!r} argument {argument.name!r}", - argument.semantic_type, - private_type_names, - ) - _raise_if_private_type_exposed(f"Public function {node.name!r} result", node.return_type, private_type_names) - - -def _raise_for_private_type_exposure_in_class( - node: models.SemanticClass, - private_type_names: set[str], -) -> None: - if not _is_public(node): - return - for base_name in node.base_classes: - if base_name in private_type_names: - raise ValueError(f"Public type {node.name!r} extends private derived type {base_name!r}") - for field in node.fields: - if _is_public(field): - _raise_if_private_type_exposed( - f"Public type {node.name!r} field {field.name!r}", - field.semantic_type, - private_type_names, - ) - for method in node.methods: - _raise_for_private_type_exposure_in_function(method, private_type_names) - for overload_set in node.overload_sets: - if _is_public(overload_set): - for procedure in overload_set.procedures: - _raise_for_private_type_exposure_in_function(procedure, private_type_names) - - -def _raise_for_private_type_exposure(node: models.SemanticModule) -> None: - private_type_names = { - semantic_class.name for semantic_class in _iter_semantic_classes(node.classes) if not _is_public(semantic_class) - } - if not private_type_names: - return - for variable in node.variables: - if _is_public(variable): - _raise_if_private_type_exposed( - f"Public module variable {variable.name!r}", - variable.semantic_type, - private_type_names, - ) - for function in node.functions: - _raise_for_private_type_exposure_in_function(function, private_type_names) - for overload_set in node.overload_sets: - if _is_public(overload_set): - for procedure in overload_set.procedures: - _raise_for_private_type_exposure_in_function(procedure, private_type_names) - for semantic_class in node.classes: - _raise_for_private_type_exposure_in_class(semantic_class, private_type_names) - - -def _has_known_iso_c_kind(semantic_type: models.SemanticType) -> bool: - source_type = (semantic_type.origin.source_type or "").casefold() - return any(token in source_type for token in _ISO_C_KIND_TOKENS) - - -def _semantic_python_exports(node, converted, scope) -> tuple[tuple[tuple[str, ...], str], ...]: - if isinstance(node, models.ProcedureOverloadSet): - metadata = node.procedures[0].metadata if node.procedures else {} - else: - metadata = node.metadata - exports = metadata.get(models.PYTHON_EXPORTS_METADATA, ()) - if not exports: - return () - public_name = str(scope.get_python_name(converted.name)) if any(item["name"] is None for item in exports) else "" - return tuple( - (tuple(item["namespace"]), public_name if item["name"] is None else str(item["name"])) for item in exports - ) - - -def _pyi_native_import( - node, - converted, - *, - native_name_filter=None, - preserve_native_alias: bool = False, -) -> Import | None: - if getattr(node, "visibility", "public") == "private": - return None - if isinstance(node, models.ProcedureOverloadSet): - if not node.procedures: - return None - origin = node.procedures[0].origin - native_names = _pyi_overload_native_names(node) - if native_name_filter is not None: - native_names = tuple(name for name in native_names if native_name_filter(name)) - else: - origin = node.origin - native_name = getattr(node, "native_name", None) or node.name - native_names = (str(native_name),) - if origin.native_scope is None: - return None - if not native_names: - return None - targets = tuple( - AsName( - converted, - str(native_name) if preserve_native_alias else str(converted.name), - source_name=str(native_name), - ) - for native_name in native_names - ) - return Import(str(origin.native_scope), target=targets) - - -def _pyi_overload_native_names(node: models.ProcedureOverloadSet) -> tuple[str, ...]: - names = {str(procedure.metadata.get(FORTRAN_GENERIC_NAME_METADATA, node.name)) for procedure in node.procedures} - return tuple(sorted(names)) - - -def _pyi_class_overload_native_imports(semantic_class: models.SemanticClass, converted: ClassDef) -> list[Import]: - imports = [] - converted_by_name = {str(overload_set.name): overload_set for overload_set in converted.overload_sets} - for overload_set in semantic_class.overload_sets: - converted_overload = converted_by_name.get(str(overload_set.name)) - if converted_overload is None: - continue - native_import = _pyi_native_import( - overload_set, - converted_overload, - native_name_filter=_is_importable_class_generic, - preserve_native_alias=True, - ) - if native_import is not None: - imports.append(native_import) - return imports - - -def _is_importable_class_generic(native_name: str) -> bool: - compact = re.sub(r"\s+", "", native_name).casefold() - return compact.startswith("operator(") or compact == "assignment(=)" - - -def _semantic_function_decorators(node): - decorators = {} - if node.projection: - decorators[NATIVE_PROJECTION_METADATA] = True - if node.metadata.get(models.RUNTIME_HOLD_GIL_METADATA): - decorators[models.RUNTIME_HOLD_GIL_METADATA] = True - raw_status_policy = node.metadata.get(models.RUNTIME_STATUS_ERROR_METADATA) - if raw_status_policy is not None: - status_policy = node.metadata.get(models.RESOLVED_RUNTIME_STATUS_ERROR_POLICY_METADATA) - if not isinstance(status_policy, NativeStatusErrorPolicy): - raise ValueError( - f"Function {node.name!r} is missing completed native status error policy; " - "run complete_semantic_policies before ir2ast lowering" - ) - decorators[models.RUNTIME_STATUS_ERROR_METADATA] = status_policy - return decorators - - -def _semantic_type_bound_name(node: models.SemanticFunction, cls_base: ClassDef | None) -> str | None: - """Return the completed native type-bound binding selected before lowering.""" - if cls_base is None: - return None - policy = node.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) - if not isinstance(policy, FunctionWrapperPolicy): - raise ValueError(f"Class method {node.name!r} has no completed wrapper policy") - return policy.class_call.type_bound_name if policy.class_call is not None else None - - -def _semantic_variable_type_and_shape(semantic_type, scope, custom_types): - rank = semantic_type.rank - dtype = _codegen_type(semantic_type.dtype, custom_types) - if _is_constant(semantic_type): - dtype = FinalType.get_new(dtype) - if rank > 0: - if isinstance(dtype, StringType): - dtype = CharType() - dtype = NumpyNDArrayType.get_new( - dtype, - rank, - order=_numpy_array_order(semantic_type, rank), - allows_strides=_array_allows_strides(semantic_type), - ) - shape = ( - _string_shape(semantic_type) if isinstance(dtype, StringType) else _codegen_array_shape(semantic_type, scope) - ) - return dtype, shape - - -def _fortran_array_category_and_source_shape(semantic_type): - storage = semantic_type.storage - if storage is not None and storage.kind == "address": - role = storage.metadata.get(ADDRESS_ROLE_METADATA) - if role == ADDRESS_ROLE_PROJECTION: - return "address_projection", () - if role == ADDRESS_ROLE_RAW: - return "raw_address", () - contract = _array_contract(semantic_type) - if contract is None: - return None, () - return contract.category, tuple(contract.source_shape) - - -def _semantic_variable_name(node, scope): - try: - return scope.get_expected_name(node.name) - except RuntimeError: - is_module_mutable = getattr(scope, "_scope_type", None) == "module" and not _is_constant(node.semantic_type) - if isinstance(node, models.SemanticArgument): - object_type = "argument" - elif isinstance(node, models.SemanticField): - object_type = "field" - else: - object_type = "variable" - if _is_public(node) and not is_module_mutable: - return scope.get_new_public_name( - node.name, - object_type=object_type, - owner=f"{object_type} {node.name}", - ) - return scope.get_new_name(node.name) - - -_VISITOR_DEFAULT = object() - - -class _SemanticIrToCodegenAstVisitor(ClassVisitor): - """Lower semantic model nodes through the shared class visitor protocol.""" - - def __init__( - self, - scope, - legacy: bool, - *, - custom_types: dict[str, object] | None, - cls_base: ClassDef | None, - class_lookup: dict[str, models.SemanticClass] | None, - class_descendants: dict[str, tuple[str, ...]] | None, - class_order: dict[str, int] | None, - enable_polymorphic_dispatch: bool, - ): - self.scope = scope - self.legacy = legacy - self.custom_types = custom_types - self.cls_base = cls_base - self.class_lookup = class_lookup - self.class_descendants = class_descendants - self.class_order = class_order - self.enable_polymorphic_dispatch = enable_polymorphic_dispatch - - @staticmethod - def _visit_not_supported(node): - """Reject semantic nodes that have no lowering visitor.""" - raise NotImplementedError(type(node)) - - def _lower_child( - self, - node, - *, - scope=_VISITOR_DEFAULT, - custom_types=_VISITOR_DEFAULT, - cls_base=_VISITOR_DEFAULT, - class_lookup=_VISITOR_DEFAULT, - class_descendants=_VISITOR_DEFAULT, - class_order=_VISITOR_DEFAULT, - enable_polymorphic_dispatch=_VISITOR_DEFAULT, - ): - return type(self)( - self.scope if scope is _VISITOR_DEFAULT else scope, - self.legacy, - custom_types=self.custom_types if custom_types is _VISITOR_DEFAULT else custom_types, - cls_base=self.cls_base if cls_base is _VISITOR_DEFAULT else cls_base, - class_lookup=self.class_lookup if class_lookup is _VISITOR_DEFAULT else class_lookup, - class_descendants=(self.class_descendants if class_descendants is _VISITOR_DEFAULT else class_descendants), - class_order=self.class_order if class_order is _VISITOR_DEFAULT else class_order, - enable_polymorphic_dispatch=( - self.enable_polymorphic_dispatch - if enable_polymorphic_dispatch is _VISITOR_DEFAULT - else enable_polymorphic_dispatch - ), - )._visit(node) - - def _callback_result_variable( - self, - semantic_type: models.SemanticType, - name: str, - scope, - ) -> Variable: - ownership_decision = _type_ownership_decision(f"Callback result {name!r}", semantic_type) - dtype = _codegen_type(semantic_type.dtype, self.custom_types) - if semantic_type.rank > 0: - dtype = NumpyNDArrayType.get_new( - dtype, - semantic_type.rank, - order=_numpy_array_order(semantic_type, semantic_type.rank), - allows_strides=_array_allows_strides(semantic_type), - ) - shape = _codegen_array_shape(semantic_type, scope) if semantic_type.rank > 0 else None - result = Variable( - dtype, - name, - shape=shape, - memory_handling=ownership_decision.storage_mode.value, - ownership_decision=ownership_decision, - ) - scope.insert_variable(result, name=name) - return result - - def _lower_polymorphic_function(self, node, dispatch_options): - name = self.scope.get_new_name(node.name) - variants = _polymorphic_dispatch_variants(node, dispatch_options) - functions = [self._lower_child(variant, enable_polymorphic_dispatch=False) for variant in variants] - native_name = node.native_name or node.name - overload_set = FunctionOverloadSet( - str(name), - functions, - native_name=native_name, - native_names=(native_name,) * len(functions), - ) - self.scope.insert_function(overload_set, name) - return overload_set - - def _semantic_function_result(self, node, func_scope): - if not node.return_type: - return FunctionDefResult(NIL) - return_dtype = _codegen_type(node.return_type.dtype, self.custom_types) - if node.return_type.rank > 0: - if isinstance(return_dtype, StringType): - return_dtype = CharType() - return_dtype = NumpyNDArrayType.get_new( - return_dtype, - node.return_type.rank, - order=_numpy_array_order(node.return_type, node.return_type.rank), - allows_strides=_array_allows_strides(node.return_type), - ) - if isinstance(return_dtype, StringType): - result_shape = _string_shape(node.return_type) - elif node.return_type.rank > 0: - result_shape = _codegen_array_shape(node.return_type, func_scope) - else: - result_shape = None - result_ownership = _function_return_ownership_decision(node) - native_array_handle_policy = _native_array_function_result_policy(node) - array_policy = array_interop_policy( - node.return_type, - owner=f"function {node.name} result", - native_array_handle_policy=native_array_handle_policy, - ) - result_var = Variable( - return_dtype, - node.name, - shape=result_shape, - memory_handling=result_ownership.storage_mode.value, - fortran_character_length=_fortran_character_length(node.return_type), - ownership_decision=result_ownership, - native_array_handle_policy=native_array_handle_policy, - array_interop_policy=array_policy, - ) - func_scope.insert_variable(result_var, name=node.name) - return FunctionDefResult(result_var) - - def _semantic_function_name(self, node, native_name): - if _is_public(node): - return self.scope.get_new_public_name( - native_name, - python_name=node.name, - object_type="function", - owner=f"function {node.name}", - ) - return self.scope.get_new_name(native_name, object_type="function") - - def _populate_codegen_class_methods(self, cls, node, class_scope): - for method in node.methods: - converted_method = self._lower_child( - method, - scope=class_scope, - cls_base=cls, - ) - if isinstance(converted_method, FunctionOverloadSet): - cls.add_new_overload_set(converted_method) - else: - cls.add_new_method(converted_method) - for overload_set in node.overload_sets: - cls.add_new_overload_set( - self._lower_child( - overload_set, - scope=class_scope, - cls_base=cls, - ) - ) - - def _prepare_semantic_module(self, node): - if node.metadata.get(PYI_LOADED_METADATA) and not node.metadata.get(NATIVE_CONTRACT_PREPARED_METADATA): - from .native_contract import prepare_pyi_native_contract - - prepare_pyi_native_contract([node]) - _raise_for_unresolved_generic_targets(node) - _raise_for_unsupported_fortran_module_features(node) - _raise_for_unsupported_array_contracts(node) - _raise_for_blocked_ownership_contracts(node) - _raise_for_native_array_handle_policies(node) - _raise_for_private_type_exposure(node) - custom_types = dict(self.custom_types or {}) - class_lookup = _semantic_class_lookup(node.classes) - class_descendants = _semantic_class_descendants(node.classes) - class_order = _semantic_class_order(node.classes) - for semantic_class in node.classes: - custom_types.setdefault(semantic_class.name, _class_type(semantic_class)) - self.scope.insert_cls_construct(custom_types[semantic_class.name]) - return custom_types, class_lookup, class_descendants, class_order - - def _lower_module_child(self, item, *, custom_types, class_lookup, class_descendants, class_order): - return self._lower_child( - item, - custom_types=custom_types, - class_lookup=class_lookup, - class_descendants=class_descendants, - class_order=class_order, - ) - - def _record_module_conversion_metadata( - self, - item, - converted, - *, - python_exports, - native_imports, - ) -> None: - python_exports[id(converted)] = _semantic_python_exports(item, converted, self.scope) - native_import = _pyi_native_import(item, converted) - if native_import is not None: - native_imports.append(native_import) - - def _lower_module_classes( - self, - node, - *, - custom_types, - class_lookup, - class_descendants, - class_order, - python_exports, - native_imports, - ): - class_items = [item for item in node.classes if _is_public(item)] - classes = [ - self._lower_module_child( - item, - custom_types=custom_types, - class_lookup=class_lookup, - class_descendants=class_descendants, - class_order=class_order, - ) - for item in class_items - ] - for item, converted in zip(class_items, classes, strict=True): - self._record_module_conversion_metadata( - item, - converted, - python_exports=python_exports, - native_imports=native_imports, - ) - native_imports.extend(_pyi_class_overload_native_imports(item, converted)) - return classes - - def _lower_module_functions( - self, - node, - *, - custom_types, - class_lookup, - class_descendants, - class_order, - python_exports, - native_imports, - ): - funcs = [] - generated_overload_sets = [] - for item in node.functions: - converted = self._lower_module_child( - item, - custom_types=custom_types, - class_lookup=class_lookup, - class_descendants=class_descendants, - class_order=class_order, - ) - if isinstance(converted, FunctionOverloadSet): - generated_overload_sets.append(converted) - else: - funcs.append(converted) - self._record_module_conversion_metadata( - item, - converted, - python_exports=python_exports, - native_imports=native_imports, - ) - return funcs, generated_overload_sets - - def _lower_module_overload_sets( - self, - node, - *, - custom_types, - class_lookup, - class_descendants, - class_order, - python_exports, - native_imports, - ): - overload_sets = [ - self._lower_module_child( - item, - custom_types=custom_types, - class_lookup=class_lookup, - class_descendants=class_descendants, - class_order=class_order, - ) - for item in node.overload_sets - ] - for item, converted in zip(node.overload_sets, overload_sets, strict=True): - self._record_module_conversion_metadata( - item, - converted, - python_exports=python_exports, - native_imports=native_imports, - ) - return overload_sets - - def _lower_module_declarations(self, node, *, custom_types, python_exports, native_imports): - declarations = [self._lower_child(item, custom_types=custom_types) for item in node.variables] - for item, converted in zip(node.variables, declarations, strict=True): - self._record_module_conversion_metadata( - item, - converted, - python_exports=python_exports, - native_imports=native_imports, - ) - return declarations - - @staticmethod - def _semantic_module_imports(node, native_imports): - if native_imports: - return native_imports - return [Import(module_name, target=()) for module_name in node.metadata.get("wrapper_native_modules", ())] - - def _visit_SemanticModule(self, node): - custom_types, class_lookup, class_descendants, class_order = self._prepare_semantic_module(node) - python_exports = {} - native_imports = [] - lowering_context = { - "custom_types": custom_types, - "class_lookup": class_lookup, - "class_descendants": class_descendants, - "class_order": class_order, - "python_exports": python_exports, - "native_imports": native_imports, - } - classes = self._lower_module_classes(node, **lowering_context) - funcs, generated_overload_sets = self._lower_module_functions(node, **lowering_context) - overload_sets = self._lower_module_overload_sets(node, **lowering_context) - declarations = self._lower_module_declarations( - node, - custom_types=custom_types, - python_exports=python_exports, - native_imports=native_imports, - ) - name = self.scope.get_new_public_name(node.name, object_type="module", owner=node.name) - explicit_exports = node.metadata.get(models.PYTHON_EXPORTS_PREPARED_METADATA) - return Module( - name, - declarations, - funcs, - overload_sets=[*generated_overload_sets, *overload_sets], - classes=classes, - imports=self._semantic_module_imports(node, native_imports), - scope=self.scope, - python_exports=python_exports if explicit_exports else None, - ) - - def _visit_ProcedureOverloadSet(self, node): - functions = [] - native_names = [] - for procedure in node.procedures: - native_name = str(procedure.metadata.get(FORTRAN_GENERIC_NAME_METADATA, node.name)) - converted = self._lower_child(procedure) - if isinstance(converted, FunctionOverloadSet): - functions.extend(converted.functions) - native_names.extend([native_name] * len(converted.functions)) - else: - functions.append(converted) - native_names.append(native_name) - name = self.scope.get_new_public_name(node.name, object_type="function", owner=f"generic {node.name}") - overload_set = FunctionOverloadSet(str(name), functions, native_names=native_names) - self.scope.insert_function(overload_set, name) - return overload_set - - def _visit_SemanticFunction(self, node): - _raise_for_unsupported_bind_c_abi(node, self.class_lookup or {}) - _raise_for_blocked_ownership_contracts_in_function(node) - _raise_for_blocked_non_derived_pointer_outputs(node) - _raise_for_native_array_handle_policies_in_function(node) - _raise_for_unsupported_assumed_type_contracts(node) - _raise_for_unsupported_array_contracts_in_function(node) - passed_object_position = _passed_object_position(node) - dispatch_options = ( - _polymorphic_dispatch_options( - node, - cls_base=self.cls_base, - passed_object_position=passed_object_position, - class_lookup=self.class_lookup or {}, - class_descendants=self.class_descendants or {}, - class_order=self.class_order or {}, - ) - if self.enable_polymorphic_dispatch - else () - ) - _raise_for_unsupported_polymorphic_contracts( - node, - cls_base=self.cls_base, - passed_object_position=passed_object_position, - dispatch_positions={position for position, _ in dispatch_options}, - ) - if dispatch_options: - return self._lower_polymorphic_function(node, dispatch_options) - func_scope = self.scope.new_child_scope( - name=node.name, - scope_type="function", - public_namespace=self.scope.child_public_namespace("function", node.name), - ) - constructor_self = _pyi_bound_constructor_self(node, self.cls_base, func_scope) - declarations = [constructor_self] if constructor_self is not None else [] - declarations.extend( - self._lower_child( - item, - scope=func_scope, - cls_base=self.cls_base if constructor_self is None and index == passed_object_position else None, - ) - for index, item in enumerate(node.arguments) - ) - if constructor_self is not None: - passed_object_position = 0 - result = self._semantic_function_result(node, func_scope) - native_name = node.native_name or node.name - name = self._semantic_function_name(node, native_name) - func = FunctionDef( - name, - _codegen_function_arguments(declarations, passed_object_position), - [], - result, - scope=func_scope, - decorators=_semantic_function_decorators(node), - is_external=( - self.legacy or (node.origin.source_language == "fortran" and node.origin.native_scope is None) - ), - is_private=node.visibility == "private", - bind_c_external_name=( - str(node.metadata.get("fortran_bind_c_name") or native_name) - if node.metadata.get("fortran_bind_c") - else None - ), - type_bound_name=_semantic_type_bound_name(node, self.cls_base), - ) - self.scope._locals["functions"][name] = func - return func - - def _visit_SemanticClass(self, node): - _raise_for_unresolved_generic_targets(node) - _raise_for_unsupported_constructor_overloads(node) - _raise_for_blocked_ownership_contracts_in_class(node) - _raise_for_native_array_handle_policies_in_class(node) - class_type = (self.custom_types or {}).get(node.name) - if class_type is None: - class_type = _class_type(node) - if self.custom_types is not None: - self.custom_types[node.name] = class_type - self.scope.insert_cls_construct(class_type) - - if _is_public(node): - name = self.scope.get_new_public_name(node.name, object_type="class", owner=f"type {node.name}") - else: - name = self.scope.get_new_name(node.name, object_type="class") - class_scope = self.scope.new_child_scope( - name=str(name), - scope_type="class", - public_namespace=self.scope.child_public_namespace("class", self.scope.get_python_name(name)), - ) - attributes = [ - self._lower_child( - item, - scope=class_scope, - custom_types=self.custom_types, - cls_base=None, - class_lookup=None, - class_descendants=None, - class_order=None, - ) - for item in node.fields - ] - superclasses = tuple( - cls for base_name in node.base_classes if (cls := self.scope.find(base_name, "classes")) is not None - ) - decorators = {} - if node.origin.metadata.get(SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA): - decorators[SUPPRESS_DEFAULT_CONSTRUCTOR_METADATA] = True - decorators[models.RESOLVED_CLASS_INSTANCE_POLICY_METADATA] = node.metadata[ - models.RESOLVED_CLASS_INSTANCE_POLICY_METADATA - ] - decorators[models.RESOLVED_CLASS_SELF_POLICY_METADATA] = node.metadata[ - models.RESOLVED_CLASS_SELF_POLICY_METADATA - ] - cls = ClassDef( - name, - attributes=attributes, - methods=(), - superclasses=superclasses, - scope=class_scope, - class_type=class_type, - decorators=decorators, - ) - self.scope.insert_class(cls) - self._populate_codegen_class_methods(cls, node, class_scope) - return cls - - def _visit_SemanticArgument(self, node): - if node.semantic_type.name == "Callable": - metadata = node.semantic_type.metadata - callback_policy = metadata.get(models.RESOLVED_CALLBACK_POLICY_METADATA) - if not isinstance(callback_policy, CallbackHandoffPolicy): - raise ValueError(f"Callback argument {node.name!r} is missing completed callback policy") - callback_arguments = metadata.get("callback_arguments") - if not isinstance(callback_arguments, list): - raise ValueError(f"Callback argument {node.name!r} is missing a complete callable argument contract") - - try: - name = self.scope.get_expected_name(node.name) - except RuntimeError: - name = self.scope.get_new_public_name( - node.name, - object_type="argument", - owner=f"callback argument {node.name}", - ) - callback_scope = self.scope.new_child_scope(f"{name}_callback", "function") - declarations = [ - self._lower_child( - item, - scope=callback_scope, - cls_base=None, - ) - for item in callback_arguments - ] - result_type = metadata.get("return") - result = ( - FunctionDefResult( - self._callback_result_variable( - result_type, - f"{name}_result", - callback_scope, - ) - ) - if isinstance(result_type, models.SemanticType) and result_type.name != "None" - else FunctionDefResult(NIL) - ) - return FunctionAddress( - name, - [FunctionDefArgument(item) for item in declarations], - result, - is_optional=node.optional, - is_argument=True, - decorators={"x2py_callback": dict(metadata)}, - scope=callback_scope, - ) - return self._visit_SemanticVariable(node) - - def _visit_SemanticVariable(self, node): - semantic_type = node.semantic_type - dtype, shape = _semantic_variable_type_and_shape(semantic_type, self.scope, self.custom_types) - name = _semantic_variable_name(node, self.scope) - ownership_decision = _variable_ownership_decision(node) - default_value = ( - node.default_value - if _is_constant(semantic_type) - else node.metadata.get(models.RESOLVED_MODULE_VARIABLE_INITIALIZER_METADATA) - ) - fortran_array_category, fortran_source_shape = _fortran_array_category_and_source_shape(semantic_type) - native_array_handle_policy = _native_array_variable_policy(node) - array_policy = array_interop_policy( - semantic_type, - owner=f"variable {node.name}", - native_array_handle_policy=native_array_handle_policy, - ) - var = Variable( - dtype, - name, - shape=shape, - memory_handling=ownership_decision.storage_mode.value, - is_private=node.visibility == "private", - is_target=bool(semantic_type.metadata.get("aliased")), - is_optional=getattr(node, "optional", False), - passes_by_value=_passes_by_value(node), - fortran_array_category=fortran_array_category, - fortran_callback_access=node.metadata.get(models.CALLBACK_DECLARATION_ACCESS_METADATA), - fortran_character_length=_fortran_character_length(semantic_type), - fortran_source_shape=fortran_source_shape, - getter_ownership_decision=node.metadata.get(models.RESOLVED_GETTER_OWNERSHIP_POLICY_METADATA), - ownership_decision=ownership_decision, - setter_ownership_decision=node.metadata.get(models.RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA), - native_array_handle_policy=native_array_handle_policy, - array_interop_policy=array_policy, - projected_output=bool(node.metadata.get(PROJECTED_OUTPUT_METADATA)), - assumed_rank=_is_assumed_rank(semantic_type), - cls_base=self.cls_base, - default_value=default_value, - ) - self.scope.insert_variable(var, name=node.name) - return var - - -def semantic_ir_to_codegen_ast( - node, - scope, - legacy: bool = False, - *, - custom_types: dict[str, object] | None = None, - cls_base: ClassDef | None = None, - class_lookup: dict[str, models.SemanticClass] | None = None, - class_descendants: dict[str, tuple[str, ...]] | None = None, - class_order: dict[str, int] | None = None, - enable_polymorphic_dispatch: bool = True, -): - """Convert one semantic IR node into the current codegen AST representation.""" - - return _SemanticIrToCodegenAstVisitor( - scope, - legacy, - custom_types=custom_types, - cls_base=cls_base, - class_lookup=class_lookup, - class_descendants=class_descendants, - class_order=class_order, - enable_polymorphic_dispatch=enable_polymorphic_dispatch, - )._visit(node) - - -ir_to_ast = semantic_ir_to_codegen_ast diff --git a/x2py/semantics/models.py b/x2py/semantics/models.py index 48922714b..01a03e1e7 100644 --- a/x2py/semantics/models.py +++ b/x2py/semantics/models.py @@ -326,6 +326,7 @@ class ProcedureOverloadSet: RESOLVED_DERIVED_FIELD_POLICY_METADATA = "resolved_derived_field_policy" RESOLVED_DERIVED_TYPE_POLICY_METADATA = "resolved_derived_type_policy" RESOLVED_CLASS_SURFACE_POLICY_METADATA = "resolved_class_surface_policy" +RESOLVED_MODULE_OVERLOAD_POLICIES_METADATA = "resolved_module_overload_policies" RESOLVED_DERIVED_TYPE_IDENTITY_METADATA = "resolved_derived_type_identity" RESOLVED_GETTER_OWNERSHIP_POLICY_METADATA = "resolved_getter_ownership_policy" RESOLVED_SETTER_OWNERSHIP_POLICY_METADATA = "resolved_setter_ownership_policy" diff --git a/x2py/semantics/ownership.py b/x2py/semantics/ownership.py index 9aa6ff8f8..1840b8845 100644 --- a/x2py/semantics/ownership.py +++ b/x2py/semantics/ownership.py @@ -1809,7 +1809,12 @@ def _native_barrier_action( if decision.kind is ObjectKind.DERIVED_TYPE: return NativeBarrierAction.PASS_WRAPPER_ADDRESS if decision.kind is ObjectKind.SCALAR: - if facts.address_role == ADDRESS_ROLE_PROJECTION or decision.codegen_action is CodegenAction.COPY_IN_OUT: + hidden_output = context.projects_result and not context.python_visible and not decision.descriptor_boundary + if ( + facts.address_role == ADDRESS_ROLE_PROJECTION + or hidden_output + or decision.codegen_action is CodegenAction.COPY_IN_OUT + ): return NativeBarrierAction.PASS_CALL_LOCAL_ADDRESS return NativeBarrierAction.PASS_VALUE return NativeBarrierAction.BLOCKED diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index a7f51419e..94db67287 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -32,8 +32,9 @@ from x2py.semantics.native_array_handles import NativeArrayHandlePolicy, native_array_descriptor_kind from x2py.semantics.wrapper_policy import ( ArgumentPolicy, - ClassOverloadArgumentPolicy, - ClassOverloadMatchKind, + OverloadArgumentPolicy, + OverloadMatchKind, + OverloadPolicy, ClassInvocationKind, ClassMethodKind, ClassMethodPolicy, @@ -50,8 +51,10 @@ build_derived_field_policy, build_derived_type_policy, build_module_variable_policy, + build_module_overload_policy, build_function_wrapper_policy, derived_member_path_policies, + overload_builtin_scalar_family, ) from x2py.semantics.wrapper_exports import complete_python_export_policy @@ -87,6 +90,8 @@ def complete_semantic_policies( semantic_ir: models.SemanticModule | Iterable[models.SemanticModule], + *, + strict_wrapper_names: bool = False, ) -> list[models.SemanticModule]: """Complete policy decisions for semantic modules after parser-to-IR conversion. @@ -101,8 +106,8 @@ def complete_semantic_policies( modules = list(semantic_ir) if not isinstance(semantic_ir, models.SemanticModule) else [semantic_ir] for module in modules: _complete_entry_export_policy(module) - complete_python_export_policy(module) - _complete_ownership_policies(module) + complete_python_export_policy(module, strict_wrapper_names=strict_wrapper_names) + _complete_ownership_policies(module, strict_wrapper_names=strict_wrapper_names) return modules @@ -133,7 +138,11 @@ def _entry_exports(declaration: object) -> object: raise TypeError(f"Unsupported semantic declaration: {type(declaration).__name__}") -def _complete_ownership_policies(module: models.SemanticModule) -> models.SemanticModule: +def _complete_ownership_policies( + module: models.SemanticModule, + *, + strict_wrapper_names: bool, +) -> models.SemanticModule: """Attach resolved ownership decisions to a full semantic module. Raw semantic types such as ``Float64[:]`` do not carry enough context to @@ -150,7 +159,11 @@ def _complete_ownership_policies(module: models.SemanticModule) -> models.Semant class_scope = str(semantic_class.origin.native_scope or module.name) _complete_class(semantic_class, f"{class_scope}.{semantic_class.name}") derived_types = _complete_derived_type_graph_policies(module.classes) - _complete_class_surface_policies(module.classes, derived_types) + _complete_class_surface_policies( + module.classes, + derived_types, + strict_wrapper_names=strict_wrapper_names, + ) class_targets = _class_root_target_names(module.classes) polymorphic_variants = _polymorphic_variant_map(module.classes) _complete_class_method_policies( @@ -178,13 +191,35 @@ def _complete_ownership_policies(module: models.SemanticModule) -> models.Semant polymorphic_variants=polymorphic_variants, ) for overload_set in module.overload_sets: + native_dispatch_name = next( + ( + str(procedure.metadata[models.FORTRAN_GENERIC_NAME_METADATA]) + for procedure in overload_set.procedures + if procedure.metadata.get(models.FORTRAN_GENERIC_NAME_METADATA) + ), + overload_set.name, + ) for procedure in overload_set.procedures: procedure_scope = str(procedure.origin.native_scope or module.name) _complete_function( procedure, f"{procedure_scope}.{overload_set.name}.{procedure.name}", derived_types=derived_types, + native_dispatch_name=native_dispatch_name, ) + overload_functions = { + f"{(procedure.origin.native_scope or module.name)!s}.{overload_set.name}.{procedure.name}": procedure + for overload_set in module.overload_sets + for procedure in overload_set.procedures + } + module.metadata[models.RESOLVED_MODULE_OVERLOAD_POLICIES_METADATA] = tuple( + _complete_overload_policy( + build_module_overload_policy(module, overload_set), + overload_functions, + require_uniform_receiver=False, + ) + for overload_set in module.overload_sets + ) module.metadata[models.POLICY_COMPLETION_PREPARED_METADATA] = True return module @@ -312,6 +347,8 @@ def complete(semantic_class: models.SemanticClass) -> None: def _complete_class_surface_policies( classes: list[models.SemanticClass], derived_types: dict[tuple[str, str], DerivedTypePolicy], + *, + strict_wrapper_names: bool, ) -> None: """Complete class orchestration after every derived identity is known.""" identities = { @@ -329,6 +366,7 @@ def _complete_class_surface_policies( owner_path=derived.owner_path, derived=derived, class_identities=identities, + strict_wrapper_names=strict_wrapper_names, ) completed_derived = replace(derived, fields=surface.effective_fields) semantic_class.metadata[models.RESOLVED_DERIVED_TYPE_POLICY_METADATA] = completed_derived @@ -378,82 +416,187 @@ def _complete_class_method_policies( polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]], ) -> None: """Complete methods once from class policy, type graphs, and dispatch sets.""" - type_bound_targets = { + type_bound_targets = _type_bound_target_names(module_functions) + module_targets = {str(function.native_name or function.name) for function in module_functions} + for semantic_class in _iter_semantic_classes(classes): + _complete_one_class_method_policy( + semantic_class, + type_bound_targets, + module_targets, + derived_types, + polymorphic_variants, + ) + + +def _type_bound_target_names(module_functions: list[models.SemanticFunction]) -> set[str]: + """Return native names explicitly marked as type-bound root targets.""" + return { str(function.native_name or function.name) for function in module_functions if function.metadata.get("fortran_type_bound_target") } - module_targets = {str(function.native_name or function.name) for function in module_functions} - for semantic_class in _iter_semantic_classes(classes): - derived = semantic_class.metadata.get(models.RESOLVED_DERIVED_TYPE_POLICY_METADATA) - surface = semantic_class.metadata.get(models.RESOLVED_CLASS_SURFACE_POLICY_METADATA) - if not isinstance(derived, DerivedTypePolicy) or not isinstance(surface, ClassSurfacePolicy): - continue - native_bindings = { - str(method.native_name or method.name): _native_type_bound_binding_name(method) - for method in semantic_class.methods - if method.name != "__init__" - } - generic_bindings = { - str(procedure.native_name or procedure.name): overload.name - for overload in semantic_class.overload_sets - if overload.name != "__init__" - for procedure in overload.procedures - } - completed_methods = tuple( - replace( - method, - invocation=ClassInvocationKind.TYPE_BOUND, - type_bound_name=native_bindings[method.native_name], - ) - if _uses_type_bound_invocation(method, type_bound_targets, module_targets) - else method - for method in surface.methods + + +def _complete_one_class_method_policy( + semantic_class: models.SemanticClass, + type_bound_targets: set[str], + module_targets: set[str], + derived_types: dict[tuple[str, str], DerivedTypePolicy], + polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]], +) -> None: + """Complete ordinary and overloaded calls for one prepared class surface.""" + derived = semantic_class.metadata.get(models.RESOLVED_DERIVED_TYPE_POLICY_METADATA) + surface = semantic_class.metadata.get(models.RESOLVED_CLASS_SURFACE_POLICY_METADATA) + if not isinstance(derived, DerivedTypePolicy) or not isinstance(surface, ClassSurfacePolicy): + return + native_bindings = { + str(method.native_name or method.name): _native_type_bound_binding_name(method) + for method in semantic_class.methods + if method.name != "__init__" + } + completed_methods = _completed_class_method_invocations( + surface, + native_bindings, + type_bound_targets, + module_targets, + ) + semantic_class.metadata[models.RESOLVED_CLASS_SURFACE_POLICY_METADATA] = replace( + surface, + methods=completed_methods, + ) + _complete_concrete_class_methods( + semantic_class, + derived, + completed_methods, + derived_types, + polymorphic_variants, + ) + _complete_class_overload_methods( + semantic_class, + derived, + type_bound_targets, + module_targets, + derived_types, + polymorphic_variants, + ) + + +def _completed_class_method_invocations( + surface: ClassSurfacePolicy, + native_bindings: dict[str, str], + type_bound_targets: set[str], + module_targets: set[str], +) -> tuple[ClassMethodPolicy, ...]: + """Select direct or type-bound invocation for every ordinary method.""" + return tuple( + replace( + method, + invocation=ClassInvocationKind.TYPE_BOUND, + type_bound_name=native_bindings[method.native_name], ) - surface = replace(surface, methods=completed_methods) - semantic_class.metadata[models.RESOLVED_CLASS_SURFACE_POLICY_METADATA] = surface - calls = {method.owner_path: method for method in completed_methods} - for method in semantic_class.methods: - if method.name == "__init__": - if method.metadata.get("bind_target"): - _complete_function( - method, - f"{derived.owner_path}.__init__", - derived_types=derived_types, - module_export=False, - polymorphic_variants=polymorphic_variants, - ) - continue - call = calls.get(f"{derived.owner_path}.{method.name}") - _complete_function( - method, - f"{derived.owner_path}.{method.name}", - derived_types=derived_types, - class_call=call, - polymorphic_variants=polymorphic_variants, - ) - for overload in semantic_class.overload_sets: - for procedure in overload.procedures: - owner_path = f"{derived.owner_path}.{overload.name}.{procedure.name}" - native_name = str(procedure.native_name or procedure.name) - passed_position = _class_overload_passed_object_position(procedure) - type_bound = native_name in type_bound_targets or ( - passed_position is not None and native_name not in module_targets - ) - call = _class_overload_call_policy( - overload, - procedure, - owner_path, - type_bound=type_bound, - type_bound_name=generic_bindings.get(native_name) if type_bound else None, - ) + if _uses_type_bound_invocation(method, type_bound_targets, module_targets) + else method + for method in surface.methods + ) + + +def _complete_concrete_class_methods( + semantic_class: models.SemanticClass, + derived: DerivedTypePolicy, + completed_methods: tuple[ClassMethodPolicy, ...], + derived_types: dict[tuple[str, str], DerivedTypePolicy], + polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]], +) -> None: + """Attach completed function policy to constructors and ordinary methods.""" + calls = {method.owner_path: method for method in completed_methods} + for method in semantic_class.methods: + owner_path = f"{derived.owner_path}.{method.name}" + if method.name == "__init__": + if method.metadata.get("bind_target"): _complete_function( - procedure, + method, owner_path, derived_types=derived_types, - class_call=call, + module_export=False, polymorphic_variants=polymorphic_variants, ) + continue + _complete_function( + method, + owner_path, + derived_types=derived_types, + class_call=calls.get(owner_path), + polymorphic_variants=polymorphic_variants, + ) + + +def _complete_class_overload_methods( + semantic_class: models.SemanticClass, + derived: DerivedTypePolicy, + type_bound_targets: set[str], + module_targets: set[str], + derived_types: dict[tuple[str, str], DerivedTypePolicy], + polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]], +) -> None: + """Complete every concrete overload through one typed call leaf.""" + generic_bindings = { + str(procedure.native_name or procedure.name): overload.name + for overload in semantic_class.overload_sets + if overload.name != "__init__" + for procedure in overload.procedures + } + for overload in semantic_class.overload_sets: + for procedure in overload.procedures: + _complete_one_class_overload_method( + overload, + procedure, + derived, + generic_bindings, + type_bound_targets, + module_targets, + derived_types, + polymorphic_variants, + ) + + +def _complete_one_class_overload_method( + overload: models.ProcedureOverloadSet, + procedure: models.SemanticFunction, + derived: DerivedTypePolicy, + generic_bindings: dict[str, str], + type_bound_targets: set[str], + module_targets: set[str], + derived_types: dict[tuple[str, str], DerivedTypePolicy], + polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]], +) -> None: + """Complete one overload candidate and its native dispatch spelling.""" + owner_path = f"{derived.owner_path}.{overload.name}.{procedure.name}" + native_name = str(procedure.native_name or procedure.name) + passed_position = _class_overload_passed_object_position(procedure) + type_bound = native_name in type_bound_targets or ( + passed_position is not None and native_name not in module_targets + ) + call = _class_overload_call_policy( + overload, + procedure, + owner_path, + type_bound=type_bound, + type_bound_name=generic_bindings.get(native_name) if type_bound else None, + ) + overload_kind = str(procedure.metadata.get(models.OVERLOAD_KIND_METADATA, "generic")) + native_dispatch_name = ( + str(procedure.metadata.get(models.FORTRAN_GENERIC_NAME_METADATA, overload.name)) + if overload_kind != "generic" + else None + ) + _complete_function( + procedure, + owner_path, + derived_types=derived_types, + class_call=call, + polymorphic_variants=polymorphic_variants, + native_dispatch_name=native_dispatch_name, + ) def _uses_type_bound_invocation( @@ -462,9 +605,9 @@ def _uses_type_bound_invocation( module_targets: set[str], ) -> bool: """Restore generated-.pyi type-bound calls when their private root target is absent.""" - return method.native_name in explicit_targets or ( - method.kind is ClassMethodKind.INSTANCE and method.native_name not in module_targets - ) + if method.kind is ClassMethodKind.STATIC: + return False + return method.native_name in explicit_targets or method.native_name not in module_targets def _native_type_bound_binding_name(method: models.SemanticMethod) -> str: @@ -491,50 +634,9 @@ def _complete_class_overload_policies(classes: list[models.SemanticClass]) -> No blockers = list(surface.blockers) overloads = [] for overload in surface.overloads: - candidates = [] - for candidate in overload.candidates: - function = functions[candidate.owner_path] - function_policy = function.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) - if not isinstance(function_policy, FunctionWrapperPolicy): - blockers.append(f"overload candidate {candidate.owner_path!r} has no completed call policy") - candidates.append(candidate) - continue - if ( - function_policy.class_call is not None - and function_policy.class_call.invocation is ClassInvocationKind.TYPE_BOUND - and function_policy.class_call.type_bound_name is None - ): - blockers.append(f"overload candidate {candidate.owner_path!r} has no accessible type-bound binding") - matches, match_blockers = _class_overload_matches(function_policy) - blockers.extend(f"overload candidate {candidate.owner_path!r}: {reason}" for reason in match_blockers) - candidates.append( - replace( - candidate, - arguments=matches, - passed_object=( - function_policy.class_call is not None - and function_policy.class_call.passed_object_position is not None - ), - ) - ) - if len({candidate.passed_object for candidate in candidates}) > 1: - blockers.append(f"overload {overload.owner_path!r} mixes instance and static candidates") - signatures = [ - tuple( - ( - argument.kind, - argument.optional, - argument.semantic_type_name, - argument.rank, - argument.derived_type_identity, - ) - for argument in candidate.arguments - ) - for candidate in candidates - ] - if len(set(signatures)) != len(signatures): - blockers.append(f"overload {overload.owner_path!r} has indistinguishable Python signatures") - overloads.append(replace(overload, candidates=tuple(candidates))) + completed = _complete_overload_policy(overload, functions, require_uniform_receiver=True) + overloads.append(completed) + blockers.extend(completed.blockers) semantic_class.metadata[models.RESOLVED_CLASS_SURFACE_POLICY_METADATA] = replace( surface, overloads=tuple(overloads), @@ -543,17 +645,95 @@ def _complete_class_overload_policies(classes: list[models.SemanticClass]) -> No ) -def _class_overload_matches( +def _complete_overload_policy( + overload: OverloadPolicy, + functions: dict[str, models.SemanticFunction], + *, + require_uniform_receiver: bool, +) -> OverloadPolicy: + """Attach exact candidate predicates after all concrete calls are complete.""" + blockers = list(overload.blockers) + candidates = [] + for candidate in overload.candidates: + function = functions[candidate.owner_path] + function_policy = function.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) + if not isinstance(function_policy, FunctionWrapperPolicy): + blockers.append(f"overload candidate {candidate.owner_path!r} has no completed call policy") + candidates.append(candidate) + continue + if ( + function_policy.class_call is not None + and function_policy.class_call.invocation is ClassInvocationKind.TYPE_BOUND + and function_policy.class_call.type_bound_name is None + ): + blockers.append(f"overload candidate {candidate.owner_path!r} has no accessible type-bound binding") + matches, match_blockers = _overload_matches(function_policy) + blockers.extend(f"overload candidate {candidate.owner_path!r}: {reason}" for reason in match_blockers) + candidates.append( + replace( + candidate, + arguments=matches, + passed_object=( + function_policy.class_call is not None + and function_policy.class_call.passed_object_position is not None + ), + ) + ) + if require_uniform_receiver and len({candidate.passed_object for candidate in candidates}) > 1: + blockers.append(f"overload {overload.owner_path!r} mixes instance and static candidates") + signatures = tuple(_overload_candidate_signature(candidate.arguments) for candidate in candidates) + if len(set(signatures)) != len(signatures): + blockers.append(f"overload {overload.owner_path!r} has indistinguishable Python signatures") + builtin_signatures = tuple(_overload_candidate_builtin_signature(candidate.arguments) for candidate in candidates) + if len(set(builtin_signatures)) != len(builtin_signatures): + blockers.append(f"overload {overload.owner_path!r} has overlapping reflected scalar signatures") + return replace(overload, candidates=tuple(candidates), blockers=tuple(dict.fromkeys(blockers))) + + +def _overload_candidate_signature(arguments: tuple[OverloadArgumentPolicy, ...]) -> tuple: + """Return only runtime-relevant facts for ambiguity detection.""" + return tuple( + ( + argument.kind, + argument.optional, + argument.semantic_type_name, + argument.rank, + argument.derived_type_identity, + ) + for argument in arguments + ) + + +def _overload_candidate_builtin_signature(arguments: tuple[OverloadArgumentPolicy, ...]) -> tuple: + """Normalize reflected Python scalar domains for overlap detection.""" + return tuple( + ( + argument.kind, + argument.optional, + ( + overload_builtin_scalar_family(argument.semantic_type_name) + if argument.accept_builtin_scalar + else argument.semantic_type_name + ), + argument.rank, + argument.derived_type_identity, + ) + for argument in arguments + ) + + +def _overload_matches( function: FunctionWrapperPolicy, -) -> tuple[tuple[ClassOverloadArgumentPolicy, ...], tuple[str, ...]]: +) -> tuple[tuple[OverloadArgumentPolicy, ...], tuple[str, ...]]: """Translate one completed call signature into exact Python predicates.""" passed_position = function.class_call.passed_object_position if function.class_call is not None else None + reflected = bool(function.class_call is not None and function.class_call.passed_object_position not in {None, 0}) matches = [] blockers = [] for argument in function.arguments: if not argument.python_visible or argument.native_position == passed_position: continue - match = _class_overload_argument_match(argument) + match = _overload_argument_match(argument, accept_builtin_scalar=reflected) if match is None: blockers.append( f"argument {argument.python_name!r} has no exact overload predicate for {argument.ownership.kind.value}" @@ -563,29 +743,34 @@ def _class_overload_matches( return tuple(matches), tuple(blockers) -def _class_overload_argument_match(argument: ArgumentPolicy) -> ClassOverloadArgumentPolicy | None: +def _overload_argument_match( + argument: ArgumentPolicy, + *, + accept_builtin_scalar: bool, +) -> OverloadArgumentPolicy | None: """Return one typed match record without embedding generated source text.""" kind = argument.ownership.kind match_kind = None derived_identity = None if kind is ObjectKind.SCALAR and argument.semantic_type_name in SEMANTIC_SCALAR_TYPE_NAMES: - match_kind = ClassOverloadMatchKind.NUMPY_SCALAR + match_kind = OverloadMatchKind.NUMPY_SCALAR elif kind is ObjectKind.STRING: - match_kind = ClassOverloadMatchKind.STRING + match_kind = OverloadMatchKind.STRING elif kind is ObjectKind.NUMPY_ARRAY and argument.array is not None: - match_kind = ClassOverloadMatchKind.NUMPY_ARRAY + match_kind = OverloadMatchKind.NUMPY_ARRAY elif kind is ObjectKind.DERIVED_TYPE and argument.derived is not None: - match_kind = ClassOverloadMatchKind.DERIVED + match_kind = OverloadMatchKind.DERIVED derived_identity = argument.derived.type_identity if match_kind is None: return None - return ClassOverloadArgumentPolicy( + return OverloadArgumentPolicy( python_name=argument.python_name, kind=match_kind, optional=argument.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR}, semantic_type_name=argument.semantic_type_name, rank=argument.rank, derived_type_identity=derived_identity, + accept_builtin_scalar=accept_builtin_scalar and match_kind is OverloadMatchKind.NUMPY_SCALAR, ) @@ -640,7 +825,7 @@ def _class_overload_call_policy( passed = _class_overload_passed_object_position(procedure) return ClassMethodPolicy( owner_path=owner_path, - python_name=overload.name, + python_name=str(procedure.metadata.get(models.PYTHON_METHOD_NAME_METADATA, overload.name)), native_name=str(procedure.native_name or procedure.name), kind=ClassMethodKind.INSTANCE if passed is not None else ClassMethodKind.STATIC, passed_object_position=passed, @@ -664,6 +849,7 @@ def _complete_function( class_call: ClassMethodPolicy | None = None, module_export: bool | None = None, polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]] | None = None, + native_dispatch_name: str | None = None, ) -> None: _complete_callable_address_policy(function) for argument in function.arguments: @@ -687,6 +873,7 @@ def _complete_function( class_call=class_call, module_export=module_export, polymorphic_variants=polymorphic_variants, + native_dispatch_name=native_dispatch_name, ) diff --git a/x2py/semantics/wrapper_exports.py b/x2py/semantics/wrapper_exports.py index dab72cb17..9b5b2219c 100644 --- a/x2py/semantics/wrapper_exports.py +++ b/x2py/semantics/wrapper_exports.py @@ -16,9 +16,13 @@ class PythonExportPolicy: name: str -def complete_python_export_policy(module: models.SemanticModule) -> None: +def complete_python_export_policy( + module: models.SemanticModule, + *, + strict_wrapper_names: bool = False, +) -> None: """Resolve every public export name within its owning Python namespace.""" - naming = NamingPolicy() + naming = NamingPolicy(strict_public_names=strict_wrapper_names) for owner in _module_export_owners(module): if getattr(owner, "visibility", "public") == "private": continue diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index 0c90ff696..6a772fd1f 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -140,6 +140,7 @@ class TransformationAction(str, Enum): """Typed representation or lifecycle operation selected before planning.""" COPY_ARRAY_REPRESENTATION = "copy_array_representation" + PUBLISH_ARRAY_REPLACEMENT = "publish_array_replacement" RELEASE_TEMPORARY = "release_temporary" @@ -209,6 +210,7 @@ class ModuleGetterAction(str, Enum): CONSTANT_VALUE = "constant_value" DIRECT_VALUE = "direct_value" NULLABLE_SNAPSHOT = "nullable_snapshot" + BORROWED_ARRAY_VIEW = "borrowed_array_view" NATIVE_ARRAY_HANDLE = "native_array_handle" DERIVED_OBJECT = "derived_object" @@ -369,6 +371,27 @@ class ClassInvocationKind(str, Enum): TYPE_BOUND = "type_bound" +class NativeInvocationKind(str, Enum): + """Completed native syntax for one concrete wrapper call.""" + + PROCEDURE = "procedure" + DEFINED_OPERATOR = "defined_operator" + DEFINED_ASSIGNMENT = "defined_assignment" + + +def overload_builtin_scalar_family(semantic_type_name: str) -> str: + """Return the Python scalar family admitted by reflected dispatch.""" + if semantic_type_name == "Bool": + return "bool" + if semantic_type_name.startswith("Int"): + return "int" + if semantic_type_name.startswith("Float"): + return "float" + if semantic_type_name.startswith("Complex"): + return "complex" + raise ValueError(f"Unsupported reflected overload scalar {semantic_type_name!r}") + + class ClassRegistrationAction(str, Enum): """Dependency-ordered Python class registration actions.""" @@ -388,8 +411,8 @@ class ConstructionLifecycleAction(str, Enum): DESTROY_OWNED = "destroy_owned" -class ClassOverloadMatchKind(str, Enum): - """Exact Python runtime category used by class overload selection.""" +class OverloadMatchKind(str, Enum): + """Exact Python runtime category used by overload selection.""" NUMPY_SCALAR = "numpy_scalar" NUMPY_ARRAY = "numpy_array" @@ -547,34 +570,39 @@ class ClassMethodPolicy: @dataclass(frozen=True) -class ClassOverloadArgumentPolicy: - """One completed exact-type predicate in a class overload signature.""" +class OverloadArgumentPolicy: + """One completed exact-type predicate in an overload signature.""" python_name: str - kind: ClassOverloadMatchKind + kind: OverloadMatchKind optional: bool semantic_type_name: str rank: int derived_type_identity: tuple[str, str] | None + accept_builtin_scalar: bool = False @dataclass(frozen=True) -class ClassOverloadCandidatePolicy: +class OverloadCandidatePolicy: """One concrete overload target and its ordered runtime predicates.""" owner_path: str - arguments: tuple[ClassOverloadArgumentPolicy, ...] + arguments: tuple[OverloadArgumentPolicy, ...] passed_object: bool @dataclass(frozen=True) -class ClassOverloadPolicy: - """One class-owned overload set with explicit concrete candidates.""" +class OverloadPolicy: + """One overload set with explicit concrete candidates and exports.""" owner_path: str python_name: str kind: str - candidates: tuple[ClassOverloadCandidatePolicy, ...] + candidates: tuple[OverloadCandidatePolicy, ...] + python_exports: tuple[PythonExportPolicy, ...] = () + blockers: tuple[str, ...] = () + unsupported_extra_argument_message: str | None = None + identity_receiver_shortcut: bool = False @dataclass(frozen=True) @@ -588,7 +616,7 @@ class ClassSurfacePolicy: effective_fields: tuple[DerivedFieldPolicy, ...] constructor: ConstructorPolicy methods: tuple[ClassMethodPolicy, ...] - overloads: tuple[ClassOverloadPolicy, ...] + overloads: tuple[OverloadPolicy, ...] registration: tuple[ClassRegistrationAction, ...] supported: bool blockers: tuple[str, ...] = () @@ -777,6 +805,7 @@ class ModuleVariablePolicy: constant_value: Any supported: bool blockers: tuple[str, ...] = () + array: ArrayHandoffPolicy | None = None native_array_handle: NativeArrayHandleWrapperPolicy | None = None derived: DerivedModuleObjectPolicy | None = None @@ -1059,6 +1088,8 @@ class FunctionWrapperPolicy: owner_path: str python_exports: tuple[PythonExportPolicy, ...] native_name: str + native_invocation: NativeInvocationKind + native_operator: str | None external: bool native_module: str | None native_is_subroutine: bool @@ -1219,9 +1250,10 @@ def build_class_surface_policy( owner_path: str, derived: DerivedTypePolicy, class_identities: dict[str, tuple[str, str]], + strict_wrapper_names: bool = False, ) -> ClassSurfacePolicy: """Complete constructor, method, inheritance, and registration decisions.""" - naming = NamingPolicy() + naming = NamingPolicy(strict_public_names=strict_wrapper_names) fields = _python_named_class_fields(derived.fields, naming, owner_path) named_derived = replace(derived, fields=fields) methods = _python_named_class_methods(semantic_class, naming, owner_path) @@ -1312,22 +1344,36 @@ def _python_named_class_overloads( semantic_class: models.SemanticClass, naming: NamingPolicy, owner_path: str, -) -> tuple[ClassOverloadPolicy, ...]: - """Reserve each generic name after concrete methods in declaration order.""" +) -> tuple[OverloadPolicy, ...]: + """Split reflected operators, then reserve every public overload name.""" namespace = (owner_path,) - return tuple( - replace( - policy, - python_name=naming.reserve_public_name( - namespace, - policy.python_name, - category="function", - owner=policy.owner_path, - ), + policies = [] + for overload in semantic_class.overload_sets: + names = tuple( + dict.fromkeys( + str(procedure.metadata.get(models.PYTHON_METHOD_NAME_METADATA, overload.name)) + for procedure in overload.procedures + ) ) - for overload in semantic_class.overload_sets - for policy in (_class_overload_policy(owner_path, overload),) - ) + for python_name in names: + procedures = tuple( + procedure + for procedure in overload.procedures + if str(procedure.metadata.get(models.PYTHON_METHOD_NAME_METADATA, overload.name)) == python_name + ) + policy = _overload_policy(owner_path, overload, python_name=python_name, procedures=procedures) + policies.append( + replace( + policy, + python_name=naming.reserve_public_name( + namespace, + policy.python_name, + category="function", + owner=policy.owner_path, + ), + ) + ) + return tuple(policies) def _class_constructor_policy( @@ -1442,24 +1488,50 @@ def _class_method_blockers(method: ClassMethodPolicy) -> str | None: return None -def _class_overload_policy(owner_path: str, overload: models.ProcedureOverloadSet) -> ClassOverloadPolicy: - """Complete one class-owned overload set from its explicit links.""" +def _overload_policy( + owner_path: str, + overload: models.ProcedureOverloadSet, + *, + python_name: str | None = None, + procedures: tuple[models.SemanticFunction, ...] | None = None, + python_exports: tuple[PythonExportPolicy, ...] = (), +) -> OverloadPolicy: + """Complete one overload set from explicit concrete-procedure links.""" + selected = tuple(overload.procedures) if procedures is None else procedures + public_name = python_name or overload.name candidates = tuple( - ClassOverloadCandidatePolicy( + OverloadCandidatePolicy( owner_path=f"{owner_path}.{overload.name}.{procedure.name}", arguments=(), passed_object=False, ) - for procedure in overload.procedures + for procedure in selected ) - kind = ( - str(overload.procedures[0].metadata.get(models.OVERLOAD_KIND_METADATA, "generic")) if candidates else "generic" - ) - return ClassOverloadPolicy( - owner_path=f"{owner_path}.{overload.name}", - python_name=overload.name, + kind = str(selected[0].metadata.get(models.OVERLOAD_KIND_METADATA, "generic")) if candidates else "generic" + return OverloadPolicy( + owner_path=f"{owner_path}.{public_name}", + python_name=public_name, kind=kind, candidates=candidates, + python_exports=python_exports, + unsupported_extra_argument_message=("modulus is not supported" if public_name == "__pow__" else None), + identity_receiver_shortcut=kind == "assignment", + ) + + +def build_module_overload_policy( + module: models.SemanticModule, + overload: models.ProcedureOverloadSet, +) -> OverloadPolicy: + """Complete the stable owner and Python exports for one module generic.""" + if not overload.procedures: + return _overload_policy(module.name, overload) + first = overload.procedures[0] + native_scope = str(first.origin.native_scope or module.name) + return _overload_policy( + native_scope, + overload, + python_exports=completed_python_exports(first, overload.name), ) @@ -1625,76 +1697,158 @@ def build_module_variable_policy( owner_path, ) if native_array_handle is not None: - blockers = _native_array_module_variable_blockers(variable, getter, setter, native_array_handle) - return ModuleVariablePolicy( - owner_path=owner_path, - name=variable.name, - python_exports=completed_python_exports(variable, variable.name), - native_name=str(variable.origin.native_name or variable.name), - native_module=str(variable.origin.native_scope or module_name), - semantic_type_name=variable.semantic_type.name, - rank=int(variable.semantic_type.rank or 0), - getter_action=ModuleGetterAction.NATIVE_ARRAY_HANDLE, - getter=getter, - setter_action=native_array_handle.setter_action, - native_assignment=native_array_handle.native_assignment, - setter=setter, - descriptor_kind=native_array_handle.descriptor_kind.value, - initializer=None, - constant_value=None, - supported=not blockers, - blockers=tuple(blockers), - native_array_handle=native_array_handle, + return _native_array_module_variable_policy( + variable, + module_name, + owner_path, + getter, + setter, + native_array_handle, ) if getter is not None and getter.kind is ObjectKind.DERIVED_TYPE: - if constant: - derived = _derived_module_constant_policy( - variable, - getter, - setter, - owner_path=owner_path, - derived_types=derived_types or {}, - ) - blockers = _derived_module_constant_blockers(variable, getter, setter, derived) - else: - derived = _derived_module_object_policy( - variable, - getter, - setter, - owner_path=owner_path, - derived_types=derived_types or {}, - ) - blockers = _derived_module_variable_blockers(variable, getter, setter, derived) - return ModuleVariablePolicy( - owner_path=owner_path, - name=variable.name, - python_exports=completed_python_exports(variable, variable.name), - native_name=str(variable.origin.native_name or variable.name), - native_module=str(variable.origin.native_scope or module_name), - semantic_type_name=variable.semantic_type.name, - rank=int(variable.semantic_type.rank or 0), - getter_action=ModuleGetterAction.DERIVED_OBJECT, - getter=getter, - setter_action=derived.replacement, - native_assignment=AssignmentMode.NONE, - setter=setter, - descriptor_kind=None, - initializer=None, - constant_value=None, - supported=not blockers, - blockers=tuple(blockers), - derived=derived, + return _derived_module_variable_policy( + variable, + module_name, + owner_path, + getter, + setter, + constant, + derived_types or {}, ) + array = _array_handoff_policy(variable.semantic_type) + if getter is not None and getter.kind is ObjectKind.NUMPY_ARRAY and array is not None: + return _ordinary_array_module_variable_policy( + variable, + module_name, + owner_path, + getter, + setter, + array, + ) + return _scalar_module_variable_policy( + variable, + module_name, + owner_path, + getter, + setter, + descriptor_kind, + constant, + ) + + +def _module_variable_policy_base( + variable: models.SemanticVariable, + module_name: str, + owner_path: str, +) -> dict[str, object]: + """Return identity fields shared by every module-variable policy family.""" + return { + "owner_path": owner_path, + "name": variable.name, + "python_exports": completed_python_exports(variable, variable.name), + "native_name": str(variable.origin.native_name or variable.name), + "native_module": str(variable.origin.native_scope or module_name), + "semantic_type_name": variable.semantic_type.name, + "rank": int(variable.semantic_type.rank or 0), + } + + +def _native_array_module_variable_policy( + variable: models.SemanticVariable, + module_name: str, + owner_path: str, + getter: OwnershipDecision | None, + setter: OwnershipDecision | None, + handle: NativeArrayHandleWrapperPolicy, +) -> ModuleVariablePolicy: + """Build one persistent native-array handle module policy.""" + blockers = _native_array_module_variable_blockers(variable, getter, setter, handle) + return ModuleVariablePolicy( + **_module_variable_policy_base(variable, module_name, owner_path), + getter_action=ModuleGetterAction.NATIVE_ARRAY_HANDLE, + getter=getter, + setter_action=handle.setter_action, + native_assignment=handle.native_assignment, + setter=setter, + descriptor_kind=handle.descriptor_kind.value, + initializer=None, + constant_value=None, + supported=not blockers, + blockers=tuple(blockers), + native_array_handle=handle, + ) + + +def _derived_module_variable_policy( + variable: models.SemanticVariable, + module_name: str, + owner_path: str, + getter: OwnershipDecision, + setter: OwnershipDecision | None, + constant: bool, + derived_types: dict[tuple[str, str], DerivedTypePolicy], +) -> ModuleVariablePolicy: + """Build one constant-copy or live derived module-object policy.""" + builder = _derived_module_constant_policy if constant else _derived_module_object_policy + derived = builder(variable, getter, setter, owner_path=owner_path, derived_types=derived_types) + blocker_builder = _derived_module_constant_blockers if constant else _derived_module_variable_blockers + blockers = blocker_builder(variable, getter, setter, derived) + return ModuleVariablePolicy( + **_module_variable_policy_base(variable, module_name, owner_path), + getter_action=ModuleGetterAction.DERIVED_OBJECT, + getter=getter, + setter_action=derived.replacement, + native_assignment=AssignmentMode.NONE, + setter=setter, + descriptor_kind=None, + initializer=None, + constant_value=None, + supported=not blockers, + blockers=tuple(blockers), + derived=derived, + ) + + +def _ordinary_array_module_variable_policy( + variable: models.SemanticVariable, + module_name: str, + owner_path: str, + getter: OwnershipDecision, + setter: OwnershipDecision | None, + array: ArrayHandoffPolicy, +) -> ModuleVariablePolicy: + """Build one borrowed ordinary module-array view policy.""" + blockers = _ordinary_array_module_variable_blockers(variable, getter, setter, array) + return ModuleVariablePolicy( + **_module_variable_policy_base(variable, module_name, owner_path), + getter_action=ModuleGetterAction.BORROWED_ARRAY_VIEW, + getter=getter, + setter_action=setter.setter_action if setter is not None else SetterAction.OMIT, + native_assignment=AssignmentMode.NONE, + setter=setter, + descriptor_kind=None, + initializer=None, + constant_value=None, + supported=not blockers, + blockers=tuple(blockers), + array=array, + ) + + +def _scalar_module_variable_policy( + variable: models.SemanticVariable, + module_name: str, + owner_path: str, + getter: OwnershipDecision | None, + setter: OwnershipDecision | None, + descriptor_kind: str | None, + constant: bool, +) -> ModuleVariablePolicy: + """Build one scalar value, snapshot, or constant module policy.""" blockers = _scalar_module_variable_blockers(variable, getter, setter, descriptor_kind, constant) initializer = variable.metadata.get(models.RESOLVED_MODULE_VARIABLE_INITIALIZER_METADATA) return ModuleVariablePolicy( - owner_path=owner_path, - name=variable.name, - python_exports=completed_python_exports(variable, variable.name), - native_name=str(variable.origin.native_name or variable.name), - native_module=str(variable.origin.native_scope or module_name), - semantic_type_name=variable.semantic_type.name, - rank=int(variable.semantic_type.rank or 0), + **_module_variable_policy_base(variable, module_name, owner_path), getter_action=_scalar_module_getter_action(getter, constant), getter=getter, setter_action=setter.setter_action if setter is not None else SetterAction.OMIT, @@ -1714,6 +1868,36 @@ def build_module_variable_policy( ) +def _ordinary_array_module_variable_blockers( + variable: models.SemanticVariable, + getter: OwnershipDecision, + setter: OwnershipDecision | None, + array: ArrayHandoffPolicy, +) -> tuple[str, ...]: + """Validate one fixed addressable module array borrowed as a live view.""" + blockers = [] + if array.rank is None or array.rank <= 0 or len(array.shape) != array.rank: + blockers.append("ordinary module array requires one concrete fixed rank") + if variable.semantic_type.name not in _PLAN_PRIMITIVE_SCALAR_TYPES: + blockers.append("ordinary module array requires a primitive numeric element type") + if not variable.semantic_type.metadata.get("aliased"): + blockers.append("ordinary module array requires addressable Aliased target storage") + expected_getter = ( + ("owner", getter.owner, OwnershipOwner.NATIVE), + ("transfer", getter.transfer, TransferMode.BORROWED_VIEW), + ("destruction", getter.destruction, DestructionPolicy.NATIVE_OWNER), + ("storage", getter.storage_mode, StorageMode.ALIAS), + ) + blockers.extend( + f"ordinary module array getter {name} is {actual.value}, not {required.value}" + for name, actual, required in expected_getter + if actual is not required + ) + if setter is None or setter.setter_action is not SetterAction.REJECT_REPLACEMENT: + blockers.append("ordinary module array must reject whole-array replacement") + return tuple(blockers) + + def completed_function_wrapper_policy(function: models.SemanticFunction) -> FunctionWrapperPolicy: """Return a completed function wrapper policy or fail before planning.""" @@ -1834,7 +2018,7 @@ def _callback_abi_kind( return CallbackABIKind.DATA_AND_LENGTH if int(semantic_type.rank or 0) > 0: return CallbackABIKind.DATA_AND_SHAPE - if bool(argument.origin.metadata.get("value")) or access == "read": + if bool(argument.origin.metadata.get("value")): return CallbackABIKind.VALUE return CallbackABIKind.REFERENCE @@ -1844,7 +2028,7 @@ def _callback_adapter_action( access: str, ) -> CallbackTransferAction: """Select adapter copy direction once from the callable declaration.""" - if bool(argument.origin.metadata.get("value")): + if bool(argument.origin.metadata.get("value")) or access == "read": return CallbackTransferAction.COPY_IN if access == "write": return CallbackTransferAction.COPY_OUT @@ -1961,6 +2145,7 @@ def build_function_wrapper_policy( class_call: ClassMethodPolicy | None = None, module_export: bool | None = None, polymorphic_variants: dict[tuple[str, str], tuple[tuple[str, str], ...]] | None = None, + native_dispatch_name: str | None = None, ) -> FunctionWrapperPolicy: """Build typed function policy from completed post-IR decisions.""" @@ -1989,16 +2174,20 @@ def build_function_wrapper_policy( + result_blockers + slot_blockers + lifecycle_blockers - + _mixed_result_writeback_blockers(results, arguments) + + _result_position_blockers(results, arguments) + _array_extent_reference_blockers(function, arguments, results) + _runtime_status_plan_blockers(status_error) + _string_result_status_blockers(results, status_error) + _string_writeback_status_blockers(arguments, status_error) ) + native_name = native_dispatch_name or _native_name(function) + native_invocation, native_operator = _native_invocation_policy(native_name) return FunctionWrapperPolicy( owner_path=owner_path, python_exports=completed_python_exports(function, function.name), - native_name=_native_name(function), + native_name=native_name, + native_invocation=native_invocation, + native_operator=native_operator, external=_is_external(function), native_module=_native_module(function, owner_path), native_is_subroutine=_native_is_subroutine(function), @@ -2020,27 +2209,14 @@ def build_function_wrapper_policy( ) -def _mixed_result_writeback_blockers( - results: tuple[ResultPolicy, ...], - arguments: list[ArgumentPolicy], -) -> tuple[str, ...]: - """Block the unsupported mixed result plus visible-writeback envelope.""" - projected = tuple(argument for argument in arguments if argument.projects_result) - if not results or not projected: - return () - if all( - result.source_kind == "hidden_output" and result.ownership.kind is ObjectKind.STRING for result in results - ) and all(argument.ownership.kind is ObjectKind.STRING for argument in projected): - positions = tuple(result.result_position for result in results) + tuple( - argument.result_position for argument in projected - ) - if all(isinstance(position, int) for position in positions) and sorted(positions) == list( - range(len(positions)) - ): - return () - return ("hidden results and visible writebacks must cover each result position exactly once",) - names = ", ".join(repr(argument.name) for argument in projected) - return (f"cannot combine native results with visible argument writeback for {names}",) +def _native_invocation_policy(native_name: str) -> tuple[NativeInvocationKind, str | None]: + """Classify procedure, defined-operator, and defined-assignment syntax once.""" + compact = "".join(native_name.split()).casefold() + if compact == "assignment(=)": + return NativeInvocationKind.DEFINED_ASSIGNMENT, "=" + if compact.startswith("operator(") and compact.endswith(")"): + return NativeInvocationKind.DEFINED_OPERATOR, compact[len("operator(") : -1] + return NativeInvocationKind.PROCEDURE, None def _argument_policies( @@ -2283,7 +2459,9 @@ def _argument_boundary_policy( optional_mode=_optional_mode(argument, decision), handoff_mode=_argument_handoff_mode(decision), nullable=decision.nullable, - writable=decision.mutates_native, + # COPY_RETURN mutates a binding-owned replacement rather than the + # immutable Python input whose payload is copied. + writable=decision.mutates_native and decision.transfer is not TransferMode.COPY_RETURN, descriptor_boundary=decision.descriptor_boundary, codegen_action=decision.codegen_action, python_barrier_action=decision.python_barrier_action, @@ -2369,11 +2547,7 @@ def _result_policies( if function.return_type is None: projected_arguments = _visible_projected_arguments(function) if hidden_results and not projected_arguments: - return hidden_results, ( - *hidden_blockers, - *_result_position_blockers(hidden_results), - *_string_result_aggregation_blockers(hidden_results), - ) + return hidden_results, hidden_blockers if projected_arguments and not hidden_results: return (), hidden_blockers if not hidden_results and not projected_arguments: @@ -2433,8 +2607,6 @@ def _result_policies( ( *blockers, *hidden_blockers, - *_result_position_blockers(results), - *_string_result_aggregation_blockers(results), ), ) @@ -3614,8 +3786,8 @@ def _scalar_boundary_blockers( and decision.native_barrier_action is not NativeBarrierAction.PASS_RAW_ADDRESS ): blockers.append(f"argument {argument.name!r} raw address is not forwarded as a raw address") - if argument.optional and decision.python_barrier_action is not PythonBarrierAction.SCALAR_VALUE: - blockers.append(f"argument {argument.name!r} optional storage/address boundaries are not supported") + if argument.optional and decision.python_barrier_action is PythonBarrierAction.RAW_ADDRESS: + blockers.append(f"argument {argument.name!r} optional raw-address boundaries are not supported") return tuple(blockers) @@ -3640,6 +3812,8 @@ def _array_storage_boundary_blockers( decision: OwnershipDecision, ) -> tuple[str, ...]: """Require one caller-owned ordinary NumPy buffer handoff.""" + if decision.transfer is TransferMode.COPY_RETURN: + return _array_replacement_boundary_blockers(argument, decision) blockers = [] if decision.owner is not OwnershipOwner.CALLER: blockers.append(f"argument {argument.name!r} array owner is {decision.owner.value}, not caller") @@ -3687,6 +3861,37 @@ def _array_storage_boundary_blockers( return tuple(blockers) +def _array_replacement_boundary_blockers( + argument: models.SemanticArgument, + decision: OwnershipDecision, +) -> tuple[str, ...]: + """Require an immutable input copied into one Python-owned replacement.""" + blockers = [] + if decision.owner is not OwnershipOwner.PYTHON: + blockers.append(f"argument {argument.name!r} replacement owner is {decision.owner.value}, not python") + if decision.destruction is not DestructionPolicy.PYTHON_REFCOUNT: + blockers.append( + f"argument {argument.name!r} replacement destruction is {decision.destruction.value}, not python_refcount" + ) + if decision.codegen_action is not CodegenAction.COPY_IN_OUT: + blockers.append( + f"argument {argument.name!r} replacement action is {decision.codegen_action.value}, not copy_in_out" + ) + if decision.storage_mode is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} replacement storage is {decision.storage_mode.value}, not stack") + if (decision.boundary_storage_mode or decision.storage_mode) is not StorageMode.STACK: + blockers.append(f"argument {argument.name!r} replacement boundary storage is not stack") + if decision.python_barrier_action is not PythonBarrierAction.ARRAY_STORAGE: + blockers.append(f"argument {argument.name!r} replacement is not sourced from array storage") + if decision.native_barrier_action is not NativeBarrierAction.PASS_ARRAY_BUFFER: + blockers.append(f"argument {argument.name!r} replacement does not pass an array buffer") + if not decision.projects_result: + blockers.append(f"argument {argument.name!r} replacement does not project a Python result") + if argument.optional or decision.nullable or decision.descriptor_boundary: + blockers.append(f"argument {argument.name!r} replacement requires nonoptional ordinary array storage") + return tuple(blockers) + + def _raw_array_address_boundary_blockers( argument: models.SemanticArgument, decision: OwnershipDecision, @@ -3941,7 +4146,10 @@ def _result_blockers(semantic_type: models.SemanticType, decision: OwnershipDeci return _derived_result_blockers(semantic_type, decision, "result") if _is_scalar_descriptor_result_type(semantic_type): return _scalar_descriptor_result_blockers(semantic_type, decision, "result") - if native_array_descriptor_kind(semantic_type) is not None: + descriptor_kind = native_array_descriptor_kind(semantic_type) + if descriptor_kind == "pointer": + return ("pointer handle results need stable owner storage and target lifetime policy before wrapping",) + if descriptor_kind is not None: return _native_array_handle_result_blockers(decision, "result") if _is_phase6_ordinary_array_type(semantic_type): return _ordinary_array_result_blockers(semantic_type, decision, "result") @@ -4015,7 +4223,10 @@ def _hidden_result_blockers( f"hidden result {argument.name!r}", mapping, ) - if native_array_descriptor_kind(argument.semantic_type) is not None: + descriptor_kind = native_array_descriptor_kind(argument.semantic_type) + if descriptor_kind == "pointer": + return ("pointer handle results need stable owner storage and target lifetime policy before wrapping",) + if descriptor_kind is not None: return _native_array_handle_result_blockers(decision, f"hidden result {argument.name!r}") if _is_phase6_ordinary_array_type(argument.semantic_type): return _ordinary_array_hidden_result_blockers(argument, decision, mapping) @@ -4282,13 +4493,6 @@ def _fixed_string_result_ownership_blockers( return tuple(blockers) -def _string_result_aggregation_blockers(results: tuple[ResultPolicy, ...]) -> tuple[str, ...]: - """Keep native string-allocation cleanup single-result in Phase 5B.""" - if any(result.ownership.kind is ObjectKind.STRING for result in results) and len(results) != 1: - return ("fixed string result lane requires exactly one Python-visible result",) - return () - - def _string_result_status_blockers( results: tuple[ResultPolicy, ...], status_error: NativeStatusErrorPolicy | None, @@ -4312,9 +4516,16 @@ def _string_writeback_status_blockers( return () -def _result_position_blockers(results: tuple[ResultPolicy, ...]) -> tuple[str, ...]: - """Require completed Python results to cover one contiguous order.""" - positions = tuple(result.result_position for result in results) +def _result_position_blockers( + results: tuple[ResultPolicy, ...], + arguments: list[ArgumentPolicy] | tuple[ArgumentPolicy, ...] = (), +) -> tuple[str, ...]: + """Require native results and visible writebacks to cover one public order.""" + positions = tuple(result.result_position for result in results) + tuple( + argument.result_position for argument in arguments if argument.projects_result + ) + if not positions: + return () if sorted(positions) == list(range(len(positions))) and len(set(positions)) == len(positions): return () return (f"binding result positions must cover 0..{len(positions) - 1} exactly once; received {positions}",) @@ -4679,8 +4890,12 @@ def _argument_transformation_policies( decision: OwnershipDecision, array: ArrayHandoffPolicy | None, ) -> tuple[tuple[TransformationPolicy, ...], tuple[str, ...]]: - """Complete explicit COPY_F ownership and lifecycle before planning.""" - if array is None or array.native_order == array.order: + """Complete binding-owned array copies and their lifecycle before planning.""" + if array is None: + return (), () + if decision.transfer is TransferMode.COPY_RETURN: + return _array_replacement_transformations(argument, decision, array) + if array.native_order == array.order: return (), () blockers = _copy_to_fortran_argument_blockers(argument, decision, array) if blockers: @@ -4722,6 +4937,51 @@ def _argument_transformation_policies( return tuple(transformations), () +def _array_replacement_transformations( + argument: models.SemanticArgument, + decision: OwnershipDecision, + array: ArrayHandoffPolicy, +) -> tuple[tuple[TransformationPolicy, ...], tuple[str, ...]]: + """Copy immutable storage once and publish the mutated temporary as output.""" + blockers = [] + if argument.optional or array.rank is None or argument.semantic_type.name == "String": + blockers.append(f"argument {argument.name!r} array replacement requires a required numeric fixed rank") + if decision.codegen_action is not CodegenAction.COPY_IN_OUT or not decision.projects_result: + blockers.append(f"argument {argument.name!r} array replacement has incomplete copy-out policy") + if blockers: + return (), tuple(blockers) + reason = "immutable array input uses one binding-owned mutable replacement" + return ( + ( + TransformationPolicy( + phase=WritebackPhase.COPY_IN, + layer=TransformationLayer.BINDING, + action=TransformationAction.COPY_ARRAY_REPRESENTATION, + source_representation="numpy_input", + target_representation="numpy_native_order", + reason=reason, + ), + TransformationPolicy( + phase=WritebackPhase.COPY_OUT, + layer=TransformationLayer.BINDING, + action=TransformationAction.PUBLISH_ARRAY_REPLACEMENT, + source_representation="numpy_native_order", + target_representation="python_result", + reason=reason, + ), + TransformationPolicy( + phase=WritebackPhase.CLEANUP, + layer=TransformationLayer.BINDING, + action=TransformationAction.RELEASE_TEMPORARY, + source_representation="numpy_native_order", + target_representation="released", + reason="binding releases the unpublished replacement on failure", + ), + ), + (), + ) + + def _copy_to_fortran_argument_blockers( argument: models.SemanticArgument, decision: OwnershipDecision, @@ -4766,7 +5026,7 @@ def _native_array_actual_policy( or array.rank is None or argument.optional or argument.semantic_type.name == "String" - or array.contiguous is not True + or decision.transfer is TransferMode.COPY_RETURN or decision.python_barrier_action is not PythonBarrierAction.ARRAY_STORAGE or decision.native_barrier_action is not NativeBarrierAction.PASS_ARRAY_BUFFER ): @@ -4788,7 +5048,7 @@ def _native_array_actual_policy( writable=decision.mutates_native, require_native_byte_order=True, require_aligned=True, - require_contiguous=True, + require_contiguous=array.contiguous is True, ) @@ -4947,17 +5207,9 @@ def _derived_module_variable_blockers( blockers.append("derived module object must retain its native module owner") if policy.handoff.release is not DerivedRelease.NATIVE_OWNER: blockers.append("derived module object cannot claim native destruction") - expected_storage = ( - DerivedObjectStorage.MODULE_ALLOCATABLE_TARGET - if variable.semantic_type.metadata.get("fortran_allocatable") - and (variable.semantic_type.metadata.get("fortran_target") or variable.semantic_type.metadata.get("aliased")) - else DerivedObjectStorage.MODULE_ALLOCATABLE - if variable.semantic_type.metadata.get("fortran_allocatable") - else DerivedObjectStorage.MODULE_POINTER - if variable.semantic_type.metadata.get("fortran_pointer") - else DerivedObjectStorage.MODULE_TARGET - if variable.semantic_type.metadata.get("fortran_target") or variable.semantic_type.metadata.get("aliased") - else DerivedObjectStorage.MODULE_PROXY + expected_storage = _derived_object_storage( + variable.semantic_type, + DerivedObjectOrigin.NATIVE_MODULE, ) if policy.handoff.storage is not expected_storage: blockers.append( @@ -5129,6 +5381,14 @@ def _array_argument_bridge_data_action( optional_mode: OptionalMode, ) -> tuple[BridgeDataAction, str | None]: """Complete one buffer, raw-address, or native-descriptor bridge view.""" + if ( + optional_mode is OptionalMode.REQUIRED + and decision.python_barrier_action is PythonBarrierAction.ARRAY_STORAGE + and decision.native_barrier_action is NativeBarrierAction.PASS_ARRAY_BUFFER + and decision.codegen_action is CodegenAction.COPY_IN_OUT + and decision.transfer is TransferMode.COPY_RETURN + ): + return BridgeDataAction.ASSOCIATE_VIEW, None if ( optional_mode in {OptionalMode.REQUIRED, OptionalMode.DESCRIPTOR} and decision.python_barrier_action is PythonBarrierAction.WRAPPER_INSTANCE @@ -5339,7 +5599,7 @@ def _array_handoff_policy(semantic_type: models.SemanticType) -> ArrayHandoffPol if array is None: return None assumed_rank = array.category == "assumed_rank" - rank = None if assumed_rank else int(array.rank or semantic_type.rank or 0) + rank = _array_handoff_rank(semantic_type, array.rank, assumed_rank) if rank is not None and rank <= 0: return None shape = tuple(str(item) for item in (array.shape or semantic_type.shape)) @@ -5348,19 +5608,58 @@ def _array_handoff_policy(semantic_type: models.SemanticType) -> ArrayHandoffPol rank=rank, shape=shape, axes=axes, - order="ORDER_F" if assumed_rank and array.order is None else array.order, - native_order=( - "ORDER_F" - if assumed_rank and array.order is None - else (array.copy_order if array.copy_order is not None else array.order) - ), - contiguous=True if assumed_rank and array.contiguous is None else array.contiguous, - itemsize=_character_length(semantic_type) if semantic_type.name == "String" else None, + order=_array_handoff_order(array.order, assumed_rank), + native_order=_array_handoff_native_order(array.order, array.copy_order, assumed_rank), + contiguous=_array_handoff_contiguous(array.contiguous, assumed_rank), + itemsize=_array_handoff_itemsize(semantic_type), category=array.category, extent_references=tuple(_array_extent_references(item) for item in shape), ) +def _array_handoff_rank( + semantic_type: models.SemanticType, + storage_rank: int | None, + assumed_rank: bool, +) -> int | None: + """Return the concrete rank, leaving assumed-rank selection explicit.""" + if assumed_rank: + return None + return int(storage_rank or semantic_type.rank or 0) + + +def _array_handoff_order(order: str | None, assumed_rank: bool) -> str | None: + """Default assumed-rank buffers to native Fortran layout.""" + if assumed_rank and order is None: + return "ORDER_F" + return order + + +def _array_handoff_native_order( + order: str | None, + copy_order: str | None, + assumed_rank: bool, +) -> str | None: + """Return the completed native-copy layout independently of input layout.""" + if assumed_rank and order is None: + return "ORDER_F" + return copy_order if copy_order is not None else order + + +def _array_handoff_contiguous(contiguous: bool | None, assumed_rank: bool) -> bool | None: + """Default assumed-rank handoff to one contiguous native buffer.""" + if assumed_rank and contiguous is None: + return True + return contiguous + + +def _array_handoff_itemsize(semantic_type: models.SemanticType) -> int | None: + """Carry fixed character width only for string array elements.""" + if semantic_type.name == "String": + return _character_length(semantic_type) + return None + + def _is_phase6_ordinary_array_type(semantic_type: models.SemanticType) -> bool: """Return whether one type is an ordinary non-descriptor array buffer.""" if _is_raw_array_address_type(semantic_type): diff --git a/x2py/wrapper_codegen/__init__.py b/x2py/wrapper_codegen/__init__.py index f3a164787..bde1838ff 100644 --- a/x2py/wrapper_codegen/__init__.py +++ b/x2py/wrapper_codegen/__init__.py @@ -82,7 +82,7 @@ ) from .planner import WrapperPlanner from .primitive_scalar_types import PrimitiveScalarTypeRegistry -from .source_printers import CSourcePrinter, FortranSourcePrinter +from .printers import CSourcePrinter, FortranSourcePrinter from .support import WrapperPlanSupportAnalyzer from .visitor import ClassVisitor, UnsupportedWrapperCodegenNodeError diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index e3628b555..cd06a2ad9 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -19,7 +19,7 @@ CallbackTransferAction, ClassConstructorKind, ClassMethodKind, - ClassOverloadMatchKind, + OverloadMatchKind, DerivedActualAccess, DerivedCallAction, DerivedDummyCategory, @@ -40,6 +40,7 @@ TransformationAction, TransformationLayer, WritebackPhase, + overload_builtin_scalar_family, ) from x2py.wrapper_codegen.nodes import ( CAllowThreadsBegin, @@ -73,8 +74,8 @@ CallbackHandoffPlan, CallbackTransferPlan, ClassMethodPlan, - ClassOverloadArgumentMatchPlan, - ClassOverloadPlan, + OverloadArgumentMatchPlan, + OverloadPlan, ClassSurfacePlan, DatatypeFamily, DerivedFieldPlan, @@ -139,6 +140,11 @@ def _require_variable_supported(self, variable: ModuleVariablePlan) -> None: if variable.datatype_family is not DatatypeFamily.STRING: PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) return + if variable.binding.getter_action is ModuleGetterAction.BORROWED_ARRAY_VIEW: + if variable.array is None or variable.array.rank is None: + raise ValueError(f"Unsupported C module array view for {variable.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) + return if variable.binding.getter_action is ModuleGetterAction.DERIVED_OBJECT: if variable.derived is None: raise ValueError(f"Unsupported C derived module object for {variable.owner_path!r}") @@ -291,6 +297,7 @@ def _require_argument_transformations_supported(argument: ArgumentTransferPlan) ) if transformation.action not in { TransformationAction.COPY_ARRAY_REPRESENTATION, + TransformationAction.PUBLISH_ARRAY_REPLACEMENT, TransformationAction.RELEASE_TEMPORARY, }: raise ValueError( @@ -570,7 +577,10 @@ def _require_writeback_supported(self, action: LifecycleActionPlan) -> None: if action.binding is None: return if action.object_kind is ObjectKind.NUMPY_ARRAY: - if action.binding.codegen_action is not CodegenAction.IN_PLACE_ARGUMENT: + if action.binding.codegen_action not in { + CodegenAction.COPY_IN_OUT, + CodegenAction.IN_PLACE_ARGUMENT, + }: raise ValueError(f"Unsupported C array writeback for {action.owner_path!r}") return if action.object_kind is ObjectKind.DERIVED_TYPE: @@ -599,6 +609,12 @@ def _require_derived_lifecycle_supported(action: LifecycleActionPlan) -> None: def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: """Return a complete C module and header from one shared plan.""" + self._class_python_names = { + surface.type_identity: surface.python_names[0] + for namespace in plan.namespaces + for surface in namespace.classes + if surface.python_names + } functions = tuple(function for namespace in plan.namespaces for function in self.visit(namespace)) needs_runtime = self.requires_runtime_support(plan) needs_free = self._module_needs_allocator(plan) @@ -924,7 +940,7 @@ def _callback_python_argument_nodes( case CallbackABIKind.VALUE: nodes = self._callback_scalar_value_nodes(transfer, target) case CallbackABIKind.REFERENCE: - nodes = self._callback_scalar_storage_nodes(transfer, target) + nodes = self._callback_scalar_reference_nodes(transfer, target) case CallbackABIKind.DATA_AND_SHAPE: nodes = self._callback_array_nodes(transfer, position, target) case CallbackABIKind.DATA_AND_LENGTH: @@ -953,6 +969,24 @@ def _callback_scalar_value_nodes( ), ) + def _callback_scalar_reference_nodes( + self, + transfer: CallbackTransferPlan, + target: str, + ) -> tuple[CDeclaration, ...]: + """Convert read-only reference input by value; preserve writable storage.""" + if transfer.access != "read": + return self._callback_scalar_storage_nodes(transfer, target) + scalar = PrimitiveScalarTypeRegistry.type_for(transfer.semantic_type_name) + parameter = self._callback_parameter_base_name(transfer) + return ( + CDeclaration( + target, + "PyObject *", + CodeExpression(f"{scalar.python_result_converter}(({scalar.c_spelling} *){parameter}_data)"), + ), + ) + def _callback_scalar_storage_nodes( self, transfer: CallbackTransferPlan, @@ -1146,7 +1180,8 @@ def _callback_array_result_nodes( "!PyArray_IS_F_CONTIGUOUS((PyArrayObject *)callback_result)", ] invalid.extend( - f"PyArray_DIM((PyArrayObject *)callback_result, {axis}) != (npy_intp)({extent})" + "PyArray_DIM((PyArrayObject *)callback_result, " + f"{axis}) != (npy_intp)({self._callback_extent_value_expression(callback, extent)})" for axis, extent in enumerate(shape) ) return ( @@ -1170,6 +1205,23 @@ def _callback_array_result_nodes( CReturn(CodeExpression("callback_value")), ) + def _callback_extent_value_expression( + self, + callback: CallbackHandoffPlan, + extent: str, + ) -> str: + """Spell one completed callback extent source in the flattened C ABI.""" + source = next((item for item in callback.arguments if item.name == extent), None) + if source is None: + return extent + base = self._callback_parameter_base_name(source) + if source.abi is CallbackABIKind.VALUE: + return base + if source.abi is CallbackABIKind.REFERENCE: + scalar = PrimitiveScalarTypeRegistry.type_for(source.semantic_type_name) + return f"*(({scalar.c_spelling} *){base}_data)" + raise ValueError(f"Callback extent {extent!r} in {callback.owner_path!r} is not a scalar value or reference") + def _callback_derived_result_nodes( self, callback: CallbackHandoffPlan, @@ -4217,19 +4269,20 @@ def _native_array_operation_declarations( ) -> tuple[CFunctionPrototype | CDeclaration, ...]: """Declare private operation wrappers and their callable definitions.""" declarations = [] - for variable in self._module_native_array_variables(plan): - declarations.extend( - ( + for variable in self._module_array_owner_variables(plan): + if variable.binding.getter_action is ModuleGetterAction.NATIVE_ARRAY_HANDLE: + declarations.append( CDeclaration( self._module_native_array_cache_name(variable), "static PyObject *", CodeExpression("NULL"), - ), - CDeclaration( - self._module_native_array_owner_name(variable), - "static PyObject *", - CodeExpression("NULL"), - ), + ) + ) + declarations.append( + CDeclaration( + self._module_native_array_owner_name(variable), + "static PyObject *", + CodeExpression("NULL"), ) ) if variable.native_array_handle is None: @@ -4300,6 +4353,15 @@ def _module_native_array_variables(self, plan: ModulePlan) -> tuple[ModuleVariab if variable.binding.getter_action is ModuleGetterAction.NATIVE_ARRAY_HANDLE ) + def _module_array_owner_variables(self, plan: ModulePlan) -> tuple[ModuleVariablePlan, ...]: + """Return module arrays whose Python values retain the native module.""" + return tuple( + variable + for variable in self._variables(plan) + if variable.binding.getter_action + in {ModuleGetterAction.BORROWED_ARRAY_VIEW, ModuleGetterAction.NATIVE_ARRAY_HANDLE} + ) + # Borrowed module native-array-handle operations. def _module_native_array_operation_function( self, @@ -5041,6 +5103,8 @@ def _lower_module_getter(self, plan: ModuleVariablePlan) -> tuple[CFunction, ... return self._lower_module_getter_direct_value(plan) case ModuleGetterAction.NULLABLE_SNAPSHOT: return self._lower_module_getter_nullable_snapshot(plan) + case ModuleGetterAction.BORROWED_ARRAY_VIEW: + return self._lower_module_getter_borrowed_array_view(plan) case ModuleGetterAction.NATIVE_ARRAY_HANDLE: return self._lower_module_getter_native_array_handle(plan) case ModuleGetterAction.DERIVED_OBJECT: @@ -5109,6 +5173,56 @@ def _lower_module_getter_nullable_snapshot(self, plan: ModuleVariablePlan) -> tu ), ) + def _lower_module_getter_borrowed_array_view(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: + """Create one live Fortran-ordered NumPy alias over fixed module storage.""" + array = plan.array + if array is None or array.rank is None: + raise ValueError(f"Module array view {plan.owner_path!r} has no fixed rank") + scalar = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) + owner = self._module_native_array_owner_name(plan) + extents = tuple(f"extent_{axis}" for axis in range(array.rank)) + strides = "strides" + return ( + CFunction( + self._module_getter_name(plan), + "PyObject *", + storage="static", + body=( + *(CDeclaration(name, "int64_t", CodeExpression("0")) for name in extents), + CDeclaration( + "data", + "void *", + CodeExpression( + f"{self._module_bridge_getter_name(plan)}({', '.join(f'&{name}' for name in extents)})" + ), + ), + CDeclaration( + f"dimensions[{array.rank}]", + "npy_intp", + CodeExpression("{" + ", ".join(extents) + "}"), + ), + CDeclaration(f"{strides}[{array.rank}]", "npy_intp"), + CExpressionStatement(CodeExpression(f"{strides}[0] = (npy_intp)sizeof({scalar.c_spelling})")), + *( + CExpressionStatement( + CodeExpression(f"{strides}[{axis}] = {strides}[{axis - 1}] * dimensions[{axis - 1}]") + ) + for axis in range(1, array.rank) + ), + CDeclaration( + "result", + "PyObject *", + CodeExpression( + f"PyArray_New(&PyArray_Type, {array.rank}, dimensions, {scalar.numpy_type_macro}, " + f"{strides}, data, 0, NPY_ARRAY_F_CONTIGUOUS | NPY_ARRAY_ALIGNED | " + "NPY_ARRAY_WRITEABLE, NULL)" + ), + ), + *self._ordinary_array_field_owner_nodes("result", owner), + ), + ), + ) + def _lower_module_getter_native_array_handle(self, plan: ModuleVariablePlan) -> tuple[CFunction, ...]: """Create one stable borrowed runtime handle from planned module operations.""" handle = plan.native_array_handle @@ -6093,8 +6207,8 @@ def _lower_argument_required_array_storage( nodes = [ *self._ordinary_array_argument_declarations(plan, names), self._array_type_and_rank_check(plan, names, array), - *self._array_access_checks(plan, array), *self._array_layout_checks(plan, array), + *self._array_access_checks(plan, array), *self._array_shape_checks(plan, context, array), ] nodes.extend(self._array_extraction_nodes(plan, names, array)) @@ -6149,6 +6263,24 @@ def _lower_argument_required_array_actual( CDeclaration(names.object_name, "PyObject *"), CDeclaration(names.value_name, "void *", CodeExpression("NULL")), *(CDeclaration(name, "int64_t", CodeExpression("0")) for name in names.extent_names), + *( + CDeclaration(name, "int64_t", CodeExpression("0")) + for name in names.upper_bound_names[: len(array.upper_bound_roles)] + ), + *( + CDeclaration(name, "int64_t", CodeExpression("1")) + for name in names.stride_names[: len(array.stride_roles)] + ), + *( + (CDeclaration(names.runtime_rank_name, "int64_t", CodeExpression("0")),) + if array.runtime_rank_role is not None + else () + ), + *( + (CDeclaration(names.itemsize_name, "int64_t", CodeExpression("0")),) + if array.itemsize_role is not None + else () + ), CDeclaration(f"{prefix}_runtime", "PyObject *", CodeExpression("NULL")), CDeclaration(f"{prefix}_helper", "PyObject *", CodeExpression("NULL")), CDeclaration(f"{prefix}_shape", "PyObject *", CodeExpression("NULL")), @@ -6192,7 +6324,9 @@ def _native_array_actual_call_nodes( f'{prefix}_packed = PyObject_CallFunction({prefix}_helper, "OsiOOiiiiiii", ' f'{names.object_name}, "{actual.dtype}", {actual.rank}, {prefix}_shape, {prefix}_layout, ' f"{int(actual.writable)}, {int(actual.require_native_byte_order)}, {int(actual.require_aligned)}, " - f"0, 0, 0, {int(actual.require_contiguous)})" + f"{int(plan.array.runtime_rank_role is not None)}, " + f"{int(plan.array.itemsize_role is not None)}, {int(bool(plan.array.stride_roles))}, " + f"{int(actual.require_contiguous)})" ) ), CExpressionStatement(CodeExpression(f"Py_DECREF({prefix}_helper)")), @@ -6263,7 +6397,7 @@ def _native_array_actual_unpack_nodes( plan: ArgumentTransferPlan, names: _CArgumentNames, ) -> tuple[CExpressionStatement, ...]: - """Unpack exactly the existing Phase 6 pointer/extent fields.""" + """Unpack the completed pointer, rank, itemsize, and axis fact roles.""" prefix = names.value_name nodes = [ CExpressionStatement( @@ -6276,12 +6410,20 @@ def _native_array_actual_unpack_nodes( ) ), ] - for axis, extent_name in enumerate(names.extent_names): + position = 1 + array = plan.array + if array is None: + raise ValueError(f"Array actual {plan.owner_path!r} is missing its handoff") + scalar_fields = ( + *((names.runtime_rank_name,) if array.runtime_rank_role is not None else ()), + *((names.itemsize_name,) if array.itemsize_role is not None else ()), + ) + for field_name in scalar_fields: nodes.extend( ( CExpressionStatement( CodeExpression( - f"{extent_name} = (int64_t)PyLong_AsLongLong(PyTuple_GetItem({prefix}_packed, {axis + 1}))" + f"{field_name} = (int64_t)PyLong_AsLongLong(PyTuple_GetItem({prefix}_packed, {position}))" ) ), CExpressionStatement( @@ -6289,6 +6431,26 @@ def _native_array_actual_unpack_nodes( ), ) ) + position += 1 + axis_fields = ( + *names.extent_names, + *names.upper_bound_names[: len(array.upper_bound_roles)], + *names.stride_names[: len(array.stride_roles)], + ) + for field_name in axis_fields: + nodes.extend( + ( + CExpressionStatement( + CodeExpression( + f"{field_name} = (int64_t)PyLong_AsLongLong(PyTuple_GetItem({prefix}_packed, {position}))" + ) + ), + CExpressionStatement( + CodeExpression(f"if (PyErr_Occurred()) {{ Py_DECREF({prefix}_packed); return NULL; }}") + ), + ) + ) + position += 1 nodes.append(CExpressionStatement(CodeExpression(f"Py_DECREF({prefix}_packed)"))) return tuple(nodes) @@ -7133,6 +7295,8 @@ def _lower_argument_nullable_value( raise ValueError( f"Unsupported optional C argument object kind for {plan.owner_path!r}: {plan.object_kind!r}" ) + if plan.binding.python_action is PythonBarrierAction.SCALAR_STORAGE: + return self._lower_argument_nullable_scalar_storage(plan, context) scalar_type = PrimitiveScalarTypeRegistry.type_for(plan.semantic_type_name) names = context.arguments[plan.owner_path] return ( @@ -7150,6 +7314,26 @@ def _lower_argument_nullable_value( ), ) + def _lower_argument_nullable_scalar_storage( + self, + plan: ArgumentTransferPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CIf, ...]: + """Preserve absent rank-zero storage or borrow one present NumPy cell.""" + names = context.arguments[plan.owner_path] + required = self._lower_argument_required_scalar_storage(plan, context) + declarations = tuple(node for node in required if isinstance(node, CDeclaration)) + body = tuple(node for node in required if not isinstance(node, CDeclaration)) + return ( + CDeclaration(names.object_name, "PyObject *", CodeExpression("Py_None")), + *(node for node in declarations if node.name != names.object_name), + CDeclaration(names.nullable_name, "void *", CodeExpression("NULL")), + CIf( + CodeExpression(f"{names.object_name} != Py_None"), + body=(*body, CExpressionStatement(CodeExpression(f"{names.nullable_name} = {names.value_name}"))), + ), + ) + # Optional ordinary-array lowering. def _lower_argument_nullable_array_storage( self, @@ -7955,45 +8139,62 @@ def _output_nodes( *self._binding_transformation_post_call_nodes(plan, context), *self._lower_status_error(plan, context), ] - if plan.results and plan.writeback_actions: - nodes.extend(self._mixed_string_output_nodes(plan, context)) - elif plan.results: - nodes.extend(self._binding_result_nodes(plan, context)) - elif plan.writeback_actions: - nodes.extend(self._writeback_nodes(plan, context)) + if plan.results or plan.writeback_actions: + nodes.extend(self._combined_output_nodes(plan, context)) else: nodes.append(CExpressionStatement(CodeExpression("Py_RETURN_NONE"))) return tuple(nodes) - def _mixed_string_output_nodes( + def _combined_output_nodes( self, plan: FunctionPlan, context: _CFunctionContext, - ) -> tuple: - """Convert the legacy-observed hidden/output string pair in one order.""" + ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: + """Convert every public output once, then aggregate by completed position.""" + published, ordinary_writebacks, derived_results, scalar_results = self._output_conversion_groups(plan) + converted: list[str] = [] nodes = [] - converted = [] - for result in sorted(plan.results, key=lambda item: item.result_position): + + # Published temporaries are converted first so every later failure owns + # an ordinary Python reference that can be released uniformly. + for action in published: + nodes.extend(self._writeback_value_nodes(plan, action, context, tuple(converted))) + converted.append(context.python_results[action.owner_path]) + + for position, result in enumerate(derived_results): + pending = self._derived_native_storage_cleanup_nodes(derived_results[position + 1 :], context) + nodes.extend(self._lower_result_derived(result, context, tuple(converted), pending)) + converted.append(context.python_results[result.owner_path]) + + for result in scalar_results: nodes.extend(self.visit(result, context=context, failure_cleanup=tuple(converted))) converted.append(context.python_results[result.owner_path]) - for action in self._ordered_output_writebacks(plan): - nodes.extend(self._mixed_string_writeback_nodes(plan, action, context, tuple(converted))) + + for action in ordinary_writebacks: + nodes.extend(self._writeback_value_nodes(plan, action, context, tuple(converted))) converted.append(context.python_results[action.owner_path]) - ordered = tuple( - name - for _position, name in sorted( - ( - *((result.result_position, context.python_results[result.owner_path]) for result in plan.results), - *( - (action.binding.result_position, context.python_results[action.owner_path]) - for action in self._ordered_output_writebacks(plan) - ), - ) - ) - ) + + ordered = tuple(context.python_results[owner] for owner, _position in self._output_owners(plan)) nodes.extend(self._python_result_aggregation_nodes(ordered, context)) return tuple(nodes) + def _output_conversion_groups( + self, + plan: FunctionPlan, + ) -> tuple[ + tuple[LifecycleActionPlan, ...], + tuple[LifecycleActionPlan, ...], + tuple[ResultPlan, ...], + tuple[ResultPlan, ...], + ]: + """Partition completed outputs into their ordered conversion leaves.""" + writebacks = self._ordered_output_writebacks(plan) + published = tuple(action for action in writebacks if self._publishes_array_replacement(plan, action)) + ordinary = tuple(action for action in writebacks if action not in published) + derived = tuple(result for result in plan.results if result.object_kind is ObjectKind.DERIVED_TYPE) + scalar = tuple(result for result in plan.results if result.object_kind is not ObjectKind.DERIVED_TYPE) + return published, ordinary, derived, scalar + def _mixed_string_writeback_nodes( self, plan: FunctionPlan, @@ -8222,45 +8423,6 @@ def _one_owned_deferred_character_materialization( ), ) - def _binding_result_nodes( - self, - plan: FunctionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: - """Convert ordered results and assemble a tuple only in the binding.""" - ordered = tuple(sorted(plan.results, key=lambda item: item.result_position)) - conversion_order = ( - *(result for result in ordered if result.object_kind is ObjectKind.DERIVED_TYPE), - *(result for result in ordered if result.object_kind is not ObjectKind.DERIVED_TYPE), - ) - converted = [] - nodes = [] - for position, result in enumerate(conversion_order): - if result.object_kind is ObjectKind.DERIVED_TYPE: - pending = tuple( - candidate - for candidate in conversion_order[position + 1 :] - if candidate.object_kind is ObjectKind.DERIVED_TYPE - ) - nodes.extend( - self._lower_result_derived( - result, - context, - tuple(converted), - self._derived_native_storage_cleanup_nodes(pending, context), - ) - ) - else: - nodes.extend(self.visit(result, context=context, failure_cleanup=tuple(converted))) - converted.append(context.python_results[result.owner_path]) - nodes.extend( - self._python_result_aggregation_nodes( - tuple(context.python_results[result.owner_path] for result in ordered), - context, - ) - ) - return tuple(nodes) - def _derived_result_allocation_failure_nodes( self, plan: FunctionPlan, @@ -8416,6 +8578,7 @@ def _lower_status_error_runtime_error( tuple(result for result in plan.results if result.object_kind is ObjectKind.DERIVED_TYPE), context, ) + transformation_cleanup = self._binding_transformation_cleanup_nodes(plan, context) if policy.message_role is None: return ( CIf( @@ -8427,6 +8590,7 @@ def _lower_status_error_runtime_error( f"(int){status_name})" ) ), + *transformation_cleanup, *derived_cleanup, CReturn(CodeExpression("NULL")), ), @@ -8439,6 +8603,7 @@ def _lower_status_error_runtime_error( CodeExpression(f"{message_name} == NULL"), body=( CExpressionStatement(CodeExpression("PyErr_NoMemory()")), + *transformation_cleanup, *derived_cleanup, CReturn(CodeExpression("NULL")), ), @@ -8451,13 +8616,14 @@ def _lower_status_error_runtime_error( CExpressionStatement(CodeExpression(f"free({message_name})")), CIf( CodeExpression(f"{message_object} == NULL"), - body=(*derived_cleanup, CReturn(CodeExpression("NULL"))), + body=(*transformation_cleanup, *derived_cleanup, CReturn(CodeExpression("NULL"))), ), CIf( condition, body=( CExpressionStatement(CodeExpression(f"PyErr_SetObject(PyExc_RuntimeError, {message_object})")), CExpressionStatement(CodeExpression(f"Py_DECREF({message_object})")), + *transformation_cleanup, *derived_cleanup, CReturn(CodeExpression("NULL")), ), @@ -8465,67 +8631,104 @@ def _lower_status_error_runtime_error( CExpressionStatement(CodeExpression(f"Py_DECREF({message_object})")), ) - def _writeback_nodes( + def _writeback_value_nodes( self, plan: FunctionPlan, + action: LifecycleActionPlan, context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return the sole replacement conversion after the native call.""" - ordered = sorted( - ( - action - for action in plan.writeback_actions - if action.phase is WritebackPhase.COPY_OUT and action.binding is not None + converted: tuple[str, ...], + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Convert one planned writeback without terminating output aggregation.""" + if action.binding is None: + raise ValueError(f"Writeback {action.owner_path!r} has no binding policy") + source = self._argument_for_role(plan, action.source_role) + if self._publishes_array_replacement(plan, action): + return self._array_replacement_writeback_nodes(source, action, context) + if action.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT: + return self._identity_writeback_value_nodes(source, action, context, converted) + if action.binding.codegen_action is CodegenAction.COPY_IN_OUT: + if action.binding.datatype_family is DatatypeFamily.STRING: + return self._mixed_string_writeback_nodes(plan, action, context, converted) + return self._scalar_writeback_value_nodes(source, action, context, converted) + raise ValueError(f"Unsupported C writeback action for {action.owner_path!r}: {action.binding.codegen_action!r}") + + def _identity_writeback_value_nodes( + self, + source: ArgumentTransferPlan, + action: LifecycleActionPlan, + context: _CFunctionContext, + converted: tuple[str, ...], + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Retain the exact mutable Python object selected by completed policy.""" + target = context.python_results[action.owner_path] + if source.derived_call is not None and source.derived_call.writeback in { + DerivedWriteback.ALLOCATION_STATE, + DerivedWriteback.POINTER_ASSOCIATION, + }: + return self._holder_writeback_value_nodes(source, target, converted, context) + source_object = context.arguments[source.owner_path].object_name + return ( + CDeclaration(target, "PyObject *", CodeExpression(source_object)), + CExpressionStatement(CodeExpression(f"Py_INCREF({target})")), + ) + + def _array_replacement_writeback_nodes( + self, + source: ArgumentTransferPlan, + action: LifecycleActionPlan, + context: _CFunctionContext, + ) -> tuple[CDeclaration | CExpressionStatement, ...]: + """Transfer one binding-owned mutable NumPy replacement to Python.""" + names = context.arguments[source.owner_path] + temporary = self._array_transformation_temp_name(names) + target = context.python_results[action.owner_path] + return ( + CDeclaration(target, "PyObject *", CodeExpression(temporary)), + CExpressionStatement(CodeExpression(f"{temporary} = NULL")), + ) + + def _scalar_writeback_value_nodes( + self, + source: ArgumentTransferPlan, + action: LifecycleActionPlan, + context: _CFunctionContext, + converted: tuple[str, ...], + ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: + """Convert one mutated scalar storage value for combined aggregation.""" + names = context.arguments[source.owner_path] + scalar_type = PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) + target = context.python_results[action.owner_path] + cleanup = tuple(CExpressionStatement(CodeExpression(f"Py_DECREF({name})")) for name in converted) + conversion = CExpressionStatement( + CodeExpression(f"{target} = {scalar_type.python_result_converter}(&{names.value_name})") + ) + failure = CIf(CodeExpression(f"{target} == NULL"), body=(*cleanup, CReturn(CodeExpression("NULL")))) + if source.bridge.descriptor_output_presence_role is None: + return (CDeclaration(target, "PyObject *", CodeExpression("NULL")), conversion, failure) + return ( + CDeclaration(target, "PyObject *", CodeExpression("NULL")), + CIf( + CodeExpression(f"!{self._descriptor_output_present_name(names)}"), + body=( + CExpressionStatement(CodeExpression("Py_INCREF(Py_None)")), + CExpressionStatement(CodeExpression(f"{target} = Py_None")), + ), + else_body=(conversion, failure), ), - key=lambda item: item.binding.result_position, ) - if ordered and all( - action.object_kind in {ObjectKind.NUMPY_ARRAY, ObjectKind.DERIVED_TYPE} - and action.binding.codegen_action is CodegenAction.IN_PLACE_ARGUMENT - for action in ordered - ): - return self._in_place_argument_writeback_nodes(plan, tuple(ordered), context) - if len(ordered) != 1: - raise ValueError(f"{plan.owner_path!r} requires exactly one writeback result") - action = ordered[0] - return self._lower_writeback(plan, action, context) - # Array and derived-object identity writeback lowering. - def _in_place_argument_writeback_nodes( + def _publishes_array_replacement( self, plan: FunctionPlan, - actions: tuple[LifecycleActionPlan, ...], - context: _CFunctionContext, - ) -> tuple[CDeclaration | CExpressionStatement | CIf | CReturn, ...]: - """Convert every in-place result once, then aggregate in result order.""" - nodes = [] - converted = [] - for action in actions: - source = self._argument_for_role(plan, action.source_role) - names = context.arguments[source.owner_path] - python_name = context.python_results[action.owner_path] - if source.derived_call is not None and source.derived_call.writeback in { - DerivedWriteback.ALLOCATION_STATE, - DerivedWriteback.POINTER_ASSOCIATION, - }: - nodes.extend( - self._holder_writeback_value_nodes( - source, - python_name, - tuple(converted), - context, - ) - ) - else: - nodes.extend( - ( - CDeclaration(python_name, "PyObject *", CodeExpression(names.object_name)), - CExpressionStatement(CodeExpression(f"Py_INCREF({python_name})")), - ) - ) - converted.append(python_name) - nodes.extend(self._python_result_aggregation_nodes(tuple(converted), context)) - return tuple(nodes) + action: LifecycleActionPlan, + ) -> bool: + """Return whether completed COPY_OUT policy transfers a NumPy temporary.""" + source = self._argument_for_role(plan, action.source_role) + return any( + transformation.phase is WritebackPhase.COPY_OUT + and transformation.action is TransformationAction.PUBLISH_ARRAY_REPLACEMENT + for transformation in source.transformations + ) def _holder_writeback_value_nodes( self, @@ -8578,179 +8781,6 @@ def _holder_writeback_value_nodes( ), ) - def _lower_writeback( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Dispatch one completed binding writeback action explicitly.""" - codegen_action = action.binding.codegen_action - match codegen_action: - case CodegenAction.COPY_IN_OUT: - return self._lower_writeback_copy_in_out(plan, action, context) - case CodegenAction.IN_PLACE_ARGUMENT: - return self._lower_writeback_in_place_argument(plan, action, context) - raise ValueError(f"Unsupported C writeback action for {action.owner_path!r}: {codegen_action!r}") - - def _lower_writeback_copy_in_out( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - if action.binding.datatype_family is DatatypeFamily.STRING: - return self._lower_writeback_string(plan, action, context) - return self._lower_writeback_value(plan, action, context) - - # String writeback lowering. - def _lower_writeback_string( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: - """Dispatch string replacement conversion from completed presence mode.""" - source = self._argument_for_role(plan, action.source_role) - if source.binding.optional_mode is OptionalMode.REQUIRED: - return self._lower_writeback_required_string(source, context) - if source.binding.optional_mode is OptionalMode.NULLABLE_VALUE: - return self._lower_writeback_optional_string(source, context) - raise ValueError(f"Unsupported string writeback presence mode for {source.owner_path!r}") - - def _lower_writeback_required_string( - self, - source: ArgumentTransferPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CIf | CReturn, ...]: - """Convert one required mutable buffer and release it exactly once.""" - names = context.arguments[source.owner_path] - python_result_name = context.python_result_name or "result_obj" - return ( - CDeclaration( - python_result_name, - "PyObject *", - CodeExpression(f'Py_BuildValue("s", (const char *){names.value_name})'), - ), - CExpressionStatement(CodeExpression(f"free({names.value_name})")), - CIf( - CodeExpression(f"{python_result_name} == NULL"), - body=(CReturn(CodeExpression("NULL")),), - ), - CReturn(CodeExpression(python_result_name)), - ) - - def _lower_writeback_optional_string( - self, - source: ArgumentTransferPlan, - context: _CFunctionContext, - ) -> tuple[CDeclaration | CIf | CReturn, ...]: - """Return None for absence or convert and release one concrete replacement.""" - names = context.arguments[source.owner_path] - python_result_name = context.python_result_name or "result_obj" - return ( - CDeclaration(python_result_name, "PyObject *", CodeExpression("NULL")), - CIf( - CodeExpression(f"{names.value_name} == NULL"), - body=( - CExpressionStatement(CodeExpression("Py_INCREF(Py_None)")), - CExpressionStatement(CodeExpression(f"{python_result_name} = Py_None")), - ), - else_body=( - CExpressionStatement( - CodeExpression(f'{python_result_name} = Py_BuildValue("s", (const char *){names.value_name})') - ), - CExpressionStatement(CodeExpression(f"free({names.value_name})")), - CIf( - CodeExpression(f"{python_result_name} == NULL"), - body=(CReturn(CodeExpression("NULL")),), - ), - ), - ), - CReturn(CodeExpression(python_result_name)), - ) - - def _lower_writeback_in_place_argument( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - if action.object_kind is ObjectKind.DERIVED_TYPE: - return self._lower_writeback_derived_identity(plan, action, context) - return self._lower_writeback_value(plan, action, context) - - # Derived-object writeback lowering. - def _lower_writeback_derived_identity( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CDeclaration | CExpressionStatement | CReturn, ...]: - """Return the caller's exact wrapper after native in-place mutation.""" - source = self._argument_for_role(plan, action.source_role) - names = context.arguments[source.owner_path] - python_result_name = context.python_results[action.owner_path] - return ( - CDeclaration(python_result_name, "PyObject *", CodeExpression(names.object_name)), - CExpressionStatement(CodeExpression(f"Py_INCREF({python_result_name})")), - CReturn(CodeExpression(python_result_name)), - ) - - # Scalar writeback lowering. - def _lower_writeback_value( - self, - plan: FunctionPlan, - action: LifecycleActionPlan, - context: _CFunctionContext, - ) -> tuple[CExpressionStatement | CDeclaration | CReturn, ...]: - """Return one Python scalar replacement from mutated bridge storage.""" - source = self._argument_for_role(plan, action.source_role) - names = context.arguments[source.owner_path] - scalar_type = PrimitiveScalarTypeRegistry.type_for(action.binding.semantic_type_name) - python_result_name = context.python_result_name or "result_obj" - if source.bridge.descriptor_output_presence_role is not None: - return self._lower_descriptor_writeback_value(names, scalar_type, python_result_name) - return ( - CDeclaration( - python_result_name, - "PyObject *", - CodeExpression(f"{scalar_type.python_result_converter}(&{names.value_name})"), - ), - CExpressionStatement(CodeExpression(f"if ({python_result_name} == NULL) return NULL")), - CReturn(CodeExpression(python_result_name)), - ) - - def _lower_descriptor_writeback_value( - self, - names: _CArgumentNames, - scalar_type, - python_result_name: str, - ) -> tuple[CDeclaration | CIf | CReturn, ...]: - """Return None or one copied scalar from a mutated required descriptor.""" - return ( - CDeclaration(python_result_name, "PyObject *", CodeExpression("NULL")), - CIf( - CodeExpression(f"!{self._descriptor_output_present_name(names)}"), - body=( - CExpressionStatement(CodeExpression("Py_INCREF(Py_None)")), - CExpressionStatement(CodeExpression(f"{python_result_name} = Py_None")), - ), - else_body=( - CExpressionStatement( - CodeExpression( - f"{python_result_name} = {scalar_type.python_result_converter}(&{names.value_name})" - ) - ), - CIf( - CodeExpression(f"{python_result_name} == NULL"), - body=(CReturn(CodeExpression("NULL")),), - ), - ), - ), - CReturn(CodeExpression(python_result_name)), - ) - @staticmethod def _descriptor_output_present_name(names: _CArgumentNames) -> str: """Name the binding-local final descriptor-state flag.""" @@ -9114,11 +9144,12 @@ def _binding_transformation_post_call_nodes( plan: FunctionPlan, context: _CFunctionContext, ) -> tuple[CExpressionStatement | CIf, ...]: - """Perform planned copy-out actions, then release every binding temporary.""" + """Copy back ordinary temporaries and retain published replacements.""" nodes = [] cleanup = self._binding_transformation_cleanup_nodes(plan, context) for argument in plan.arguments: - if not self._has_transformation_phase(argument, WritebackPhase.COPY_OUT): + action = self._transformation_action(argument, WritebackPhase.COPY_OUT) + if action is not TransformationAction.COPY_ARRAY_REPRESENTATION: continue names = context.arguments[argument.owner_path] temporary = self._array_transformation_temp_name(names) @@ -9130,9 +9161,27 @@ def _binding_transformation_post_call_nodes( body=(*cleanup, CReturn(CodeExpression("NULL"))), ) ) - nodes.extend(cleanup) + nodes.extend(self._binding_transformation_success_cleanup_nodes(plan, context)) return tuple(nodes) + def _binding_transformation_success_cleanup_nodes( + self, + plan: FunctionPlan, + context: _CFunctionContext, + ) -> tuple[CExpressionStatement, ...]: + """Release temporaries whose successful path does not publish ownership.""" + return tuple( + CExpressionStatement( + CodeExpression( + f"Py_XDECREF({self._array_transformation_temp_name(context.arguments[item.owner_path])})" + ) + ) + for item in reversed(plan.arguments) + if self._has_transformation_phase(item, WritebackPhase.CLEANUP) + and self._transformation_action(item, WritebackPhase.COPY_OUT) + is not TransformationAction.PUBLISH_ARRAY_REPLACEMENT + ) + def _binding_transformation_cleanup_nodes( self, plan: FunctionPlan, @@ -9154,6 +9203,19 @@ def _has_transformation_phase(argument: ArgumentTransferPlan, phase: WritebackPh """Return whether one completed transfer owns an action in a lifecycle phase.""" return any(transformation.phase is phase for transformation in argument.transformations) + @staticmethod + def _transformation_action( + argument: ArgumentTransferPlan, + phase: WritebackPhase, + ) -> TransformationAction | None: + """Return the sole completed transformation action for one lifecycle phase.""" + actions = tuple( + transformation.action for transformation in argument.transformations if transformation.phase is phase + ) + if len(actions) > 1: + raise ValueError(f"Argument {argument.owner_path!r} has repeated {phase.value} transformations") + return actions[0] if actions else None + @staticmethod def _array_transformation_temp_name(names: _CArgumentNames) -> str: """Name the binding-owned NumPy representation temporary.""" @@ -9568,17 +9630,48 @@ def _module_variable_bridge_prototypes( plan: ModuleVariablePlan, ) -> tuple[CFunctionPrototype, ...]: """Return getter/setter ABI declarations selected by the variable plan.""" - if plan.binding.getter_action is ModuleGetterAction.NATIVE_ARRAY_HANDLE: - return self._module_native_array_bridge_prototypes(plan) - if plan.binding.getter_action is ModuleGetterAction.DERIVED_OBJECT: - if plan.derived is None: - return () - if plan.derived.access is ModuleObjectAccessMechanism.MEMBER_PROXY: - prototypes = [] - if self._nullable_derived_module_proxy(plan): - prototypes.append(CFunctionPrototype(self._module_derived_presence_bridge_name(plan), "bool")) - return tuple(prototypes) + handler = { + ModuleGetterAction.NATIVE_ARRAY_HANDLE: self._module_native_array_bridge_prototypes, + ModuleGetterAction.BORROWED_ARRAY_VIEW: self._module_borrowed_array_bridge_prototypes, + ModuleGetterAction.DERIVED_OBJECT: self._module_derived_bridge_prototypes, + }.get(plan.binding.getter_action) + if handler is not None: + return handler(plan) + return self._module_scalar_bridge_prototypes(plan) + + def _module_borrowed_array_bridge_prototypes( + self, + plan: ModuleVariablePlan, + ) -> tuple[CFunctionPrototype, ...]: + """Declare one borrowed array getter with explicit extent outputs.""" + if plan.array is None or plan.array.rank is None: + return () + return ( + CFunctionPrototype( + self._module_bridge_getter_name(plan), + "void *", + tuple(CParameter(f"extent_{axis}", "int64_t *") for axis in range(plan.array.rank)), + ), + ) + + def _module_derived_bridge_prototypes( + self, + plan: ModuleVariablePlan, + ) -> tuple[CFunctionPrototype, ...]: + """Declare the selected direct or member-proxy derived getter ABI.""" + if plan.derived is None: + return () + if plan.derived.access is not ModuleObjectAccessMechanism.MEMBER_PROXY: return (CFunctionPrototype(self._module_bridge_getter_name(plan), "void *"),) + if self._nullable_derived_module_proxy(plan): + return (CFunctionPrototype(self._module_derived_presence_bridge_name(plan), "bool"),) + return () + + def _module_scalar_bridge_prototypes( + self, + plan: ModuleVariablePlan, + ) -> tuple[CFunctionPrototype, ...]: + """Declare ordinary scalar getter and setter bridge functions.""" prototypes = [] if plan.bridge.getter_role is not None: return_type = ( @@ -10244,26 +10337,26 @@ def _namespace_configuration_nodes( ) return ( *property_nodes, - *self._derived_type_initializer_nodes(namespace, object_name), + *self._namespace_python_initializer_nodes(namespace, object_name), *self._module_native_array_owner_nodes(namespace, object_name), *self._derived_module_owner_nodes(namespace, object_name), *self._module_initializer_nodes(namespace), *self._module_constant_nodes(namespace, object_name), ) - def _derived_type_initializer_nodes( + def _namespace_python_initializer_nodes( self, namespace: NamespacePlan, module_object: str, ) -> tuple[CDeclaration | CExpressionStatement | CIf, ...]: - """Install minimal non-constructible opaque Python wrapper types.""" + """Install exact overload dispatch plus generated opaque wrapper types.""" has_proxy = any(variable.derived is not None for variable in namespace.variables) - if not namespace.derived_types and not has_proxy: + if not namespace.derived_types and not has_proxy and not namespace.overloads: return () - source = self._derived_namespace_python_source(namespace) + source = self._namespace_python_source(namespace) literal = self._c_string_literal(source) - result_name = f"{self._namespace_symbol(namespace)}_derived_setup" - dictionary = f"{self._namespace_symbol(namespace)}_derived_dict" + result_name = f"{self._namespace_symbol(namespace)}_python_setup" + dictionary = f"{self._namespace_symbol(namespace)}_python_dict" return ( CDeclaration(dictionary, "PyObject *", CodeExpression(f"PyModule_GetDict({module_object})")), CIf(CodeExpression(f"{dictionary} == NULL"), body=(CReturn(CodeExpression("NULL")),)), @@ -10276,8 +10369,8 @@ def _derived_type_initializer_nodes( CExpressionStatement(CodeExpression(f"Py_DECREF({result_name})")), ) - def _derived_namespace_python_source(self, namespace: NamespacePlan) -> str: - """Return opaque classes plus typed direct/module member operation maps.""" + def _namespace_python_source(self, namespace: NamespacePlan) -> str: + """Return overloads, opaque classes, and typed member operation maps.""" surfaces = {surface.type_identity: surface for surface in namespace.classes} class_names = { surface.type_identity: surface.python_names[0] for surface in namespace.classes if surface.python_names @@ -10286,6 +10379,7 @@ def _derived_namespace_python_source(self, namespace: NamespacePlan) -> str: sections = [ "_x2py_unset = object()", "import numpy as _x2py_numpy", + *(self._module_overload_python_source(overload) for overload in namespace.overloads), *( self._derived_type_python_source( derived, @@ -10373,7 +10467,7 @@ def _class_method_python_source_lines(self, methods: tuple[ClassMethodPlan, ...] """Flatten public method descriptors while preserving plan order.""" return tuple(line for method in methods for line in self._class_method_python_lines(method)) - def _class_overload_python_source_lines(self, overloads: tuple[ClassOverloadPlan, ...]) -> tuple[str, ...]: + def _class_overload_python_source_lines(self, overloads: tuple[OverloadPlan, ...]) -> tuple[str, ...]: """Flatten overload descriptors while preserving plan order.""" return tuple(line for overload in overloads for line in self._class_overload_python_lines(overload)) @@ -10418,6 +10512,7 @@ def _absent_constructor_python_lines(surface: ClassSurfacePlan | None) -> tuple[ ) return ( " def __new__(cls, *args, **kwargs):", + f" {surface.constructor.docstring!r}" if surface is not None else " 'Construction disabled.'", f" raise TypeError({message!r})", ) @@ -10430,6 +10525,7 @@ def _default_constructor_python_lines(self, surface: ClassSurfacePlan) -> tuple[ " def __new__(cls, *args, **kwargs):", f" return {self._class_create_method_name(surface)}()", f" def __init__(self{signature}):", + f" {surface.constructor.docstring!r}", ] if not fields: lines.append(" pass") @@ -10452,6 +10548,7 @@ def _bound_constructor_python_lines(self, surface: ClassSurfacePlan) -> tuple[st " def __new__(cls, *args, **kwargs):", f" return {self._class_create_method_name(surface)}()", f" def __init__(self{self._python_parameter_suffix(parameters)}):", + f" {surface.constructor.docstring!r}", " _x2py_arguments = {'self': self}", ] lines.extend(self._optional_keyword_collection_lines(parameters, indent=" ")) @@ -10466,7 +10563,11 @@ def _overloaded_constructor_python_lines(self, surface: ClassSurfacePlan) -> tup return ( " def __new__(cls, *args, **kwargs):", f" return {self._class_create_method_name(surface)}()", - *self._class_overload_python_lines(overload, constructor=True), + *self._class_overload_python_lines( + overload, + constructor=True, + docstring=surface.constructor.docstring, + ), ) def _class_method_python_lines(self, method: ClassMethodPlan) -> tuple[str, ...]: @@ -10488,6 +10589,7 @@ def _class_method_python_lines(self, method: ClassMethodPlan) -> tuple[str, ...] lines.extend( ( f" def {method.python_name}({signature}):", + f" {method.docstring!r}", f" return {method.function.binding.python_name}({', '.join(call_names)})", ) ) @@ -10495,57 +10597,171 @@ def _class_method_python_lines(self, method: ClassMethodPlan) -> tuple[str, ...] def _class_overload_python_lines( self, - overload: ClassOverloadPlan, + overload: OverloadPlan, *, constructor: bool = False, + docstring: str | None = None, ) -> tuple[str, ...]: """Render deterministic exact-type selection without trial candidate calls.""" - self._require_class_overload_complete(overload) passed_object = True if constructor else overload.candidate_passed_objects[0] method_name = "__init__" if constructor else overload.python_name signature = "self, *args, **kwargs" if passed_object else "*args, **kwargs" - lines = [*((" @staticmethod",) if not passed_object else ()), f" def {method_name}({signature}):"] + return self._overload_python_lines( + overload, + method_name=method_name, + signature=signature, + indent=" ", + receiver_object="self" if passed_object or constructor else None, + static=not passed_object, + docstring=docstring or overload.docstring, + ) + + def _module_overload_python_source(self, overload: OverloadPlan) -> str: + """Render one namespace generic through the shared exact-match path.""" + return "\n".join( + self._overload_python_lines( + overload, + method_name=overload.python_name, + signature="*args, **kwargs", + indent="", + receiver_object=None, + static=False, + docstring=overload.docstring, + ) + ) + + def _overload_python_lines( + self, + overload: OverloadPlan, + *, + method_name: str, + signature: str, + indent: str, + receiver_object: str | None, + static: bool, + docstring: str, + ) -> tuple[str, ...]: + """Render one deterministic overload dispatcher at any namespace depth.""" + self._require_overload_complete(overload) + body_indent = f"{indent} " + lines = [ + *((f"{indent}@staticmethod",) if static else ()), + f"{indent}def {method_name}({signature}):", + f"{body_indent}{docstring!r}", + ] + if overload.unsupported_extra_argument_message is not None: + lines.extend( + ( + f"{body_indent}if len(args) > 1:", + f"{body_indent} raise TypeError({overload.unsupported_extra_argument_message!r})", + ) + ) + if overload.identity_receiver_shortcut and receiver_object is not None: + lines.extend( + ( + f"{body_indent}if len(args) == 1 and not kwargs and args[0] is {receiver_object}:", + f"{body_indent} return {receiver_object}", + ) + ) for candidate, matches, candidate_passed in zip( overload.candidates, overload.candidate_matches, overload.candidate_passed_objects, strict=True, ): - lines.extend(self._class_overload_candidate_python_lines(candidate, matches, candidate_passed, constructor)) - lines.append(f" raise TypeError('no matching overload for {overload.python_name}')") + candidate_receiver = ( + self._overload_receiver_name(candidate) if candidate_passed or receiver_object is not None else None + ) + lines.extend( + self._overload_candidate_python_lines( + candidate, + matches, + receiver_name=candidate_receiver, + receiver_object=receiver_object, + indent=body_indent, + ) + ) + lines.append(f"{body_indent}raise TypeError('no matching overload for {overload.python_name}')") return tuple(lines) @staticmethod - def _require_class_overload_complete(overload: ClassOverloadPlan) -> None: + def _require_overload_complete(overload: OverloadPlan) -> None: """Reject incomplete editable overload plans before Python source assembly.""" if not overload.candidates: - raise ValueError(f"Class overload {overload.owner_path!r} has no candidates") + raise ValueError(f"Overload {overload.owner_path!r} has no candidates") if not (len(overload.candidates) == len(overload.candidate_matches) == len(overload.candidate_passed_objects)): - raise ValueError(f"Class overload {overload.owner_path!r} has incomplete candidate metadata") + raise ValueError(f"Overload {overload.owner_path!r} has incomplete candidate metadata") - def _class_overload_candidate_python_lines( + def _overload_candidate_python_lines( self, candidate: FunctionPlan, matches: tuple, - candidate_passed: bool, - constructor: bool, + *, + receiver_name: str | None, + receiver_object: str | None, + indent: str, ) -> tuple[str, ...]: """Render one exact predicate and its single non-speculative call leaf.""" names = tuple(match.python_name for match in matches) condition = " and ".join(self._overload_dictionary_argument_predicate(item) for item in matches) or "True" - receiver = "self, " if candidate_passed or constructor else "" + receiver_line = ( + (f"{indent} _x2py_arguments[{receiver_name!r}] = {receiver_object}",) + if receiver_name is not None and receiver_object is not None + else () + ) + coercion_lines = tuple( + line + for match in matches + if match.accept_builtin_scalar + for line in self._overload_builtin_coercion_lines(match, f"{indent} ") + ) return ( - f" _x2py_names = {names!r}", - " if (", - " len(args) <= len(_x2py_names)", - " and all(_x2py_name in _x2py_names for _x2py_name in kwargs)", - " and not any(_x2py_name in kwargs for _x2py_name in _x2py_names[:len(args)])", - " ):", - " _x2py_arguments = dict(zip(_x2py_names, args))", - " _x2py_arguments.update(kwargs)", - f" if {condition}:", - f" return {candidate.binding.python_name}({receiver}**_x2py_arguments)", + f"{indent}_x2py_names = {names!r}", + f"{indent}if (", + f"{indent} len(args) <= len(_x2py_names)", + f"{indent} and all(_x2py_name in _x2py_names for _x2py_name in kwargs)", + f"{indent} and not any(_x2py_name in kwargs for _x2py_name in _x2py_names[:len(args)])", + f"{indent}):", + f"{indent} _x2py_arguments = dict(zip(_x2py_names, args))", + f"{indent} _x2py_arguments.update(kwargs)", + f"{indent} if {condition}:", + *coercion_lines, + *receiver_line, + f"{indent} return {candidate.binding.python_name}(**_x2py_arguments)", + ) + + def _overload_builtin_coercion_lines( + self, + match: OverloadArgumentMatchPlan, + indent: str, + ) -> tuple[str, ...]: + """Restore the NumPy scalar type lost before reflected dispatch.""" + name = match.python_name + assignment = ( + f"_x2py_arguments[{name!r}] = _x2py_numpy.{self._numpy_scalar_type_name(match.semantic_type_name)}(" + f"_x2py_arguments[{name!r}])" ) + if match.optional: + return (f"{indent}if {name!r} in _x2py_arguments:", f"{indent} {assignment}") + return (f"{indent}{assignment}",) + + @staticmethod + def _overload_receiver_name(candidate: FunctionPlan) -> str: + """Return the completed Python argument that receives the class instance.""" + call = candidate.class_call + if call is None or call.passed_object_position is None: + raise ValueError(f"Overload candidate {candidate.owner_path!r} has no completed receiver position") + receiver = next( + ( + argument + for argument in candidate.arguments + if argument.native_position == call.passed_object_position and argument.python_visible + ), + None, + ) + if receiver is None: + raise ValueError(f"Overload candidate {candidate.owner_path!r} has no visible receiver argument") + return receiver.binding.python_name @staticmethod def _callable_public_arguments(function: FunctionPlan) -> tuple[ArgumentTransferPlan, ...]: @@ -10593,7 +10809,7 @@ def _optional_keyword_collection_lines( def _overload_dictionary_argument_predicate( self, - argument: ClassOverloadArgumentMatchPlan, + argument: OverloadArgumentMatchPlan, ) -> str: """Match one normalized candidate argument without invoking its target.""" name = argument.python_name @@ -10607,25 +10823,30 @@ def _overload_dictionary_argument_predicate( def _required_overload_argument_predicate( self, - argument: ClassOverloadArgumentMatchPlan, + argument: OverloadArgumentMatchPlan, name: str, ) -> str: """Dispatch one typed match record into a small source leaf.""" - if argument.kind is ClassOverloadMatchKind.DERIVED: + if argument.kind is OverloadMatchKind.DERIVED: if argument.derived_type_identity is None: raise ValueError(f"Derived overload argument {name!r} has no type identity") return f"type({name}) is {self._class_python_names[argument.derived_type_identity]}" - if argument.kind is ClassOverloadMatchKind.NUMPY_ARRAY: + if argument.kind is OverloadMatchKind.NUMPY_ARRAY: return self._numpy_array_overload_predicate(argument, name) - if argument.kind is ClassOverloadMatchKind.STRING: + if argument.kind is OverloadMatchKind.STRING: return f"isinstance({name}, str)" - if argument.kind is ClassOverloadMatchKind.NUMPY_SCALAR: - return f"type({name}) is _x2py_numpy.{self._numpy_scalar_type_name(argument.semantic_type_name)}" + if argument.kind is OverloadMatchKind.NUMPY_SCALAR: + numpy_type = self._numpy_scalar_type_name(argument.semantic_type_name) + predicate = f"type({name}) is _x2py_numpy.{numpy_type}" + if argument.accept_builtin_scalar: + builtin = self._builtin_scalar_type_name(argument.semantic_type_name) + predicate = f"({predicate} or type({name}) is {builtin})" + return predicate raise ValueError(f"Unsupported class overload match kind: {argument.kind.value}") def _numpy_array_overload_predicate( self, - argument: ClassOverloadArgumentMatchPlan, + argument: OverloadArgumentMatchPlan, name: str, ) -> str: """Render one exact NumPy array rank and dtype predicate.""" @@ -10654,6 +10875,11 @@ def _numpy_scalar_type_name(semantic_type_name: str) -> str: except KeyError as exc: raise ValueError(f"Unsupported NumPy overload scalar {semantic_type_name!r}") from exc + @staticmethod + def _builtin_scalar_type_name(semantic_type_name: str) -> str: + """Return the Python scalar produced before reflected NumPy dispatch.""" + return overload_builtin_scalar_family(semantic_type_name) + @staticmethod def _class_base_name( surface: ClassSurfacePlan | None, @@ -10788,18 +11014,21 @@ def _c_string_literal(value: str) -> str: def _module_native_array_owner_nodes( self, namespace: NamespacePlan, - module_object: str, + _module_object: str, ) -> tuple[CExpressionStatement, ...]: - """Retain the owning Python module for every borrowed native handle.""" + """Retain the root extension package for every borrowed native array.""" nodes = [] for variable in namespace.variables: - if variable.binding.getter_action is not ModuleGetterAction.NATIVE_ARRAY_HANDLE: + if variable.binding.getter_action not in { + ModuleGetterAction.BORROWED_ARRAY_VIEW, + ModuleGetterAction.NATIVE_ARRAY_HANDLE, + }: continue owner = self._module_native_array_owner_name(variable) nodes.extend( ( - CExpressionStatement(CodeExpression(f"Py_INCREF({module_object})")), - CExpressionStatement(CodeExpression(f"{owner} = {module_object}")), + CExpressionStatement(CodeExpression("Py_INCREF(mod)")), + CExpressionStatement(CodeExpression(f"{owner} = mod")), ) ) return tuple(nodes) diff --git a/x2py/wrapper_codegen/checks.py b/x2py/wrapper_codegen/checks.py index 4b38a7095..d49b0430f 100644 --- a/x2py/wrapper_codegen/checks.py +++ b/x2py/wrapper_codegen/checks.py @@ -17,6 +17,8 @@ INFRASTRUCTURE_MODULES = frozenset({"__init__.py", "checks.py", "visitor.py"}) +SEMANTIC_PRINTER_MODULE = "pyi_printer.py" +SEMANTIC_PRINTER_FUNCTIONS = frozenset({"emit_module", "emit_module_stubs", "opaque_dependency_modules"}) VISITOR_CLASS_SUFFIXES = ("Analyzer", "Emitter", "Generator", "Planner", "Validator") REGISTRY_SUFFIXES = ("_DISPATCHER", "_HANDLERS", "_REGISTRY") HANDLER_PREFIXES = ("_convert_", "_emit_", "_handle_", "_visit_") @@ -76,13 +78,12 @@ def check_wrapper_codegen_package( ) -> tuple[WrapperCodegenViolation, ...]: """Check every Python module in the isolated wrapper-codegen package.""" root = package_root or Path(__file__).resolve().parent - return check_wrapper_codegen_paths(sorted(root.rglob("*.py")), package_root=root, config=config) + return check_wrapper_codegen_paths(sorted(root.rglob("*.py")), config=config) def check_wrapper_codegen_paths( paths: list[Path], *, - package_root: Path, config: WrapperCodegenCheckConfig | None = None, ) -> tuple[WrapperCodegenViolation, ...]: """Check selected wrapper-codegen modules.""" @@ -92,19 +93,19 @@ def check_wrapper_codegen_paths( for path in paths: source = path.read_text(encoding="utf-8") tree = ast.parse(source, filename=str(path)) - violations.extend(_module_violations(path, tree, package_root)) - violations.extend(_function_size_violations(path, tree, source, resolved_config, tiered_limits)) + violations.extend(_module_violations(path, tree)) + if path.name != SEMANTIC_PRINTER_MODULE: + violations.extend(_function_size_violations(path, tree, source, resolved_config, tiered_limits)) violations.extend(_registry_violations(path, tree)) return tuple(violations) -def _module_violations(path: Path, tree: ast.Module, package_root: Path) -> list[WrapperCodegenViolation]: +def _module_violations(path: Path, tree: ast.Module) -> list[WrapperCodegenViolation]: if path.name in INFRASTRUCTURE_MODULES: return [] return [ *_module_function_violations(path, tree), *_visitor_class_violations(path, tree), - *_dependency_violations(path, tree, package_root), ] @@ -113,6 +114,7 @@ def _module_function_violations(path: Path, tree: ast.Module) -> list[WrapperCod _violation(path, node, "module-function", f"move production function {node.name!r} onto a class") for node in tree.body if isinstance(node, ast.FunctionDef) + and not (path.name == SEMANTIC_PRINTER_MODULE and node.name in SEMANTIC_PRINTER_FUNCTIONS) ] @@ -127,16 +129,6 @@ def _visitor_class_violations(path: Path, tree: ast.Module) -> list[WrapperCodeg ] -def _dependency_violations(path: Path, tree: ast.Module, package_root: Path) -> list[WrapperCodegenViolation]: - if not _is_under(path, package_root): - return [] - return [ - _violation(path, node, "legacy-codegen-import", "wrapper_codegen must not import x2py.codegen") - for node in ast.walk(tree) - if _imports_legacy_codegen(node) - ] - - def _function_size_violations( path: Path, tree: ast.Module, @@ -279,19 +271,5 @@ def _base_name(node: ast.AST) -> str | None: return None -def _imports_legacy_codegen(node: ast.AST) -> bool: - if isinstance(node, ast.Import): - return any(alias.name == "x2py.codegen" or alias.name.startswith("x2py.codegen.") for alias in node.names) - return isinstance(node, ast.ImportFrom) and bool(node.module) and _is_codegen_module(node.module) - - -def _is_codegen_module(module_name: str) -> bool: - return module_name == "x2py.codegen" or module_name.startswith("x2py.codegen.") - - -def _is_under(path: Path, directory: Path) -> bool: - return path.resolve().is_relative_to(directory.resolve()) - - def _violation(path: Path, node: ast.AST, code: str, message: str) -> WrapperCodegenViolation: return WrapperCodegenViolation(path, getattr(node, "lineno", 1), code, message) diff --git a/x2py/wrapper_codegen/docstrings.py b/x2py/wrapper_codegen/docstrings.py new file mode 100644 index 000000000..d3cd8d937 --- /dev/null +++ b/x2py/wrapper_codegen/docstrings.py @@ -0,0 +1,633 @@ +"""Python-facing documentation rendered from completed wrapper-plan records.""" + +from __future__ import annotations + +from x2py.semantics.ownership import OwnershipOwner, SetterAction, TransferMode +from x2py.semantics.wrapper_policy import ( + ClassConstructorKind, + ModuleGetterAction, + NativeArrayDescriptorKind, + OptionalMode, +) +from x2py.wrapper_codegen.plan import ( + ArgumentTransferPlan, + ArrayHandoffPlan, + BindingStatusErrorPlan, + CallbackHandoffPlan, + CallbackTransferPlan, + ClassMethodPlan, + ConstructorPlan, + DatatypeFamily, + DerivedFieldPlan, + FunctionPlan, + ModuleVariablePlan, + OverloadPlan, + ResultPlan, +) + + +_SCALAR_TYPES = { + "Bool": "bool", + "Int8": "int8", + "Int16": "int16", + "Int32": "int32", + "Int64": "int64", + "Float32": "float32", + "Float64": "float64", + "Complex64": "complex64", + "Complex128": "complex128", + "String": "str", +} + +_UNKNOWN_EXTENTS = frozenset({"", ":", "::", "*", ".."}) + + +class WrapperDocstringBuilder: + """Build compact NumPy-style documentation without backend inference.""" + + # Namespace and class summaries. + def namespace( + self, + module_name: str, + path: tuple[str, ...], + functions: tuple[FunctionPlan, ...], + variables: tuple[ModuleVariablePlan, ...], + classes, + overloads: tuple[OverloadPlan, ...], + ) -> str: + """Index every public owner in one generated Python namespace.""" + qualified_name = ".".join((module_name, *path)) + lines = [qualified_name, "", f"Generated Python interface for native namespace {qualified_name}."] + callable_lines = ( + *(self._first_line(function.binding.docstring) for function in functions if function.binding.public), + *(self._first_line(overload.docstring) for overload in overloads), + ) + self._append_section(lines, "Functions", callable_lines) + self._append_section( + lines, + "Module Attributes", + tuple(line for variable in variables for line in self._module_variable_summary_lines(variable)), + ) + self._append_section(lines, "Classes", tuple(name for surface in classes for name in surface.python_names)) + return "\n".join(lines) + + def class_surface( + self, + python_name: str, + native_type_name: str, + constructor: ConstructorPlan, + fields: tuple[DerivedFieldPlan, ...], + methods: tuple[ClassMethodPlan, ...], + overloads: tuple[OverloadPlan, ...], + ) -> str: + """Summarize one opaque class and its complete public descriptors.""" + lines = [python_name, "", f"Opaque wrapper for native type {native_type_name}."] + self._append_section(lines, "Constructor", (self._first_line(constructor.docstring),)) + self._append_section(lines, "Fields", tuple(self._first_line(field.docstring) for field in fields)) + self._append_section( + lines, + "Methods", + ( + *(self._first_line(method.docstring) for method in methods if method.public), + *(self._first_line(overload.docstring) for overload in overloads), + ), + ) + return "\n".join(lines) + + # Callable documentation. + def function( + self, + python_name: str, + arguments: tuple[ArgumentTransferPlan, ...], + results: tuple[ResultPlan, ...], + *, + status_error: BindingStatusErrorPlan | None = None, + excluded_native_position: int | None = None, + ) -> str: + """Describe one Python callable from its completed transfers.""" + visible = self._visible_arguments(arguments, excluded_native_position) + outputs = self._documented_outputs(arguments, results) + lines = [self._callable_signature(python_name, visible, outputs)] + self._append_section( + lines, + "Parameters", + tuple(line for argument in visible for line in self._argument_lines(argument)), + ) + self._append_section( + lines, + "Returns", + tuple(line for output in outputs for line in self._output_lines(output, arguments)) or ("None",), + ) + self._append_section(lines, "Raises", self._raise_lines(visible, outputs, status_error)) + return "\n".join(lines) + + def method(self, method: ClassMethodPlan) -> str: + """Describe one public method while omitting its passed-object slot.""" + docstring = self.function( + method.python_name, + method.function.arguments, + method.function.results, + status_error=method.function.binding.status_error, + excluded_native_position=method.passed_object_position, + ) + if method.passed_object_position is None: + return docstring + receiver = self._argument_at_native_position(method.function.arguments, method.passed_object_position) + if receiver.mutates_native: + docstring += "\n\nNotes\n-----\nUpdates the wrapped native instance in place." + return docstring + + def overload(self, overload: OverloadPlan) -> str: + """List accepted public signatures without exposing private candidates.""" + signatures = tuple( + self._candidate_signature(overload.python_name, candidate, passed) + for candidate, passed in zip( + overload.candidates, + overload.candidate_passed_objects, + strict=True, + ) + ) + lines = [f"{overload.python_name}(*args, **kwargs)"] + self._append_section(lines, "Supported Signatures", signatures) + self._append_section( + lines, + "Raises", + ("TypeError", " If no supported signature matches the supplied arguments."), + ) + self._append_section(lines, "Notes", self._overload_notes(overload)) + return "\n".join(lines) + + def constructor( + self, + python_name: str, + constructor: ConstructorPlan, + fields: tuple[DerivedFieldPlan, ...], + ) -> str: + """Describe the selected construction route for one generated class.""" + if constructor.kind is ClassConstructorKind.ABSENT: + return self._absent_constructor(python_name, constructor) + handlers = { + ClassConstructorKind.DEFAULT_FIELDS: self._default_constructor, + ClassConstructorKind.BOUND_PROCEDURE: self._bound_constructor, + ClassConstructorKind.OVERLOAD_SET: self._overloaded_constructor, + } + handler = handlers.get(constructor.kind) + if handler is None: # pragma: no cover - policy validation owns the enum envelope + raise ValueError(f"Unsupported constructor kind: {constructor.kind.value}") + lines = handler(python_name, constructor, fields) + self._append_section(lines, "Returns", (python_name, " New wrapper-owned native instance.")) + self._append_section( + lines, + "Raises", + ("TypeError", " If the supplied arguments do not satisfy the constructor contract."), + ) + return "\n".join(lines) + + @staticmethod + def _absent_constructor(python_name: str, constructor: ConstructorPlan) -> str: + """Describe an explicitly nonconstructible wrapper class.""" + return "\n".join( + ( + f"{python_name}(*args, **kwargs)", + "", + "Raises", + "------", + "TypeError", + f" {constructor.rejection_message or 'Direct construction is disabled.'}", + ) + ) + + def _default_constructor( + self, + python_name: str, + constructor: ConstructorPlan, + fields: tuple[DerivedFieldPlan, ...], + ) -> list[str]: + """Document the keyword-only editable-field constructor.""" + by_name = {field.name: field for field in fields} + parameters = tuple(by_name[item.name] for item in constructor.fields if item.name in by_name) + lines = [self._keyword_field_signature(python_name, constructor, parameters)] + self._append_section( + lines, + "Parameters", + tuple(line for field in parameters for line in self._constructor_field_lines(field)), + ) + return lines + + def _bound_constructor( + self, + python_name: str, + constructor: ConstructorPlan, + _fields: tuple[DerivedFieldPlan, ...], + ) -> list[str]: + """Document a constructor backed by one completed native call.""" + target = constructor.target + if target is None: + raise ValueError(f"Bound constructor {python_name!r} has no target plan") + passed = target.class_call.passed_object_position if target.class_call else None + arguments = self._visible_arguments(target.arguments, passed) + lines = [self._signature(python_name, arguments, python_name)] + self._append_section( + lines, + "Parameters", + tuple(line for argument in arguments for line in self._argument_lines(argument)), + ) + return lines + + def _overloaded_constructor( + self, + python_name: str, + constructor: ConstructorPlan, + _fields: tuple[DerivedFieldPlan, ...], + ) -> list[str]: + """Document exact signatures accepted by an overloaded constructor.""" + overload = constructor.overload + if overload is None: + raise ValueError(f"Overloaded constructor {python_name!r} has no overload plan") + signatures = tuple( + self._candidate_signature(python_name, candidate, passed, result_type=python_name) + for candidate, passed in zip( + overload.candidates, + overload.candidate_passed_objects, + strict=True, + ) + ) + lines = [f"{python_name}(*args, **kwargs) -> {python_name}"] + self._append_section(lines, "Supported Signatures", signatures) + return lines + + # Attribute documentation. + def module_variable(self, variable: ModuleVariablePlan) -> str: + """Describe a module attribute where CPython cannot attach a descriptor docstring.""" + name = variable.binding.python_names[0] + nullable = variable.binding.getter_action is ModuleGetterAction.NULLABLE_SNAPSHOT + lines = [f"{name} : {self._type(variable, nullable=nullable, signature=False)}"] + lines.extend(self._array_lines(variable.array)) + if variable.binding.getter_action is ModuleGetterAction.CONSTANT_VALUE: + lines.append(" Read-only native constant.") + elif variable.binding.getter_action is ModuleGetterAction.BORROWED_ARRAY_VIEW: + lines.append(" Native-owned borrowed view; mutations affect module storage.") + elif variable.native_array_handle is not None: + lines.append(f" Persistent {variable.native_array_handle.descriptor_kind.value} descriptor handle.") + elif variable.derived is not None: + lines.append(" Live native module object.") + if variable.binding.setter_action is SetterAction.WRITE_THROUGH: + lines.append(" Assignment writes through to native storage.") + elif variable.binding.setter_action is SetterAction.REJECT_REPLACEMENT: + lines.append(" Replacement assignment is not supported.") + return "\n".join(lines) + + def field(self, field: DerivedFieldPlan) -> str: + """Describe one property and its native lifetime or assignment behavior.""" + lines = [f"{field.name} : {self._type(field, nullable=False, signature=False)}"] + lines.extend(self._array_lines(field.array)) + if field.native_array_handle is not None: + lines.append(f" Live {field.native_array_handle.descriptor_kind.value} array descriptor handle.") + lines.append(" The parent wrapper retains the descriptor owner.") + elif field.array is not None: + lines.append(" Borrowed native view retained by the parent wrapper.") + if field.setter_action is SetterAction.WRITE_THROUGH: + lines.append(" Assignment writes through to native storage.") + elif field.setter_action is SetterAction.REJECT_REPLACEMENT: + lines.append(" Replacement assignment is not supported.") + else: + lines.append(" Read-only attribute.") + return "\n".join(lines) + + # Shared signature and section helpers. + @staticmethod + def _append_section(lines: list[str], heading: str, body: tuple[str, ...]) -> None: + """Append one nonempty NumPy-style section.""" + if not body: + return + lines.extend(("", heading, "-" * len(heading), *body)) + + @staticmethod + def _first_line(docstring: str) -> str: + """Return the stable summary line of a rendered docstring.""" + return docstring.splitlines()[0] if docstring else "" + + def _callable_signature( + self, + name: str, + arguments: tuple[ArgumentTransferPlan, ...], + outputs: tuple[ArgumentTransferPlan | ResultPlan, ...], + ) -> str: + return self._signature(name, arguments, self._result_summary(outputs)) + + def _signature( + self, + name: str, + arguments: tuple[ArgumentTransferPlan, ...], + result_type: str, + ) -> str: + parameters = ", ".join(self._signature_parameter(argument) for argument in arguments) + return f"{name}({parameters}) -> {result_type}" + + @staticmethod + def _signature_parameter(argument: ArgumentTransferPlan) -> str: + name = argument.binding.python_name + if argument.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR}: + return f"{name}=..." + return name + + def _candidate_signature( + self, + name: str, + candidate: FunctionPlan, + passed_object: bool, + *, + result_type: str | None = None, + ) -> str: + passed = candidate.class_call.passed_object_position if passed_object and candidate.class_call else None + arguments = self._visible_arguments(candidate.arguments, passed) + outputs = self._documented_outputs(candidate.arguments, candidate.results) + parameters = ", ".join(self._typed_signature_parameter(argument) for argument in arguments) + return f"{name}({parameters}) -> {result_type or self._result_summary(outputs)}" + + @staticmethod + def _overload_candidates(overload: OverloadPlan): + """Pair candidates with their completed passed-object flags.""" + return zip(overload.candidates, overload.candidate_passed_objects, strict=True) + + @staticmethod + def _argument_at_native_position( + arguments: tuple[ArgumentTransferPlan, ...], + native_position: int, + ) -> ArgumentTransferPlan: + """Return the validated receiver selected by class-call policy.""" + return next(argument for argument in arguments if argument.native_position == native_position) + + def _candidate_mutates_receiver(self, candidate: FunctionPlan, passed_object: bool) -> bool: + """Report receiver mutation for one completed overload candidate.""" + if not passed_object or candidate.class_call is None: + return False + receiver = self._argument_at_native_position(candidate.arguments, candidate.class_call.passed_object_position) + return receiver.mutates_native + + def _overload_notes(self, overload: OverloadPlan) -> tuple[str, ...]: + """Describe only receiver behavior shared by class-owned candidates.""" + if not any(overload.candidate_passed_objects): + return () + notes = ["Dispatches to a native operation on the wrapped instance."] + if any( + self._candidate_mutates_receiver(candidate, passed) + for candidate, passed in self._overload_candidates(overload) + ): + notes.append("Updates the wrapped native instance in place.") + return tuple(notes) + + def _typed_signature_parameter(self, argument: ArgumentTransferPlan) -> str: + """Render enough public type information to distinguish overloads.""" + parameter = f"{argument.binding.python_name}: {self._type(argument, nullable=argument.binding.nullable, signature=True)}" + if argument.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR}: + return f"{parameter} = ..." + return parameter + + @staticmethod + def _visible_arguments( + arguments: tuple[ArgumentTransferPlan, ...], + excluded_native_position: int | None, + ) -> tuple[ArgumentTransferPlan, ...]: + return tuple( + argument + for argument in sorted(arguments, key=lambda item: item.python_position) + if argument.python_visible and argument.native_position != excluded_native_position + ) + + @staticmethod + def _documented_outputs( + arguments: tuple[ArgumentTransferPlan, ...], + results: tuple[ResultPlan, ...], + ) -> tuple[ArgumentTransferPlan | ResultPlan, ...]: + by_position = { + argument.result_position: argument + for argument in arguments + if argument.projects_result and argument.result_position is not None + } + by_position.update((result.result_position, result) for result in results) + return tuple(by_position[position] for position in sorted(by_position)) + + def _result_summary(self, outputs: tuple[ArgumentTransferPlan | ResultPlan, ...]) -> str: + types = tuple( + self._type( + output, + nullable=( + output.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR} + if isinstance(output, ArgumentTransferPlan) + else output.nullable + ), + signature=True, + ) + for output in outputs + ) + if not types: + return "None" + if len(types) == 1: + return types[0] + return f"tuple[{', '.join(types)}]" + + # Parameter and result details. + def _argument_lines(self, argument: ArgumentTransferPlan) -> tuple[str, ...]: + optional = argument.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR} + nullable = optional or argument.binding.nullable + lines = [f"{argument.binding.python_name} : {self._type(argument, nullable=nullable, signature=False)}"] + lines.extend(self._array_lines(argument.array)) + lines.extend(self._optional_lines(argument)) + lines.extend(self._mutation_lines(argument)) + if argument.native_array_handle is not None: + lines.append(f" Descriptor ownership: {argument.native_array_handle.descriptor_ownership.value}.") + return tuple(lines) + + def _output_lines( + self, + output: ArgumentTransferPlan | ResultPlan, + arguments: tuple[ArgumentTransferPlan, ...], + ) -> tuple[str, ...]: + if isinstance(output, ArgumentTransferPlan): + name = output.binding.python_name + nullable = output.binding.optional_mode not in {OptionalMode.REQUIRED, OptionalMode.REQUIRED_DESCRIPTOR} + else: + name = self._result_name(output, arguments) + nullable = output.nullable + lines = [f"{name} : {self._type(output, nullable=nullable, signature=False)}"] + lines.extend(self._array_lines(output.array)) + if output.native_array_handle is not None: + handle = output.native_array_handle + lines.append(f" Descriptor ownership: {handle.descriptor_ownership.value}.") + state = "Unallocated" if handle.descriptor_kind is NativeArrayDescriptorKind.ALLOCATABLE else "Unassociated" + lines.append(f" {state} state remains inside the returned handle.") + if isinstance(output, ArgumentTransferPlan): + lines.extend(self._ownership_lines(output.ownership_owner)) + if output.transfer_mode is TransferMode.COPY_RETURN: + lines.append(" Detached replacement; the original Python value is unchanged.") + elif output.datatype_family is DatatypeFamily.DERIVED or output.array is not None: + lines.extend(self._ownership_lines(output.ownership_owner)) + if nullable and output.native_array_handle is None: + lines.append(" May be None.") + return tuple(lines) + + @staticmethod + def _optional_lines(argument: ArgumentTransferPlan) -> tuple[str, ...]: + mode = argument.binding.optional_mode + if mode is OptionalMode.DESCRIPTOR: + return ( + " Omit to make the native optional dummy absent.", + " Pass None for a present unallocated or unassociated descriptor.", + ) + if mode is OptionalMode.REQUIRED_DESCRIPTOR: + return (" Pass None for an unallocated or unassociated required descriptor.",) + if mode is not OptionalMode.REQUIRED: + return (" May be omitted or passed as None.",) + if argument.binding.nullable: + return (" May be passed as None.",) + return () + + @staticmethod + def _mutation_lines(argument: ArgumentTransferPlan) -> tuple[str, ...]: + if not argument.mutates_native: + return () + if argument.transfer_mode is TransferMode.COPY_RETURN: + return ( + " Native code writes to a private copy.", + " The original Python value is unchanged; the replacement is returned.", + ) + if argument.projects_result: + return (" Native code may update this value; the updated value is returned.",) + return (" Native code may update the supplied storage in place.",) + + def _raise_lines( + self, + arguments: tuple[ArgumentTransferPlan, ...], + outputs: tuple[ArgumentTransferPlan | ResultPlan, ...], + status_error: BindingStatusErrorPlan | None, + ) -> tuple[str, ...]: + exceptions = [("TypeError", "If an argument has an incompatible Python type or dtype.")] + if any(item.array is not None or item.native_array_handle is not None for item in (*arguments, *outputs)): + exceptions.append(("ValueError", "If rank, shape, layout, or descriptor state violates the contract.")) + if any(item.datatype_family is DatatypeFamily.DERIVED for item in arguments): + exceptions.append(("RuntimeError", "If a derived-object transaction cannot be acquired or restored.")) + if status_error is not None: + exceptions.append( + ( + status_error.exception_kind.value, + f"If native status differs from the success value {status_error.success}.", + ) + ) + return self._merged_exception_lines(exceptions) + + @staticmethod + def _merged_exception_lines(exceptions: list[tuple[str, str]]) -> tuple[str, ...]: + """Group descriptions under one heading per public exception type.""" + grouped: dict[str, list[str]] = {} + for exception, description in exceptions: + grouped.setdefault(exception, []).append(description) + return tuple( + line + for exception, descriptions in grouped.items() + for line in (exception, *(f" {item}" for item in descriptions)) + ) + + # Type, array, ownership, and constructor helpers. + def _type(self, transfer, *, nullable: bool, signature: bool) -> str: + type_name = self._base_type(transfer) + if not nullable: + return type_name + return f"{type_name} | None" if signature else f"{type_name} or None" + + def _base_type(self, transfer) -> str: + if getattr(transfer, "datatype_family", None) is DatatypeFamily.CALLBACK: + return self._callback_type(transfer.callback) + if getattr(transfer, "datatype_family", None) is DatatypeFamily.DERIVED: + return transfer.semantic_type_name + scalar = _SCALAR_TYPES.get(transfer.semantic_type_name, transfer.semantic_type_name) + handle = getattr(transfer, "native_array_handle", None) + if handle is not None: + prefix = ( + "AllocatableArray" + if handle.descriptor_kind is NativeArrayDescriptorKind.ALLOCATABLE + else "PointerArray" + ) + return f"{prefix}[{scalar}]" + if getattr(transfer, "array", None) is not None: + element = "bytes" if transfer.semantic_type_name == "String" else scalar + return f"ndarray[{element}]" + return scalar + + def _callback_type(self, callback: CallbackHandoffPlan | None) -> str: + if callback is None: + raise ValueError("Callback documentation requires a completed handoff plan") + arguments = ", ".join(self._callback_transfer_type(item) for item in callback.arguments) + result = "None" if callback.result.transfer is None else self._callback_transfer_type(callback.result.transfer) + return f"Callable[[{arguments}], {result}]" + + @staticmethod + def _callback_transfer_type(transfer: CallbackTransferPlan) -> str: + if transfer.derived_type_identity is not None: + return transfer.semantic_type_name + scalar = _SCALAR_TYPES.get(transfer.semantic_type_name, transfer.semantic_type_name) + if transfer.array is not None or (transfer.abi.value == "reference" and transfer.access != "read"): + return f"ndarray[{scalar}]" + return scalar + + @staticmethod + def _array_lines(array: ArrayHandoffPlan | None) -> tuple[str, ...]: + if array is None: + return () + lines = [" Rank: 1..15" if array.rank is None else f" Rank: {array.rank}"] + if array.shape and all(str(extent) not in _UNKNOWN_EXTENTS for extent in array.shape): + lines.append(f" Shape: ({', '.join(map(str, array.shape))})") + if (array.rank is None or array.rank > 1) and array.order in {"ORDER_C", "ORDER_F"}: + layout = "C-contiguous" if array.order == "ORDER_C" else "F-contiguous" + lines.append(f" Layout: {layout}") + return tuple(lines) + + @staticmethod + def _ownership_lines(owner: OwnershipOwner) -> tuple[str, ...]: + label = { + OwnershipOwner.CALLER: "Caller-owned", + OwnershipOwner.NATIVE: "Native-owned", + OwnershipOwner.PYTHON: "Python-owned", + OwnershipOwner.WRAPPER: "Wrapper-owned", + OwnershipOwner.TEMPORARY: "Temporary", + OwnershipOwner.UNKNOWN: "Unknown", + }[owner] + return (f" Ownership: {label}.",) + + @staticmethod + def _result_name(result: ResultPlan, arguments: tuple[ArgumentTransferPlan, ...]) -> str: + projected = next( + ( + argument.binding.python_name + for argument in arguments + if argument.projects_result and argument.result_position == result.result_position + ), + None, + ) + if projected is not None: + return projected + if result.native_call_slot is not None and result.native_call_slot.python_name: + return result.native_call_slot.python_name + return "result" if result.result_position == 0 else f"result_{result.result_position}" + + def _module_variable_summary_lines(self, variable: ModuleVariablePlan) -> tuple[str, ...]: + first, *details = variable.docstring.splitlines() + _name, separator, type_name = first.partition(" : ") + if not separator: + return (first,) + return tuple(line for name in variable.binding.python_names for line in (f"{name} : {type_name}", *details)) + + def _keyword_field_signature( + self, + python_name: str, + constructor: ConstructorPlan, + fields: tuple[DerivedFieldPlan, ...], + ) -> str: + defaults = {field.name: field.default_value for field in constructor.fields} + parameters = ", ".join( + f"{field.name}={defaults[field.name] if defaults[field.name] is not None else '...'}" for field in fields + ) + return f"{python_name}(*, {parameters}) -> {python_name}" if parameters else f"{python_name}() -> {python_name}" + + def _constructor_field_lines(self, field: DerivedFieldPlan) -> tuple[str, ...]: + return (f"{field.name} : {self._type(field, nullable=False, signature=False)}",) diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index 44de9ad77..4f27b48c6 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -34,6 +34,7 @@ NativeArrayDescriptorInterop, NativeArrayOperation, NativeDescriptorHandoffABI, + NativeInvocationKind, OptionalMode, TransformationLayer, ) @@ -478,17 +479,39 @@ def _require_string_plan_result_supported(self, result: ResultPlan) -> None: def _require_variable_supported(self, variable: ModuleVariablePlan) -> None: """Reject unsupported actions in one planned module variable.""" - if variable.binding.getter_action is ModuleGetterAction.NATIVE_ARRAY_HANDLE: - handle = variable.native_array_handle - if handle is None or handle.array.rank is None: - raise ValueError(f"Unsupported Fortran module handle for {variable.owner_path!r}") - if variable.datatype_family is not DatatypeFamily.STRING: - PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) - return - if variable.binding.getter_action is ModuleGetterAction.DERIVED_OBJECT: - if variable.derived is None: - raise ValueError(f"Unsupported Fortran derived module object for {variable.owner_path!r}") + validator = { + ModuleGetterAction.NATIVE_ARRAY_HANDLE: self._require_module_handle_supported, + ModuleGetterAction.BORROWED_ARRAY_VIEW: self._require_module_array_view_supported, + ModuleGetterAction.DERIVED_OBJECT: self._require_module_derived_supported, + }.get(variable.binding.getter_action) + if validator is not None: + validator(variable) return + self._require_module_scalar_supported(variable) + + def _require_module_handle_supported(self, variable: ModuleVariablePlan) -> None: + """Require one completed allocatable or pointer array handle.""" + handle = variable.native_array_handle + if handle is None or handle.array.rank is None: + raise ValueError(f"Unsupported Fortran module handle for {variable.owner_path!r}") + if variable.datatype_family is not DatatypeFamily.STRING: + PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) + + def _require_module_array_view_supported(self, variable: ModuleVariablePlan) -> None: + """Require one ranked borrowed module-array view.""" + if variable.array is None or variable.array.rank is None: + raise ValueError(f"Unsupported Fortran module array view for {variable.owner_path!r}") + PrimitiveScalarTypeRegistry.type_for(variable.semantic_type_name) + + @staticmethod + def _require_module_derived_supported(variable: ModuleVariablePlan) -> None: + """Require the completed derived-object module handoff.""" + if variable.derived is None: + raise ValueError(f"Unsupported Fortran derived module object for {variable.owner_path!r}") + + @staticmethod + def _require_module_scalar_supported(variable: ModuleVariablePlan) -> None: + """Require ordinary scalar assignment and nullable snapshot actions.""" if variable.bridge.native_assignment not in { AssignmentMode.NONE, AssignmentMode.VALUE_COPY, @@ -762,8 +785,10 @@ def _callback_adapter_procedure( def _callback_native_parameter(self, transfer: CallbackTransferPlan) -> FortranParameter: """Declare the exact native callback dummy represented by one transfer.""" attributes = list(self._callback_intent_attributes(transfer)) - if transfer.abi is CallbackABIKind.VALUE and transfer.access == "unspecified": - attributes.extend(("intent(in)", "value")) + if transfer.abi is CallbackABIKind.VALUE: + if transfer.access == "unspecified": + attributes.append("intent(in)") + attributes.append("value") if transfer.abi is not CallbackABIKind.VALUE and transfer.adapter_action in { CallbackTransferAction.BORROW_READ_ONLY, CallbackTransferAction.BORROW_WRITABLE, @@ -1741,6 +1766,8 @@ def _lower_module_getter(self, plan: ModuleVariablePlan) -> tuple[FortranFunctio return self._lower_module_getter_direct_value(plan) case ModuleGetterAction.NULLABLE_SNAPSHOT: return self._lower_module_getter_nullable_snapshot(plan) + case ModuleGetterAction.BORROWED_ARRAY_VIEW: + return self._lower_module_getter_borrowed_array_view(plan) case ModuleGetterAction.DERIVED_OBJECT: return self._lower_module_getter_derived_object(plan) raise ValueError(f"Unsupported Fortran module getter action for {plan.owner_path!r}: {action!r}") @@ -2418,6 +2445,39 @@ def _lower_module_getter_direct_value(self, plan: ModuleVariablePlan) -> tuple[F ), ) + def _lower_module_getter_borrowed_array_view( + self, + plan: ModuleVariablePlan, + ) -> tuple[FortranFunction, ...]: + """Expose one addressable fixed module array through pointer and extents.""" + array = plan.array + if array is None or array.rank is None: + raise ValueError(f"Module array view {plan.owner_path!r} has no fixed rank") + name = self._module_bridge_getter_name(plan) + native = self._native_variable_name(plan) + return ( + FortranFunction( + name=name, + parameters=tuple( + FortranParameter(f"extent_{axis}", "integer(c_int64_t)", ("intent(out)",)) + for axis in range(array.rank) + ), + result_name="result", + result_type="type(c_ptr)", + bind_name=name, + body=( + *( + FortranAssignment( + f"extent_{axis}", + CodeExpression(f"int(size({native}, {axis + 1}), c_int64_t)"), + ) + for axis in range(array.rank) + ), + FortranAssignment("result", CodeExpression(f"c_loc({native})")), + ), + ), + ) + def _lower_module_getter_nullable_snapshot( self, plan: ModuleVariablePlan, @@ -3062,7 +3122,11 @@ def _native_invocation( present: frozenset[str], result_name: str | None, replacements: dict[str, str], - ) -> FortranAssignment | FortranCall: + ) -> FortranAssignment | FortranCall | FortranPointerAssignment: + if plan.bridge.native_invocation is NativeInvocationKind.DEFINED_OPERATOR: + return self._defined_operator_invocation(plan, present, result_name, replacements) + if plan.bridge.native_invocation is NativeInvocationKind.DEFINED_ASSIGNMENT: + return self._defined_assignment_invocation(plan, present, replacements) native_name, receiver_position = self._native_invocation_target(plan, replacements) arguments = self._native_arguments( plan, @@ -3074,6 +3138,34 @@ def _native_invocation( return FortranCall(native_name, arguments) return self._native_function_result_invocation(plan, result_name, native_name, arguments) + def _defined_operator_invocation( + self, + plan: FunctionPlan, + present: frozenset[str], + result_name: str | None, + replacements: dict[str, str], + ) -> FortranAssignment | FortranCall | FortranPointerAssignment: + """Lower one completed public defined operator without private specifics.""" + token = plan.bridge.native_operator + arguments = self._native_arguments(plan, present, replacements) + if token is None or len(arguments) not in {1, 2}: + raise ValueError(f"Defined operator {plan.owner_path!r} has an incomplete invocation plan") + values = tuple(argument.text for argument in arguments) + expression = f"{token} {values[0]}" if len(values) == 1 else f"{values[0]} {token} {values[1]}" + return self._native_result_expression_invocation(plan, result_name, expression) + + def _defined_assignment_invocation( + self, + plan: FunctionPlan, + present: frozenset[str], + replacements: dict[str, str], + ) -> FortranAssignment: + """Lower one completed defined assignment in native argument order.""" + arguments = self._native_arguments(plan, present, replacements) + if len(arguments) != 2: + raise ValueError(f"Defined assignment {plan.owner_path!r} must have two native arguments") + return FortranAssignment(arguments[0].text, arguments[1]) + def _native_function_result_invocation( self, plan: FunctionPlan, @@ -3085,6 +3177,15 @@ def _native_function_result_invocation( if result_name is None: raise ValueError(f"{plan.owner_path!r} native function is missing a bridge result") expression = f"{native_name}({', '.join(item.text for item in arguments)})" + return self._native_result_expression_invocation(plan, result_name, expression) + + def _native_result_expression_invocation( + self, + plan: FunctionPlan, + result_name: str | None, + expression: str, + ) -> FortranAssignment | FortranCall | FortranPointerAssignment: + """Store one completed native result expression through its handoff leaf.""" direct_result = self._direct_result(plan) collector = self._native_result_collector_name(plan, direct_result) if collector is not None: @@ -4907,6 +5008,12 @@ def _add_derived_module_uses(self, plan: ModulePlan, modules: dict[str, list[str def _add_function_module_uses(self, plan: ModulePlan, modules: dict[str, list[str]]) -> None: """Import module procedures, excluding direct type-bound invocation.""" for function in self._functions(plan): + if ( + function.bridge.native_module is not None + and function.bridge.native_invocation is not NativeInvocationKind.PROCEDURE + ): + modules.setdefault(function.bridge.native_module, []).append(function.bridge.native_name) + continue if function.bridge.native_module is not None and ( function.class_call is None or function.class_call.invocation is ClassInvocationKind.MODULE_PROCEDURE ): @@ -6561,10 +6668,8 @@ def _native_result_slots_need_allocator(self, function: FunctionPlan) -> bool: ) def _external_interface_procedure(self, plan: FunctionPlan) -> FortranInterfaceProcedure: - parameters = tuple( - self._external_interface_parameter(plan, argument) - for argument in sorted(plan.arguments, key=lambda item: item.native_position) - ) + arguments = tuple(sorted(plan.arguments, key=lambda item: item.native_position)) + parameters = tuple(self._external_interface_parameter(plan, argument) for argument in arguments) imports = tuple(dict.fromkeys(self._iso_symbol(argument.semantic_type_name) for argument in plan.arguments)) result_name = None if plan.bridge.native_is_subroutine else "native_result" direct_result = self._direct_result(plan) @@ -6575,11 +6680,40 @@ def _external_interface_procedure(self, plan: FunctionPlan) -> FortranInterfaceP name=plan.bridge.native_name, imports=imports, parameters=parameters, + parameter_declarations=self._external_interface_parameter_declarations(arguments, parameters), result_name=result_name, result_type=result_type, is_subroutine=plan.bridge.native_is_subroutine, ) + @staticmethod + def _external_interface_parameter_declarations( + arguments: tuple[ArgumentTransferPlan, ...], + parameters: tuple[FortranParameter, ...], + ) -> tuple[FortranParameter, ...]: + """Declare extent providers first without changing native ABI order.""" + pending = list(zip(arguments, parameters, strict=True)) + declarations = [] + emitted_roles = set() + while pending: + for index, (argument, parameter) in enumerate(pending): + dependencies = ( + {role for axis_roles in argument.array.extent_reference_roles for role in axis_roles} + if argument.array is not None + else set() + ) + if dependencies <= emitted_roles: + declarations.append(parameter) + emitted_roles.add(argument.binding.handoff_role) + pending.pop(index) + break + else: + # Central plan validation owns missing or cyclic extent roles. + # Preserve native order here so emission remains deterministic. + declarations.extend(parameter for _, parameter in pending) + break + return tuple(declarations) + def _native_result_type(self, plan: FunctionPlan, result: ResultPlan | None) -> str: """Return the native procedure result type inside an external interface.""" if result is None: diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index 7d8af2358..669f5f637 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -51,6 +51,7 @@ ORDINARY_ARRAY_RESULT_COPY_REASON, ModuleGetterAction, ModuleObjectAccessMechanism, + NativeInvocationKind, NativeArrayDescriptorInterop, NativeArrayDescriptorKind, NativeArrayDescriptorOwnership, @@ -74,6 +75,7 @@ TransformationAction, TransformationLayer, WritebackPhase, + overload_builtin_scalar_family, ) from x2py.wrapper_codegen.c.binding import CBindingGenerator from x2py.wrapper_codegen.fortran.bridge import FortranBridgeGenerator @@ -81,7 +83,7 @@ ArgumentTransferPlan, CallbackHandoffPlan, CallbackTransferPlan, - ClassOverloadPlan, + OverloadPlan, ClassSurfacePlan, DatatypeFamily, FunctionPlan, @@ -94,7 +96,7 @@ ResultPlan, WrapperPlanDiagnostic, ) -from x2py.wrapper_codegen.source_printers import CSourcePrinter, FortranSourcePrinter +from x2py.wrapper_codegen.printers import CSourcePrinter, FortranSourcePrinter class WrapperCodeGenerator: @@ -152,6 +154,9 @@ def _plan_diagnostics(self, plan: ModulePlan) -> tuple[WrapperPlanDiagnostic, .. diagnostics.extend(self._module_variable_diagnostics(variable)) for class_surface in namespace.classes: diagnostics.extend(self._class_surface_diagnostics(namespace, class_surface)) + functions = {id(function) for function in namespace.functions} + for overload in namespace.overloads: + diagnostics.extend(self._overload_diagnostics(overload, functions)) diagnostics.extend(self._class_graph_diagnostics(plan)) diagnostics.extend(self._generated_symbol_diagnostics(plan)) diagnostics.extend(self._required_header_diagnostics(plan)) @@ -259,7 +264,7 @@ def _class_surface_diagnostics( *( diagnostic for overload in surface.overloads - for diagnostic in self._class_overload_diagnostics(overload, functions) + for diagnostic in self._overload_diagnostics(overload, functions) ), *self._constructor_diagnostics(surface), *self._constructor_reference_diagnostics(surface, functions), @@ -305,50 +310,63 @@ def _constructor_reference_diagnostics( self._diagnostic(surface.owner_path, "missing-constructor-target", constructor.target.owner_path) ) if constructor.overload is not None: - diagnostics.extend(self._class_overload_diagnostics(constructor.overload, functions)) + diagnostics.extend(self._overload_diagnostics(constructor.overload, functions)) return tuple(diagnostics) - def _class_overload_diagnostics( + def _overload_diagnostics( self, - overload: ClassOverloadPlan, + overload: OverloadPlan, functions: set[int], ) -> tuple[WrapperPlanDiagnostic, ...]: """Validate candidate references and exact runtime signatures once.""" - diagnostics = [] - if not overload.candidates: - diagnostics.append(self._diagnostic(overload.owner_path, "empty-class-overload", overload.python_name)) - if len(overload.candidates) != len(overload.candidate_matches): - diagnostics.append( - self._diagnostic( - overload.owner_path, - "incomplete-class-overload-match-plan", - (len(overload.candidates), len(overload.candidate_matches)), - ) - ) - if len(overload.candidates) != len(overload.candidate_passed_objects): - diagnostics.append( - self._diagnostic( - overload.owner_path, - "incomplete-class-overload-call-plan", - (len(overload.candidates), len(overload.candidate_passed_objects)), - ) - ) - diagnostics.extend( - self._diagnostic(overload.owner_path, "missing-class-overload-candidate", candidate.owner_path) + missing = tuple( + self._diagnostic(overload.owner_path, "missing-overload-candidate", candidate.owner_path) for candidate in overload.candidates if id(candidate) not in functions ) - signatures = tuple(self._class_overload_signature(matches) for matches in overload.candidate_matches) + return ( + *self._overload_cardinality_diagnostics(overload), + *missing, + *self._overload_signature_diagnostics(overload), + ) + + def _overload_cardinality_diagnostics( + self, + overload: OverloadPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Require one match and passed-object record per overload candidate.""" + diagnostics = [] + if not overload.candidates: + diagnostics.append(self._diagnostic(overload.owner_path, "empty-overload", overload.python_name)) + expected = len(overload.candidates) + for actual, code in ( + (len(overload.candidate_matches), "incomplete-overload-match-plan"), + (len(overload.candidate_passed_objects), "incomplete-overload-call-plan"), + ): + if actual != expected: + diagnostics.append(self._diagnostic(overload.owner_path, code, (expected, actual))) + return tuple(diagnostics) + + def _overload_signature_diagnostics( + self, + overload: OverloadPlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject ambiguous predicates and inconsistent receiver selection.""" + diagnostics = [] + signatures = tuple(self._overload_signature(matches) for matches in overload.candidate_matches) if len(set(signatures)) != len(signatures): - diagnostics.append(self._diagnostic(overload.owner_path, "ambiguous-class-overload", overload.python_name)) - if len(set(overload.candidate_passed_objects)) > 1: + diagnostics.append(self._diagnostic(overload.owner_path, "ambiguous-overload", overload.python_name)) + builtin_signatures = tuple(self._overload_builtin_signature(matches) for matches in overload.candidate_matches) + if len(set(builtin_signatures)) != len(builtin_signatures): diagnostics.append( - self._diagnostic(overload.owner_path, "mixed-class-overload-receivers", overload.python_name) + self._diagnostic(overload.owner_path, "overlapping-reflected-overload", overload.python_name) ) + if len(set(overload.candidate_passed_objects)) > 1: + diagnostics.append(self._diagnostic(overload.owner_path, "mixed-overload-receivers", overload.python_name)) return tuple(diagnostics) @staticmethod - def _class_overload_signature(matches: tuple) -> tuple: + def _overload_signature(matches: tuple) -> tuple: """Return the runtime-relevant signature of one overload candidate.""" return tuple( ( @@ -361,6 +379,24 @@ def _class_overload_signature(matches: tuple) -> tuple: for match in matches ) + @staticmethod + def _overload_builtin_signature(matches: tuple) -> tuple: + """Normalize reflected Python scalar domains for overlap validation.""" + return tuple( + ( + match.kind, + match.optional, + ( + overload_builtin_scalar_family(match.semantic_type_name) + if match.accept_builtin_scalar + else match.semantic_type_name + ), + match.rank, + match.derived_type_identity, + ) + for match in matches + ) + def _constructor_diagnostics(self, surface: ClassSurfacePlan) -> tuple[WrapperPlanDiagnostic, ...]: """Require one complete constructor kind and its exact lifecycle.""" constructor = surface.constructor @@ -570,6 +606,7 @@ def _python_export_name_diagnostics(self, plan: NamespacePlan) -> tuple[WrapperP names = [function.binding.python_name for function in plan.functions] names.extend(name for variable in plan.variables for name in variable.binding.python_names) names.extend(name for derived in plan.derived_types for name in derived.python_names) + names.extend(overload.python_name for overload in plan.overloads) return tuple( self._diagnostic(plan.owner_path, "duplicate-python-export", name) for name, count in Counter(names).items() @@ -593,6 +630,12 @@ def _export_owner_diagnostics(self, plan: NamespacePlan) -> tuple[WrapperPlanDia diagnostics.append( self._diagnostic(variable.owner_path, "inconsistent-variable-export-owner", expected_owner) ) + for overload in plan.overloads: + expected_owner = f"{plan.owner_path}.{overload.python_name}" + if overload.owner_path != expected_owner: + diagnostics.append( + self._diagnostic(overload.owner_path, "inconsistent-overload-export-owner", expected_owner) + ) return tuple(diagnostics) def _generated_symbol_diagnostics(self, plan: ModulePlan) -> tuple[WrapperPlanDiagnostic, ...]: @@ -737,6 +780,8 @@ def _module_getter_diagnostics(self, plan: ModuleVariablePlan) -> tuple[WrapperP action = plan.binding.getter_action if action is ModuleGetterAction.NATIVE_ARRAY_HANDLE: return self._module_native_array_handle_diagnostics(plan) + if action is ModuleGetterAction.BORROWED_ARRAY_VIEW: + return self._module_borrowed_array_view_diagnostics(plan) if action is ModuleGetterAction.CONSTANT_VALUE: diagnostics = [] if plan.bridge.getter_role is not None: @@ -757,6 +802,27 @@ def _module_getter_diagnostics(self, plan: ModuleVariablePlan) -> tuple[WrapperP return (self._diagnostic(plan.owner_path, "missing-module-descriptor-kind", action.value),) return () + def _module_borrowed_array_view_diagnostics( + self, + plan: ModuleVariablePlan, + ) -> tuple[WrapperPlanDiagnostic, ...]: + """Validate one native-owned fixed array view and its pointer/shape ABI.""" + diagnostics = [] + array = plan.array + if array is None or array.rank is None or array.rank <= 0: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-module-array-view", array)) + if plan.native_array_handle is not None or plan.derived is not None: + diagnostics.append(self._diagnostic(plan.owner_path, "module-array-view-has-unrelated-facet", None)) + if plan.bridge.getter_role is None: + diagnostics.append(self._diagnostic(plan.owner_path, "missing-module-array-getter-role", None)) + if plan.bridge.native_assignment is not AssignmentMode.NONE: + diagnostics.append( + self._diagnostic( + plan.owner_path, "module-array-view-has-native-assignment", plan.bridge.native_assignment + ) + ) + return tuple(diagnostics) + def _derived_module_getter_role_diagnostics( self, plan: ModuleVariablePlan, @@ -883,7 +949,9 @@ def _module_nonwriting_action_diagnostics( ) -> tuple[WrapperPlanDiagnostic, ...]: """Validate descriptor rejection and constant omission choices.""" action = plan.binding.setter_action - if action is SetterAction.REJECT_REPLACEMENT and plan.derived is not None: + if action is SetterAction.REJECT_REPLACEMENT and ( + plan.derived is not None or plan.binding.getter_action is ModuleGetterAction.BORROWED_ARRAY_VIEW + ): return () if action is SetterAction.REJECT_REPLACEMENT and plan.bridge.descriptor_kind not in { "allocatable", @@ -915,6 +983,7 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost *self._string_result_aggregation_diagnostics(plan), *self._status_error_diagnostics(plan), *self._class_call_diagnostics(plan), + *self._native_invocation_diagnostics(plan), ] slots = {slot.native_position: slot for slot in plan.native_call_slots} for slot in plan.native_call_slots: @@ -929,6 +998,44 @@ def _function_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnost diagnostics.extend(self._string_writeback_diagnostics(plan)) return tuple(diagnostics) + def _native_invocation_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require one internally consistent completed native call syntax.""" + invocation = plan.bridge.native_invocation + validator = { + NativeInvocationKind.PROCEDURE: self._procedure_invocation_diagnostics, + NativeInvocationKind.DEFINED_OPERATOR: self._defined_operator_diagnostics, + NativeInvocationKind.DEFINED_ASSIGNMENT: self._defined_assignment_diagnostics, + }.get(invocation) + if validator is None: + return (self._diagnostic(plan.owner_path, "unknown-native-invocation", invocation),) + return validator(plan) + + def _procedure_invocation_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Reject an operator token on an ordinary procedure call.""" + operator = plan.bridge.native_operator + if operator is None: + return () + return (self._diagnostic(plan.owner_path, "unexpected-native-operator", operator),) + + def _defined_operator_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require the canonical native spelling for a defined operator.""" + operator = plan.bridge.native_operator + expected = f"operator({operator})" if operator else None + compact_name = "".join(plan.bridge.native_name.split()).casefold() + if expected is not None and compact_name == expected: + return () + return (self._diagnostic(plan.owner_path, "invalid-defined-operator", plan.bridge),) + + def _defined_assignment_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: + """Require subroutine form and canonical spelling for defined assignment.""" + compact_name = "".join(plan.bridge.native_name.split()).casefold() + valid = ( + compact_name == "assignment(=)" and plan.bridge.native_operator == "=" and plan.bridge.native_is_subroutine + ) + if valid: + return () + return (self._diagnostic(plan.owner_path, "invalid-defined-assignment", plan.bridge),) + def _class_call_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Validate receiver selection before either backend sees a class call.""" call = plan.class_call @@ -978,15 +1085,17 @@ def _argument_transformation_diagnostics( plan: ArgumentTransferPlan, ) -> tuple[WrapperPlanDiagnostic, ...]: """Validate layer ownership and lifecycle for explicit representation copies.""" - if plan.array is None or plan.array.native_order == plan.array.order: + replacement = self._publishes_array_replacement(plan) + if plan.array is None or (plan.array.native_order == plan.array.order and not replacement): return ( (self._diagnostic(plan.owner_path, "unexpected-transformations", plan.transformations),) if plan.transformations else () ) - representation = self._array_representation_transformation_diagnostics(plan) - if representation: - return representation + if not replacement: + representation = self._array_representation_transformation_diagnostics(plan) + if representation: + return representation return ( *self._transformation_phase_diagnostics(plan), *( @@ -1022,7 +1131,7 @@ def _transformation_phase_diagnostics( expected_phases = [] if plan.binding.codegen_action is not CodegenAction.IDENTITY_OUTPUT: expected_phases.append(WritebackPhase.COPY_IN) - if plan.mutates_native: + if plan.mutates_native or self._publishes_array_replacement(plan): expected_phases.append(WritebackPhase.COPY_OUT) expected_phases.append(WritebackPhase.CLEANUP) if tuple(item.phase for item in plan.transformations) == tuple(expected_phases): @@ -1046,11 +1155,15 @@ def _one_transformation_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "invalid-transformation-layer", transformation.layer.value) ) - expected_action = ( - TransformationAction.RELEASE_TEMPORARY - if transformation.phase is WritebackPhase.CLEANUP - else TransformationAction.COPY_ARRAY_REPRESENTATION - ) + expected_action = { + WritebackPhase.COPY_IN: TransformationAction.COPY_ARRAY_REPRESENTATION, + WritebackPhase.COPY_OUT: ( + TransformationAction.PUBLISH_ARRAY_REPLACEMENT + if self._publishes_array_replacement(plan) + else TransformationAction.COPY_ARRAY_REPRESENTATION + ), + WritebackPhase.CLEANUP: TransformationAction.RELEASE_TEMPORARY, + }[transformation.phase] if transformation.action is not expected_action: diagnostics.append( self._diagnostic( @@ -1061,6 +1174,15 @@ def _one_transformation_diagnostics( ) return tuple(diagnostics) + @staticmethod + def _publishes_array_replacement(plan: ArgumentTransferPlan) -> bool: + """Return whether COPY_OUT transfers a mutable NumPy replacement.""" + return any( + transformation.phase is WritebackPhase.COPY_OUT + and transformation.action is TransformationAction.PUBLISH_ARRAY_REPLACEMENT + for transformation in plan.transformations + ) + def _argument_family_diagnostics( self, plan: ArgumentTransferPlan, @@ -1659,7 +1781,7 @@ def _scalar_boundary_diagnostics( diagnostics.append( self._diagnostic(plan.owner_path, "invalid-scalar-address-data-action", plan.bridge.data_action.value) ) - if plan.binding.optional_mode is not OptionalMode.REQUIRED: + if plan.binding.optional_mode is not OptionalMode.REQUIRED and action is PythonBarrierAction.RAW_ADDRESS: diagnostics.append(self._diagnostic(plan.owner_path, "optional-scalar-address-boundary", action.value)) return tuple(diagnostics) @@ -1764,9 +1886,10 @@ def _array_ownership_diagnostics( ) -> tuple[WrapperPlanDiagnostic, ...]: """Validate caller-owned ordinary-array lifetime facts.""" diagnostics = [] + replacement = plan.transfer_mode is TransferMode.COPY_RETURN expected = ( ("object-kind", plan.object_kind, ObjectKind.NUMPY_ARRAY), - ("owner", plan.ownership_owner, OwnershipOwner.CALLER), + ("owner", plan.ownership_owner, OwnershipOwner.PYTHON if replacement else OwnershipOwner.CALLER), ("storage", plan.storage_mode, StorageMode.STACK), ("boundary-storage", plan.boundary_storage_mode, StorageMode.STACK), ) @@ -1775,8 +1898,16 @@ def _array_ownership_diagnostics( for name, actual, required in expected if actual is not required ) - expected_transfer = TransferMode.IN_PLACE if plan.mutates_native else TransferMode.CALL_LOCAL - expected_destruction = DestructionPolicy.CALLER if plan.mutates_native else DestructionPolicy.NONE + expected_transfer = ( + TransferMode.COPY_RETURN + if replacement + else (TransferMode.IN_PLACE if plan.mutates_native else TransferMode.CALL_LOCAL) + ) + expected_destruction = ( + DestructionPolicy.PYTHON_REFCOUNT + if replacement + else (DestructionPolicy.CALLER if plan.mutates_native else DestructionPolicy.NONE) + ) if plan.transfer_mode is not expected_transfer: diagnostics.append(self._diagnostic(plan.owner_path, "invalid-array-transfer", plan.transfer_mode.value)) if plan.destruction_policy is not expected_destruction: @@ -1842,11 +1973,18 @@ def _native_array_actual_validation_diagnostics( if actual is None: return () diagnostics = [] - if actual.writable != plan.binding.writable: + expected_writable = plan.mutates_native or self._publishes_array_replacement(plan) + if actual.writable != expected_writable: diagnostics.append( self._diagnostic(plan.owner_path, "inconsistent-array-actual-writeability", actual.writable) ) - if not actual.require_native_byte_order or not actual.require_aligned or not actual.require_contiguous: + array = plan.array + expected_contiguous = bool(array is not None and array.contiguous is True) + if ( + not actual.require_native_byte_order + or not actual.require_aligned + or actual.require_contiguous != expected_contiguous + ): diagnostics.append(self._diagnostic(plan.owner_path, "incomplete-array-actual-validation", None)) return tuple(diagnostics) @@ -2114,6 +2252,7 @@ def _array_buffer_action_diagnostics( ) if plan.binding.codegen_action not in { CodegenAction.CALL_LOCAL_INPUT, + CodegenAction.COPY_IN_OUT, CodegenAction.IN_PLACE_ARGUMENT, CodegenAction.IDENTITY_OUTPUT, }: @@ -2629,8 +2768,6 @@ def _string_replacement_diagnostics( for name, actual, required in expected if actual is not required ] - if not plan.mutates_native: - diagnostics.append(self._diagnostic(plan.owner_path, "string-replacement-without-mutation", False)) if not plan.projects_result or plan.result_position is None: diagnostics.append( self._diagnostic(plan.owner_path, "string-replacement-without-result", plan.result_position) @@ -2935,9 +3072,7 @@ def _string_result_aggregation_diagnostics( self, plan: FunctionPlan, ) -> tuple[WrapperPlanDiagnostic, ...]: - """Keep fixed string allocation cleanup single-result in Phase 5B.""" - if any(result.object_kind is ObjectKind.STRING for result in plan.results) and len(plan.results) != 1: - return (self._diagnostic(plan.owner_path, "mixed-string-result-aggregation", len(plan.results)),) + """Mixed fixed strings use the same ordered output aggregation path.""" return () def _string_result_diagnostics(self, plan: ResultPlan) -> tuple[WrapperPlanDiagnostic, ...]: @@ -3708,12 +3843,10 @@ def _lifecycle_operation_count(actions, source_role, operation) -> int: return sum(action.source_role == source_role and action.operation is operation for action in actions) def _mixed_output_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: - """Allow hidden outputs plus writebacks only with one contiguous order.""" + """Require one contiguous public order across results and writebacks.""" if not plan.results or not plan.writeback_actions: return () writebacks = self._projected_writebacks(plan) - if not self._supports_mixed_string_outputs(plan.results, writebacks): - return (self._diagnostic(plan.owner_path, "mixed-result-and-writeback", plan.owner_path),) positions = tuple(result.result_position for result in plan.results) + tuple( action.binding.result_position for action in writebacks ) @@ -3728,14 +3861,6 @@ def _projected_writebacks(plan: FunctionPlan) -> tuple: if action.phase is WritebackPhase.COPY_OUT and action.binding is not None ) - @staticmethod - def _supports_mixed_string_outputs(results: tuple, writebacks: tuple) -> bool: - """Recognize the one legacy-observed hidden-string/writeback envelope.""" - hidden_strings = all( - result.source_kind == "hidden_output" and result.object_kind is ObjectKind.STRING for result in results - ) - return hidden_strings and all(action.object_kind is ObjectKind.STRING for action in writebacks) - def _binding_result_diagnostics(self, plan: FunctionPlan) -> tuple[WrapperPlanDiagnostic, ...]: """Validate ordered consumers and the sole direct native result.""" diagnostics = [] diff --git a/x2py/wrapper_codegen/nodes.py b/x2py/wrapper_codegen/nodes.py index 51aefabc0..1e055f0fd 100644 --- a/x2py/wrapper_codegen/nodes.py +++ b/x2py/wrapper_codegen/nodes.py @@ -373,6 +373,7 @@ class FortranInterfaceProcedure(StageRecord): name: str imports: tuple[str, ...] = () parameters: tuple[FortranParameter, ...] = () + parameter_declarations: tuple[FortranParameter, ...] = () result_name: str | None = None result_type: str | None = None is_subroutine: bool = False diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py index 15e1bf7b8..f5e901c15 100644 --- a/x2py/wrapper_codegen/plan.py +++ b/x2py/wrapper_codegen/plan.py @@ -31,7 +31,7 @@ ClassConstructorKind, ClassInvocationKind, ClassMethodKind, - ClassOverloadMatchKind, + OverloadMatchKind, ClassRegistrationAction, ConstructionLifecycleAction, DerivedActualAccess, @@ -63,6 +63,7 @@ NativeArrayRelease, NativeArraySourceKind, NativeDescriptorHandoffABI, + NativeInvocationKind, OptionalMode, PythonExceptionKind, TransformationAction, @@ -205,7 +206,8 @@ class ConstructorPlan(StageRecord): lifecycle: tuple[ConstructionLifecycleAction, ...] rejection_message: str | None = None target: FunctionPlan | None = None - overload: ClassOverloadPlan | None = None + overload: OverloadPlan | None = None + docstring: str = "" @dataclass @@ -218,30 +220,35 @@ class ClassMethodPlan(StageRecord): passed_object_position: int | None public: bool function: FunctionPlan + docstring: str = "" @dataclass -class ClassOverloadArgumentMatchPlan(StageRecord): +class OverloadArgumentMatchPlan(StageRecord): """One editable exact-type predicate for overload dispatch.""" python_name: str - kind: ClassOverloadMatchKind + kind: OverloadMatchKind optional: bool semantic_type_name: str rank: int derived_type_identity: tuple[str, str] | None + accept_builtin_scalar: bool = False @dataclass -class ClassOverloadPlan(StageRecord): - """One class-owned overload and its concrete editable candidates.""" +class OverloadPlan(StageRecord): + """One exact-match overload and its concrete editable candidates.""" owner_path: str python_name: str kind: str candidates: tuple[FunctionPlan, ...] - candidate_matches: tuple[tuple[ClassOverloadArgumentMatchPlan, ...], ...] + candidate_matches: tuple[tuple[OverloadArgumentMatchPlan, ...], ...] candidate_passed_objects: tuple[bool, ...] + unsupported_extra_argument_message: str | None = None + identity_receiver_shortcut: bool = False + docstring: str = "" @dataclass @@ -254,7 +261,7 @@ class ClassSurfacePlan(StageRecord): base_identities: tuple[tuple[str, str], ...] constructor: ConstructorPlan methods: tuple[ClassMethodPlan, ...] - overloads: tuple[ClassOverloadPlan, ...] + overloads: tuple[OverloadPlan, ...] registration: tuple[ClassRegistrationAction, ...] docstring: str = "" @@ -433,8 +440,10 @@ class ModuleVariablePlan(StageRecord): datatype_family: DatatypeFamily binding: BindingModuleVariablePlan bridge: BridgeModuleVariablePlan + array: ArrayHandoffPlan | None native_array_handle: NativeArrayHandlePlan | None derived: DerivedModuleObjectPlan | None = None + docstring: str = "" @dataclass @@ -445,6 +454,7 @@ class BindingFunctionPlan(StageRecord): docstring: str hold_gil: bool status_error: BindingStatusErrorPlan | None + public: bool = True @dataclass @@ -452,6 +462,8 @@ class BridgeFunctionPlan(StageRecord): """Bridge-facing function facts.""" native_name: str + native_invocation: NativeInvocationKind + native_operator: str | None external: bool native_module: str | None native_is_subroutine: bool @@ -744,6 +756,7 @@ class NamespacePlan(StageRecord): variables: tuple[ModuleVariablePlan, ...] = () derived_types: tuple[DerivedTypePlan, ...] = () classes: tuple[ClassSurfacePlan, ...] = () + overloads: tuple[OverloadPlan, ...] = () docstring: str = "" diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py index 82f4ee153..b9d406aeb 100644 --- a/x2py/wrapper_codegen/planner.py +++ b/x2py/wrapper_codegen/planner.py @@ -3,13 +3,13 @@ from __future__ import annotations from collections import Counter, defaultdict +from dataclasses import replace from x2py.semantics import models from x2py.semantics.native_array_handles import NATIVE_ARRAY_POINTER_C_DESCRIPTOR_HEADER from x2py.semantics.wrapper_policy import ( ArgumentHandoffMode, ArrayHandoffPolicy, - CallbackABIKind, CallbackHandoffPolicy, CallbackResultPolicy, CallbackTransferPolicy, @@ -22,12 +22,12 @@ ModuleGetterAction, ModuleObjectAccessMechanism, ModuleVariablePolicy, + OverloadPolicy, OptionalMode, ArgumentPolicy, FunctionWrapperPolicy, LifecycleOperation, LifecyclePolicy, - NativeArrayDescriptorKind, NativeCallSlotPolicy, NativeArrayActualPolicy, NativeArrayHandleWrapperPolicy, @@ -46,6 +46,7 @@ ) from x2py.semantics.wrapper_exports import PythonExportPolicy from x2py.semantics.ownership import NativeBarrierAction, SetterAction +from x2py.wrapper_codegen.docstrings import WrapperDocstringBuilder from x2py.wrapper_codegen.plan import ( ArrayHandoffPlan, ArgumentTransferPlan, @@ -67,8 +68,8 @@ CallbackTransferPlan, ClassMethodPlan, ClassCallPlan, - ClassOverloadArgumentMatchPlan, - ClassOverloadPlan, + OverloadArgumentMatchPlan, + OverloadPlan, ClassSurfacePlan, ConstructorFieldPlan, ConstructorPlan, @@ -113,19 +114,6 @@ "String": DatatypeFamily.STRING, } -_DOCUMENTATION_SCALAR_TYPES = { - "Bool": "bool", - "Int8": "int8", - "Int16": "int16", - "Int32": "int32", - "Int64": "int64", - "Float32": "float32", - "Float64": "float64", - "Complex64": "complex64", - "Complex128": "complex128", - "String": "str", -} - class WrapperPlanner(ClassVisitor): """Project completed semantic policies into one editable shared plan.""" @@ -134,6 +122,7 @@ def __init__(self, *, support_analyzer: WrapperPlanSupportAnalyzer | None = None """Create a planner with an optional support analyzer.""" super().__init__() self.support_analyzer = support_analyzer or WrapperPlanSupportAnalyzer() + self.docstrings = WrapperDocstringBuilder() def build(self, module: models.SemanticModule) -> ModulePlan: """Mechanically project one editable wrapper plan.""" @@ -147,9 +136,10 @@ def _visit_SemanticModule(self, module: models.SemanticModule) -> ModulePlan: if not report.supported: raise ValueError(self._support_error(module.name, report.blockers)) self._complete_derived_backend_symbols(module) - functions, variables, derived_types, classes = self._namespace_member_plans(module) + functions, variables, derived_types, classes, overloads = self._namespace_member_plans(module) self._attach_class_functions(functions, classes) - namespaces = self._namespace_plans(module.name, functions, variables, derived_types, classes) + self._attach_overload_functions(functions, overloads) + namespaces = self._namespace_plans(module.name, functions, variables, derived_types, classes, overloads) return ModulePlan( owner_path=module.name, binding=BindingModulePlan(module.name), @@ -158,13 +148,14 @@ def _visit_SemanticModule(self, module: models.SemanticModule) -> ModulePlan: required_headers=self._required_headers(namespaces), ) - def _namespace_member_plans(self, module: models.SemanticModule) -> tuple[dict, dict, dict, dict]: - """Build the four namespace-owned plan maps before linking classes.""" + def _namespace_member_plans(self, module: models.SemanticModule) -> tuple[dict, dict, dict, dict, dict]: + """Build namespace-owned plan maps before linking private callables.""" return ( self._functions_by_namespace(module), self._variables_by_namespace(module), self._derived_types_by_namespace(module), self._classes_by_namespace(module), + self._module_overloads_by_namespace(module), ) def _attach_class_functions(self, functions: dict, classes: dict) -> None: @@ -174,6 +165,12 @@ def _attach_class_functions(self, functions: dict, classes: dict) -> None: function for surface in surfaces for function in self._class_function_plans(surface) ) + @staticmethod + def _attach_overload_functions(functions: dict, overloads: dict) -> None: + """Expose private module-overload candidates to the shared C method table.""" + for namespace, plans in overloads.items(): + functions[namespace].extend(candidate for overload in plans for candidate in overload.candidates) + def _namespace_plans( self, module_name: str, @@ -181,10 +178,11 @@ def _namespace_plans( variables: dict, derived_types: dict, classes: dict, + overloads: dict, ) -> tuple[NamespacePlan, ...]: """Freeze linked namespace members in dependency-safe path order.""" self._complete_generated_symbols(functions, variables) - namespace_paths = self._namespace_paths((*functions, *variables, *derived_types, *classes)) + namespace_paths = self._namespace_paths((*functions, *variables, *derived_types, *classes, *overloads)) return tuple( self._namespace_plan( module_name, @@ -193,6 +191,7 @@ def _namespace_plans( tuple(variables[path]), tuple(derived_types[path]), tuple(classes[path]), + tuple(overloads[path]), ) for path in namespace_paths ) @@ -205,6 +204,7 @@ def _namespace_plan( variables: tuple[ModuleVariablePlan, ...], derived_types: tuple[DerivedTypePlan, ...], classes: tuple[ClassSurfacePlan, ...], + overloads: tuple[OverloadPlan, ...], ) -> NamespacePlan: """Freeze one namespace with stable public documentation.""" return NamespacePlan( @@ -214,30 +214,15 @@ def _namespace_plan( variables=variables, derived_types=derived_types, classes=classes, - docstring=self._namespace_docstring(module_name, path, functions, classes), - ) - - @staticmethod - def _namespace_docstring( - module_name: str, - path: tuple[str, ...], - functions: tuple[FunctionPlan, ...], - classes: tuple[ClassSurfacePlan, ...], - ) -> str: - """List the completed callable and class exports in one namespace.""" - qualified_name = ".".join((module_name, *path)) - return "\n".join( - ( - qualified_name, - "", - "Functions", - "---------", - *(function.binding.python_name for function in functions), - "", - "Classes", - "-------", - *(name for surface in classes for name in surface.python_names), - ) + overloads=overloads, + docstring=self.docstrings.namespace( + module_name, + path, + functions, + variables, + classes, + overloads, + ), ) def _complete_derived_backend_symbols(self, module: models.SemanticModule) -> None: @@ -364,7 +349,7 @@ def _class_surface_plan( ) -> ClassSurfacePlan: """Compose one class plan from completed method and constructor facts.""" methods = self._class_method_plans(module_name, namespace, semantic_class, policy) - overloads_by_name = {overload.owner_path.rsplit(".", 1)[-1]: overload for overload in policy.overloads} + overloads_by_name = {overload.python_name: overload for overload in policy.overloads} overloads = self._class_overload_plans( module_name, namespace, @@ -372,60 +357,36 @@ def _class_surface_plan( policy.type_identity, overloads_by_name, ) + fields = tuple(self._derived_field_plan(field) for field in policy.effective_fields) + constructor = self._constructor_plan( + module_name, + namespace, + semantic_class, + policy, + methods, + overloads_by_name, + python_name=python_names[0], + fields=fields, + ) return ClassSurfacePlan( owner_path=policy.owner_path, type_identity=policy.type_identity, python_names=python_names, base_identities=policy.base_identities, - constructor=self._constructor_plan( - module_name, - namespace, - semantic_class, - policy, - methods, - overloads_by_name, - ), + constructor=constructor, methods=methods, overloads=overloads, registration=policy.registration, - docstring=self._class_docstring(policy, python_names), - ) - - def _class_docstring( - self, - policy: ClassSurfacePolicy, - python_names: tuple[str, ...], - ) -> str: - """Describe fields and methods from the completed class surface.""" - fields = self._class_documented_fields(policy) - methods = self._class_documented_methods(policy) - return "\n".join( - ( + docstring=self.docstrings.class_surface( python_names[0], - "", - "Fields", - "------", - *(fields or ("None",)), - "", - "Methods", - "-------", - *(methods or ("None",)), - ) - ) - - def _class_documented_fields(self, policy: ClassSurfacePolicy) -> tuple[str, ...]: - """Return concise public field signatures in declaration order.""" - return tuple( - f"{field.name} : {self._derived_field_documentation_type(field)}" for field in policy.effective_fields + policy.type_identity[1], + constructor, + fields, + methods, + overloads, + ), ) - @staticmethod - def _class_documented_methods(policy: ClassSurfacePolicy) -> tuple[str, ...]: - """Return concrete and overloaded public descriptors in plan order.""" - concrete = tuple(method.python_name for method in policy.methods if method.public) - overloaded = tuple(overload.python_name for overload in policy.overloads) - return (*concrete, *overloaded) - def _class_method_plans( self, module_name: str, @@ -470,13 +431,20 @@ def _class_overload_plans( namespace: tuple[str, ...], semantic_class: models.SemanticClass, type_identity: tuple[str, str], - policies: dict, - ) -> tuple[ClassOverloadPlan, ...]: + policies: dict[str, OverloadPolicy], + ) -> tuple[OverloadPlan, ...]: """Link every non-constructor overload set to ordinary function plans.""" + functions = self._class_overload_functions(semantic_class, policies.values()) return tuple( - self._class_overload_plan(module_name, namespace, type_identity, overload, policies[overload.name]) - for overload in semantic_class.overload_sets - if overload.name != "__init__" + self._overload_plan( + module_name, + namespace, + policy, + functions, + private_name=lambda name, index: self._class_callable_name(type_identity, f"{name}_{index}"), + ) + for policy in policies.values() + if policy.python_name != "__init__" ) def _constructor_plan( @@ -487,6 +455,9 @@ def _constructor_plan( policy: ClassSurfacePolicy, methods: tuple[ClassMethodPlan, ...], overloads_by_name: dict, + *, + python_name: str, + fields: tuple[DerivedFieldPlan, ...], ) -> ConstructorPlan: """Link one completed constructor to its target and lifecycle records.""" constructor = policy.constructor @@ -498,7 +469,7 @@ def _constructor_plan( policy.type_identity, overloads_by_name, ) - return ConstructorPlan( + plan = ConstructorPlan( kind=constructor.kind, fields=tuple(self._constructor_field_plan(field) for field in constructor.fields), target_owner_path=constructor.target_owner_path, @@ -508,6 +479,8 @@ def _constructor_plan( target=target, overload=overload, ) + plan.docstring = self.docstrings.constructor(python_name, plan, fields) + return plan def _constructor_overload_plan( self, @@ -515,18 +488,19 @@ def _constructor_overload_plan( namespace: tuple[str, ...], semantic_class: models.SemanticClass, type_identity: tuple[str, str], - policies: dict, - ) -> ClassOverloadPlan | None: + policies: dict[str, OverloadPolicy], + ) -> OverloadPlan | None: """Return the constructor-owned overload set, when one was completed.""" - for overload in semantic_class.overload_sets: - if overload.name == "__init__": - return self._class_overload_plan( - module_name, - namespace, - type_identity, - overload, - policies[overload.name], - ) + policy = policies.get("__init__") + if policy is not None: + functions = self._class_overload_functions(semantic_class, (policy,)) + return self._overload_plan( + module_name, + namespace, + policy, + functions, + private_name=lambda name, index: self._class_callable_name(type_identity, f"{name}_{index}"), + ) return None @staticmethod @@ -554,8 +528,9 @@ def _class_method_plan( function_policy, PythonExportPolicy(namespace, private_name), module_name, + public=False, ) - return ClassMethodPlan( + plan = ClassMethodPlan( owner_path=policy.owner_path, python_name=policy.python_name, kind=policy.kind, @@ -563,47 +538,87 @@ def _class_method_plan( public=policy.public, function=function, ) + plan.docstring = self.docstrings.method(plan) + return plan - def _class_overload_plan( + def _overload_plan( self, module_name: str, namespace: tuple[str, ...], - type_identity: tuple[str, str], - overload: models.ProcedureOverloadSet, - policy, - ) -> ClassOverloadPlan: + policy: OverloadPolicy, + functions: dict[str, models.SemanticFunction], + *, + private_name, + ) -> OverloadPlan: """Link one overload plan to its explicit concrete candidates.""" candidates = tuple( self._function_plan( - completed_function_wrapper_policy(procedure), + completed_function_wrapper_policy(functions[candidate.owner_path]), PythonExportPolicy( namespace, - self._class_callable_name(type_identity, f"{overload.name}_{index}"), + private_name(policy.python_name, index), ), module_name, + public=False, ) - for index, procedure in enumerate(overload.procedures) + for index, candidate in enumerate(policy.candidates) ) - return ClassOverloadPlan( + plan = OverloadPlan( owner_path=policy.owner_path, python_name=policy.python_name, kind=policy.kind, candidates=candidates, candidate_matches=tuple( tuple( - ClassOverloadArgumentMatchPlan( + OverloadArgumentMatchPlan( python_name=argument.python_name, kind=argument.kind, optional=argument.optional, semantic_type_name=argument.semantic_type_name, rank=argument.rank, derived_type_identity=argument.derived_type_identity, + accept_builtin_scalar=argument.accept_builtin_scalar, ) for argument in candidate.arguments ) for candidate in policy.candidates ), candidate_passed_objects=tuple(candidate.passed_object for candidate in policy.candidates), + unsupported_extra_argument_message=policy.unsupported_extra_argument_message, + identity_receiver_shortcut=policy.identity_receiver_shortcut, + ) + plan.docstring = self.docstrings.overload(plan) + return plan + + @staticmethod + def _class_overload_functions( + semantic_class: models.SemanticClass, + policies, + ) -> dict[str, models.SemanticFunction]: + """Index only concrete class procedures selected by completed overload policy.""" + selected = WrapperPlanner._selected_overload_owner_paths(policies) + owner_path = completed_class_surface_policy(semantic_class).owner_path + return { + path: procedure + for path, procedure in WrapperPlanner._class_overload_entries(semantic_class, owner_path) + if path in selected + } + + @staticmethod + def _selected_overload_owner_paths(policies) -> set[str]: + """Return concrete owner paths referenced by completed overloads.""" + return {candidate.owner_path for policy in policies for candidate in policy.candidates} + + @staticmethod + def _class_overload_entries( + semantic_class: models.SemanticClass, + owner_path: str, + ) -> tuple[tuple[str, models.SemanticFunction], ...]: + """Pair every concrete class procedure with its completed owner path.""" + return tuple( + (f"{owner_path}.{overload.name}.{procedure.name}", procedure) + for overload in semantic_class.overload_sets + for procedure in overload.procedures ) def _class_callable_name(self, type_identity: tuple[str, str], name: str) -> str: @@ -648,35 +663,12 @@ def _derived_field_plan(self, policy: DerivedFieldPolicy) -> DerivedFieldPlan: array=array, ), derived=self._derived_handoff_plan(policy.derived), - docstring=self._derived_field_docstring(policy), + docstring="", ) + plan.docstring = self.docstrings.field(plan) self._derived_field_plans[policy.owner_path] = plan return plan - def _derived_field_docstring(self, policy: DerivedFieldPolicy) -> str: - """Describe one generated property from its completed field policy.""" - lines = [f"{policy.name} : {self._derived_field_documentation_type(policy)}"] - if policy.native_array_handle is not None: - descriptor = policy.native_array_handle.descriptor_kind.value - lines.append(f" Provides a live {descriptor} array descriptor handle.") - return "\n".join(lines) - - @staticmethod - def _derived_field_documentation_type(policy: DerivedFieldPolicy) -> str: - """Spell the public field type without backend inspection.""" - scalar = _DOCUMENTATION_SCALAR_TYPES.get(policy.semantic_type_name, policy.semantic_type_name) - if policy.native_array_handle is not None: - prefix = ( - "AllocatableArray" - if policy.native_array_handle.descriptor_kind is NativeArrayDescriptorKind.ALLOCATABLE - else "PointerArray" - ) - return f"{prefix}[{scalar}]" - if policy.array is not None: - element = "bytes" if policy.semantic_type_name == "String" else scalar - return f"ndarray[{element}]" - return scalar - def _functions_by_namespace(self, module: models.SemanticModule) -> dict[tuple[str, ...], list[FunctionPlan]]: """Group exported function plans by completed Python namespace.""" functions = defaultdict(list) @@ -690,6 +682,65 @@ def _functions_by_namespace(self, module: models.SemanticModule) -> dict[tuple[s functions[export.namespace].append(self._function_plan(policy, export, module.name)) return functions + def _module_overloads_by_namespace( + self, + module: models.SemanticModule, + ) -> dict[tuple[str, ...], list[OverloadPlan]]: + """Group completed module generics and their private concrete calls.""" + grouped = defaultdict(list) + policies = module.metadata.get(models.RESOLVED_MODULE_OVERLOAD_POLICIES_METADATA, ()) + functions = self._module_overload_function_index(module) + for item in policies: + policy = self._completed_module_overload_policy(module, item) + for export in policy.python_exports: + grouped[export.namespace].append(self._exported_module_overload_plan(module, policy, export, functions)) + return grouped + + @staticmethod + def _module_overload_function_index( + module: models.SemanticModule, + ) -> dict[str, models.SemanticFunction]: + """Index concrete module-generic procedures by completed owner path.""" + return { + f"{(procedure.origin.native_scope or module.name)!s}.{overload.name}.{procedure.name}": procedure + for overload in module.overload_sets + for procedure in overload.procedures + } + + @staticmethod + def _completed_module_overload_policy(module: models.SemanticModule, policy) -> OverloadPolicy: + """Require one completed module-overload policy before plan projection.""" + if not isinstance(policy, OverloadPolicy): + raise ValueError(f"Module {module.name!r} has an incomplete overload policy") + return policy + + def _exported_module_overload_plan( + self, + module: models.SemanticModule, + policy: OverloadPolicy, + export: PythonExportPolicy, + functions: dict[str, models.SemanticFunction], + ) -> OverloadPlan: + """Project one namespace export onto the shared concrete candidates.""" + exported = replace( + policy, + owner_path=self._export_owner_path(module.name, export.namespace, export.name), + python_name=export.name, + ) + return self._overload_plan( + module.name, + export.namespace, + exported, + functions, + private_name=self._module_overload_callable_name, + ) + + @staticmethod + def _module_overload_callable_name(name: str, index: int) -> str: + """Return one private Python export for a module-overload candidate.""" + stem = name.strip("_").casefold() or "call" + return f"_x2py_overload_{stem}_{index}" + @staticmethod def _module_function_policy(function: models.SemanticFunction) -> FunctionWrapperPolicy | None: """Return a public function plan policy, excluding class-only root targets.""" @@ -765,7 +816,7 @@ def _module_variable_plan( ) -> ModuleVariablePlan: getter_role = self._module_getter_role(policy) setter_role = f"{policy.owner_path}:setter" if policy.setter_action is SetterAction.WRITE_THROUGH else None - return ModuleVariablePlan( + plan = ModuleVariablePlan( owner_path=self._export_owner_path(module_name, namespace, python_names[0]), symbol_name=policy.native_name.casefold(), semantic_type_name=policy.semantic_type_name, @@ -789,6 +840,7 @@ def _module_variable_plan( getter_role=getter_role, setter_role=setter_role, ), + array=self._array_plan(policy.array, policy.owner_path), native_array_handle=self._native_array_handle_plan(policy.native_array_handle, policy.owner_path), derived=( DerivedModuleObjectPlan( @@ -809,7 +861,10 @@ def _module_variable_plan( if policy.derived is not None else None ), + docstring="", ) + plan.docstring = self.docstrings.module_variable(plan) + return plan @staticmethod def _module_getter_role(policy: ModuleVariablePolicy) -> str | None: @@ -825,22 +880,33 @@ def _function_plan( policy: FunctionWrapperPolicy, export: PythonExportPolicy, module_name: str, + *, + public: bool = True, ) -> FunctionPlan: """Return one exported function plan from completed policy.""" native_call_slots = self._native_slot_plans(policy) arguments = self._argument_plans(policy, native_call_slots) results = self._result_plans(policy, native_call_slots) + status_error = self._status_error_plan(policy.status_error, native_call_slots) return FunctionPlan( owner_path=self._export_owner_path(module_name, export.namespace, export.name), symbol_name=export.name.casefold(), binding=BindingFunctionPlan( python_name=export.name, - docstring=self._function_docstring(export.name, arguments, results), + docstring=self.docstrings.function( + export.name, + arguments, + results, + status_error=status_error, + ), hold_gil=policy.hold_gil, - status_error=self._status_error_plan(policy.status_error, native_call_slots), + status_error=status_error, + public=public, ), bridge=BridgeFunctionPlan( policy.native_name, + policy.native_invocation, + policy.native_operator, policy.external, policy.native_module, policy.native_is_subroutine, @@ -874,257 +940,6 @@ def _class_call_plan(policy: FunctionWrapperPolicy) -> ClassCallPlan | None: type_bound_name=policy.class_call.type_bound_name, ) - # Stable Python function documentation from completed transfer plans. - def _function_docstring( - self, - python_name: str, - arguments: tuple[ArgumentTransferPlan, ...], - results: tuple[ResultPlan, ...], - ) -> str: - """Describe the Python boundary without asking a backend to infer policy.""" - visible_arguments = tuple(argument for argument in arguments if argument.python_visible) - documented_outputs = self._documented_outputs(arguments, results) - parameter_names = ", ".join(argument.binding.python_name for argument in visible_arguments) - result_summary = self._result_documentation_summary(documented_outputs) - lines = ( - f"{python_name}({parameter_names}) -> {result_summary}", - *self._parameter_documentation_section(visible_arguments), - *self._return_documentation_section(documented_outputs, arguments), - "", - "Raises", - "------", - "TypeError", - " If an argument violates its completed dtype, rank, shape, layout, or handle contract.", - ) - return "\n".join(lines) - - def _parameter_documentation_section( - self, - arguments: tuple[ArgumentTransferPlan, ...], - ) -> tuple[str, ...]: - """Return the parameter section for visible completed transfers.""" - if not arguments: - return () - body = tuple(line for argument in arguments for line in self._argument_documentation_lines(argument)) - return ("", "Parameters", "----------", *body) - - def _return_documentation_section( - self, - outputs: tuple[ArgumentTransferPlan | ResultPlan, ...], - arguments: tuple[ArgumentTransferPlan, ...], - ) -> tuple[str, ...]: - """Return the result section for ordered result transfers.""" - body = tuple( - line - for output in outputs - for line in ( - self._projected_argument_documentation_lines(output) - if isinstance(output, ArgumentTransferPlan) - else self._result_documentation_lines(output, arguments) - ) - ) - return ("", "Returns", "-------", *(body or ("None",))) - - def _argument_documentation_lines(self, argument: ArgumentTransferPlan) -> tuple[str, ...]: - """Render one argument solely from its completed transfer facts.""" - optional = argument.binding.optional_mode is not OptionalMode.REQUIRED or ( - argument.nullable and argument.native_array_handle is None - ) - type_name = self._transfer_documentation_type(argument, nullable=optional, signature=False) - lines = [f"{argument.binding.python_name} : {type_name}"] - lines.extend(self._array_documentation_lines(argument.array)) - lines.extend(self._argument_handle_documentation_lines(argument)) - lines.extend(self._argument_optional_documentation_lines(argument, optional=optional)) - lines.extend((" Mutates: yes",) if argument.mutates_native else ()) - return tuple(lines) - - def _projected_argument_documentation_lines( - self, - argument: ArgumentTransferPlan, - ) -> tuple[str, ...]: - """Render one identity/replacement output owned by an argument transfer.""" - nullable = argument.binding.optional_mode is not OptionalMode.REQUIRED - type_name = self._transfer_documentation_type(argument, nullable=nullable, signature=False) - lines = [f"{argument.binding.python_name} : {type_name}"] - lines.extend(self._array_documentation_lines(argument.array)) - lines.extend(self._argument_handle_documentation_lines(argument)) - return tuple(lines) - - @staticmethod - def _argument_handle_documentation_lines(argument: ArgumentTransferPlan) -> tuple[str, ...]: - """Describe persistent descriptor ownership when one is planned.""" - if argument.native_array_handle is None: - return () - return (f" Descriptor ownership: {argument.native_array_handle.descriptor_ownership.value}",) - - @staticmethod - def _argument_optional_documentation_lines( - argument: ArgumentTransferPlan, - *, - optional: bool, - ) -> tuple[str, ...]: - """Describe omission and present-null semantics selected by policy.""" - if argument.binding.optional_mode is OptionalMode.DESCRIPTOR: - return ( - " Omit to make the native optional dummy absent.", - " Pass None for a present unallocated or unassociated descriptor.", - ) - if argument.binding.optional_mode is OptionalMode.REQUIRED_DESCRIPTOR: - return (" Pass None for an unallocated or unassociated required descriptor.",) - if optional: - return (" May be omitted or passed as None.",) - return () - - def _result_documentation_lines( - self, - result: ResultPlan, - arguments: tuple[ArgumentTransferPlan, ...], - ) -> tuple[str, ...]: - """Render one result from its owning result plan and projection index.""" - name = self._result_documentation_name(result, arguments) - type_name = self._transfer_documentation_type(result, nullable=result.nullable, signature=False) - lines = [f"{name} : {type_name}"] - lines.extend(self._array_documentation_lines(result.array)) - if result.native_array_handle is not None: - handle = result.native_array_handle - lines.append(f" Descriptor ownership: {handle.descriptor_ownership.value}") - state = "Unallocated" if handle.descriptor_kind is NativeArrayDescriptorKind.ALLOCATABLE else "Unassociated" - lines.append(f" {state} state remains inside the returned handle.") - return tuple(lines) - - def _result_documentation_summary( - self, - outputs: tuple[ArgumentTransferPlan | ResultPlan, ...], - ) -> str: - """Return the stable signature spelling for ordered Python results.""" - types = tuple( - self._transfer_documentation_type( - output, - nullable=( - output.binding.optional_mode is not OptionalMode.REQUIRED - if isinstance(output, ArgumentTransferPlan) - else output.nullable - ), - signature=True, - ) - for output in outputs - ) - if not types: - return "None" - if len(types) == 1: - return types[0] - return f"tuple[{', '.join(types)}]" - - @staticmethod - def _documented_outputs( - arguments: tuple[ArgumentTransferPlan, ...], - results: tuple[ResultPlan, ...], - ) -> tuple[ArgumentTransferPlan | ResultPlan, ...]: - """Merge result transfers with argument-owned projected outputs by position.""" - by_position = dict(WrapperPlanner._projected_documented_outputs(arguments)) - by_position.update((result.result_position, result) for result in results) - return tuple(by_position[position] for position in sorted(by_position)) - - @staticmethod - def _projected_documented_outputs( - arguments: tuple[ArgumentTransferPlan, ...], - ) -> tuple[tuple[int, ArgumentTransferPlan], ...]: - """Return projected argument outputs with concrete result positions.""" - return tuple( - (argument.result_position, argument) - for argument in arguments - if argument.projects_result and argument.result_position is not None - ) - - def _transfer_documentation_type(self, transfer, *, nullable: bool, signature: bool) -> str: - """Map one completed transfer representation to its Python documentation type.""" - type_name = self._transfer_base_documentation_type(transfer) - return self._nullable_documentation_type(type_name, nullable=nullable, signature=signature) - - def _transfer_base_documentation_type(self, transfer) -> str: - """Map a completed representation to its non-null Python type.""" - if transfer.datatype_family is DatatypeFamily.CALLBACK: - return self._callback_documentation_type(transfer.callback) - if transfer.datatype_family is DatatypeFamily.DERIVED: - return transfer.semantic_type_name - return self._non_derived_documentation_type(transfer) - - def _non_derived_documentation_type(self, transfer) -> str: - """Spell scalar, handle, and array Python types from completed plans.""" - scalar_type = _DOCUMENTATION_SCALAR_TYPES[transfer.semantic_type_name] - if transfer.native_array_handle is not None: - return self._native_handle_documentation_type(transfer, scalar_type) - if transfer.array is not None: - element_type = "bytes" if transfer.datatype_family is DatatypeFamily.STRING else scalar_type - return f"ndarray[{element_type}]" - return scalar_type - - def _callback_documentation_type(self, callback: CallbackHandoffPlan | None) -> str: - """Render one callable signature directly from its typed transfer plan.""" - if callback is None: - raise ValueError("Callback documentation requires a completed handoff plan") - arguments = ", ".join(self._callback_transfer_documentation_type(item) for item in callback.arguments) - result = ( - "None" - if callback.result.transfer is None - else self._callback_transfer_documentation_type(callback.result.transfer) - ) - return f"Callable[[{arguments}], {result}]" - - @staticmethod - def _callback_transfer_documentation_type(transfer: CallbackTransferPlan) -> str: - """Spell one callback-side Python value from completed ABI facts.""" - if transfer.derived_type_identity is not None: - return transfer.semantic_type_name - scalar = _DOCUMENTATION_SCALAR_TYPES.get(transfer.semantic_type_name, transfer.semantic_type_name) - if transfer.array is not None or transfer.abi is CallbackABIKind.REFERENCE: - return f"ndarray[{scalar}]" - return scalar - - @staticmethod - def _native_handle_documentation_type(transfer, scalar_type: str) -> str: - """Spell one allocatable or pointer array handle type.""" - prefix = ( - "AllocatableArray" - if transfer.native_array_handle.descriptor_kind is NativeArrayDescriptorKind.ALLOCATABLE - else "PointerArray" - ) - return f"{prefix}[{scalar_type}]" - - @staticmethod - def _nullable_documentation_type(type_name: str, *, nullable: bool, signature: bool) -> str: - """Add signature or section nullability spelling when selected.""" - if not nullable: - return type_name - return f"{type_name} | None" if signature else f"{type_name} or None" - - @staticmethod - def _array_documentation_lines(array: ArrayHandoffPlan | None) -> tuple[str, ...]: - """Describe rank and layout already selected in one array handoff plan.""" - if array is None: - return () - if array.rank is None: - return (" Rank: 1..15", " Layout: F-contiguous") - lines = [f" Rank: {array.rank}"] - if array.rank > 1: - layout = "C-contiguous" if array.order == "ORDER_C" else "F-contiguous" - lines.append(f" Layout: {layout}") - return tuple(lines) - - @staticmethod - def _result_documentation_name( - result: ResultPlan, - arguments: tuple[ArgumentTransferPlan, ...], - ) -> str: - """Name a projected result through its owning argument when available.""" - projected = next( - (argument for argument in arguments if argument.result_position == result.result_position), - None, - ) - if projected is not None: - return projected.binding.python_name - return result.bridge.native_name or "result" - def _argument_plans( self, policy: FunctionWrapperPolicy, diff --git a/x2py/wrapper_codegen/printers/__init__.py b/x2py/wrapper_codegen/printers/__init__.py new file mode 100644 index 000000000..0f9a398f0 --- /dev/null +++ b/x2py/wrapper_codegen/printers/__init__.py @@ -0,0 +1,13 @@ +"""Canonical source and semantic-contract printers.""" + +from .pyi_printer import PyiPrinter, emit_module, emit_module_stubs, opaque_dependency_modules +from .source_printers import CSourcePrinter, FortranSourcePrinter + +__all__ = ( + "CSourcePrinter", + "FortranSourcePrinter", + "PyiPrinter", + "emit_module", + "emit_module_stubs", + "opaque_dependency_modules", +) diff --git a/x2py/codegen/printers/pyi_printer.py b/x2py/wrapper_codegen/printers/pyi_printer.py similarity index 100% rename from x2py/codegen/printers/pyi_printer.py rename to x2py/wrapper_codegen/printers/pyi_printer.py diff --git a/x2py/wrapper_codegen/source_printers.py b/x2py/wrapper_codegen/printers/source_printers.py similarity index 94% rename from x2py/wrapper_codegen/source_printers.py rename to x2py/wrapper_codegen/printers/source_printers.py index abebe763f..8ef57898a 100644 --- a/x2py/wrapper_codegen/source_printers.py +++ b/x2py/wrapper_codegen/printers/source_printers.py @@ -456,25 +456,43 @@ def _visit_FortranInterface(self, node: FortranInterface) -> str: def _visit_FortranInterfaceProcedure(self, node: FortranInterfaceProcedure) -> str: """Render one native procedure declaration inside an interface.""" kind = "subroutine" if node.is_subroutine else "function" - suffix = f" result({node.result_name})" if node.result_name is not None else "" - binding = ( - f' bind(c, name="{node.bind_name}")' if node.bind_name is not None else " bind(c)" if node.bind_c else "" - ) - lines = [ - self._continued_call( - f"{kind} {node.name}(", - tuple(parameter.name for parameter in node.parameters), - suffix=f"){binding}{suffix}", - ) - ] - if node.imports: - lines.append(self._indented(f"import :: {', '.join(node.imports)}")) - lines.extend(self._indented(self.visit(parameter)) for parameter in node.parameters) - if node.result_name is not None and node.result_type is not None: - lines.append(self._indented(f"{node.result_type} :: {node.result_name}")) + lines = [self._interface_procedure_signature(node, kind)] + lines.extend(self._interface_import_lines(node)) + declarations = node.parameter_declarations or node.parameters + lines.extend(self._indented(self.visit(parameter)) for parameter in declarations) + lines.extend(self._interface_result_lines(node)) lines.append(f"end {kind} {node.name}") return "\n".join(lines) + def _interface_procedure_signature(self, node: FortranInterfaceProcedure, kind: str) -> str: + """Render the ordered parameter list and optional native binding.""" + suffix = f" result({node.result_name})" if node.result_name is not None else "" + binding = self._interface_binding_suffix(node) + return self._continued_call( + f"{kind} {node.name}(", + tuple(parameter.name for parameter in node.parameters), + suffix=f"){binding}{suffix}", + ) + + @staticmethod + def _interface_binding_suffix(node: FortranInterfaceProcedure) -> str: + """Spell one named, unnamed, or absent C binding clause.""" + if node.bind_name is not None: + return f' bind(c, name="{node.bind_name}")' + return " bind(c)" if node.bind_c else "" + + def _interface_import_lines(self, node: FortranInterfaceProcedure) -> tuple[str, ...]: + """Render the optional interface import declaration.""" + if not node.imports: + return () + return (self._indented(f"import :: {', '.join(node.imports)}"),) + + def _interface_result_lines(self, node: FortranInterfaceProcedure) -> tuple[str, ...]: + """Render a complete function result declaration when present.""" + if node.result_name is None or node.result_type is None: + return () + return (self._indented(f"{node.result_type} :: {node.result_name}"),) + def _function_signature(self, node: FortranFunction) -> str: """Render a Fortran function signature.""" suffix = f" result({node.result_name})" if node.result_name is not None else "" diff --git a/x2py/wrapper_codegen/support.py b/x2py/wrapper_codegen/support.py index d6c98334c..e04be2999 100644 --- a/x2py/wrapper_codegen/support.py +++ b/x2py/wrapper_codegen/support.py @@ -15,9 +15,11 @@ FunctionWrapperPolicy, ModuleObjectAccessMechanism, ModuleVariablePolicy, + ModuleGetterAction, NativeArrayDescriptorKind, NativeArrayHandleKind, NativeDescriptorHandoffABI, + OverloadPolicy, ) from x2py.wrapper_codegen.plan import WrapperPlanSupportBlocker, WrapperPlanSupportReport from x2py.wrapper_codegen.visitor import ClassVisitor @@ -70,7 +72,12 @@ def _visit_SemanticVariable(self, variable: models.SemanticVariable) -> WrapperP def _module_variable_support_report(self, policy: ModuleVariablePolicy) -> WrapperPlanSupportReport: """Classify one completed module-variable policy without backend inference.""" blockers = list(policy.blockers) - if policy.supported and policy.rank > 0 and policy.native_array_handle is None: + if ( + policy.supported + and policy.rank > 0 + and policy.native_array_handle is None + and policy.getter_action is not ModuleGetterAction.BORROWED_ARRAY_VIEW + ): blockers.append("rank-positive module array snapshots are not implemented by wrapper-plan lowering") return WrapperPlanSupportReport( owner_path=policy.owner_path, @@ -90,6 +97,8 @@ def _module_variable_lanes(policy: ModuleVariablePolicy, blockers: list[str]) -> ModuleObjectAccessMechanism.VALUE_COPY: "derived-module-constant-values", }[policy.derived.access] return (lane,) + if policy.getter_action is ModuleGetterAction.BORROWED_ARRAY_VIEW: + return ("borrowed-module-array-views",) if policy.native_array_handle is None: return ("scalar-module-variables",) return (f"{policy.native_array_handle.descriptor_kind.value}-module-handles",) @@ -111,12 +120,61 @@ def _visit_SemanticFunction(self, function: models.SemanticFunction) -> WrapperP ) def _module_blockers(self, module: models.SemanticModule) -> tuple[WrapperPlanSupportBlocker, ...]: - """Return blockers for module-level owners outside completed lanes.""" + """Return blockers for completed class and module-overload orchestration.""" blockers = [] blockers.extend(self._class_orchestration_blockers(module)) - blockers.extend(self._owner_blockers(module.name, "overload sets", module.overload_sets)) + blockers.extend(self._module_overload_blockers(module)) return tuple(blockers) + def _module_overload_blockers(self, module: models.SemanticModule) -> tuple[WrapperPlanSupportBlocker, ...]: + """Validate every module generic and its ordinary concrete call policy.""" + policies = module.metadata.get(models.RESOLVED_MODULE_OVERLOAD_POLICIES_METADATA) + if module.overload_sets and not isinstance(policies, tuple): + return (WrapperPlanSupportBlocker(module.name, "missing completed module-overload policies"),) + return ( + *self._completed_overload_policy_blockers(module, policies or ()), + *self._overload_candidate_blockers(module), + ) + + @staticmethod + def _completed_overload_policy_blockers( + module: models.SemanticModule, + policies: tuple, + ) -> tuple[WrapperPlanSupportBlocker, ...]: + """Return blockers already completed on module-overload policies.""" + blockers = [] + for policy in policies: + if not isinstance(policy, OverloadPolicy): + blockers.append(WrapperPlanSupportBlocker(module.name, "incomplete module-overload policy")) + continue + blockers.extend(WrapperPlanSupportBlocker(policy.owner_path, reason) for reason in policy.blockers) + return tuple(blockers) + + def _overload_candidate_blockers( + self, + module: models.SemanticModule, + ) -> tuple[WrapperPlanSupportBlocker, ...]: + """Collect completed call-policy blockers from concrete candidates.""" + return tuple( + blocker + for overload in module.overload_sets + for procedure in overload.procedures + for blocker in self._one_overload_candidate_blockers(module, overload.name, procedure) + ) + + @staticmethod + def _one_overload_candidate_blockers( + module: models.SemanticModule, + overload_name: str, + procedure: models.SemanticFunction, + ) -> tuple[WrapperPlanSupportBlocker, ...]: + """Require and project one concrete overload candidate call policy.""" + policy = procedure.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) + if not isinstance(policy, FunctionWrapperPolicy): + owner = f"{module.name}.{overload_name}.{procedure.name}" + return (WrapperPlanSupportBlocker(owner, "missing completed overload-candidate call policy"),) + return tuple(WrapperPlanSupportBlocker(policy.owner_path, reason) for reason in policy.blockers) + def _class_orchestration_blockers( self, module: models.SemanticModule, @@ -494,6 +552,7 @@ def _module_lanes( lanes.extend(self._derived_type_lanes(module)) lanes.extend(self._class_surface_lanes(module)) lanes.extend(self._class_function_lanes(module)) + lanes.extend(self._module_overload_lanes(module)) policies = ( *( function.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) @@ -514,6 +573,21 @@ def _module_lanes( lanes.append("python-namespaces") return tuple(lanes) + def _module_overload_lanes(self, module: models.SemanticModule) -> tuple[str, ...]: + """Reuse ordinary transfer lanes and add one orchestration lane for generics.""" + if not module.overload_sets: + return () + lanes = ["module-overloads"] + for overload in module.overload_sets: + for procedure in overload.procedures: + policy = procedure.metadata.get(models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA) + if not isinstance(policy, FunctionWrapperPolicy): + continue + for lane in self._function_lanes(policy): + if lane not in lanes: + lanes.append(lane) + return tuple(lanes) + def _class_surface_lanes(self, module: models.SemanticModule) -> tuple[str, ...]: """Return completed public class orchestration lanes.""" policies = tuple( From 3c964c0d7752c3e7b379c263c2a7c4f6e4d84620 Mon Sep 17 00:00:00 2001 From: said Date: Thu, 16 Jul 2026 13:59:59 +0100 Subject: [PATCH 13/30] port the tests to the new implementation and remove codegen/ folder, ir2ast.py and python_wrapper.py --- .pymon-journal | Bin 8720 -> 0 bytes docs/developer/source-map.md | 2 +- tests/wrapper/CHECKLIST_COVERAGE.md | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 .pymon-journal diff --git a/.pymon-journal b/.pymon-journal deleted file mode 100644 index 5158fd9b80a4960a0f3f89f72f6712df0e92d509..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 8720 zcmeHLO>Em#9Cx~QE!93iLW)2_dY7rK=!c!ONjf2Qp8MK{#ZG2B+X_O~tNqe?b?nS` zx@9M}se&69#APRr2x$knj05+bk&rmTc^7VQbm(R5GRF)%D6%vAnuoEP=9^ zMzM82B%KZotxn*%c5lLxb>Ic|abyLZ*bRK*SphJI!Ch*{)Ul#Cbp1n%I8I2T$Z7>) zJNnm@L~9Xk6F+wCXf5$?ZG!IAPOsJVZRXw9_dkEYg0}VS`OTN7NZ9)F;cvUo-WIlg zKmGcTU%&hI8dg7L^rzw*GTd^;&6*oi#JnakD#AZNxtwqfA>qZdHgYW77 zGrQ%^JWqaPI#zFgc=7U|!pz(Y;nMTVKhA$M_hR0e%$}`g2xJIk2xJIk2xJIk2>cTS zp1d26m8My(YnD>iOnk>Q##d$(9m^(0rhL7I(KsC~&Lf1dcFWpB zCnU6wNVr&9UqYSGZIiHvj%g41L5%#a=b^6e-s{q3CZRN%V6~~4$n|4-NW>fT3VRCB+9&l(zv-NM>-T_2JP8>J;q(`D5%>!w_@)axVKR9V@pYS_~1+o{G4 zsgQ^bs3Z1@uMlNjNp~&v4I2A~iB+qr?bOG;Y3!SOfcFTI*_>ZeX3pntOF6;y9r_@; z=fQ!~BHcLPzpZiLEopr1{O1?W&E~IM$$jxL53q?1(^9diR~7IneV!f(>RCz>PGn%i zV0n@54r02DR#uP!$9FhEBiEsD;BL7zWYHW?0M3m46E_U}Hn1XxMt11(;ow!_D#Ob# zW!<#Qs+tC4U=z1mFh#g06h$!!V-&k>ih>r3k0@dXeeMBQ(vZ#6<#llrLc*MGkoagI z0Jyf7;VAEU!F?D>P@@+gvHi3=0jQ2T)P`NzgpCP9y=ff9^-7{rBdHHe<&f99CF@lg z@_JAWg`rHnKN_J92|*l2qKDiLbB%xtpk_7IvtwAbx-vnHPCXCz4qU=~0H#sLS4X5B8L6BT1tA)YP24x7 zt~Tl#KC?lVzesD=Y8-150zw^R2LPzg0`IcT^|)+OhdjnDFCcLu5X8hwVhZSkK-l$t z78w@aA+${c>q&N`k~|L*8P`YMh=ve9VpDGG2G%uPd%egp4SnjJ5n^4bX~?@6rB>`t z7jy$8T&)VJ`YMM?9v~QNc}bu)Z8Oxt8z(_|r>^7bu7+;leYD6MET%m3OOK~=v-yRE+!G~j2Lr|tsj2aXFyQ0;eR4y1ZIgdZ zv(THGwas2CgAy6!-Zp_Ln-wEZP4iSL^eYh75xj2vL&y?`b8@}dQ1ohp lz17g&2x@p~bLxCvUHH4kIs$_wjr?S@{2Y6~J{2##`xi$SzmEU_ diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index d24fe6222..3e4ecd13f 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -70,7 +70,7 @@ X2PY_C_DOCS_END --> | `x2py/runtime/` | Python runtime objects consumed by generated extensions | `handles.py` | runtime handle and wrapper runtime tests | | `x2py/types/` | Semantic-to-Python ecosystem type mappings | `numpy.py` | `tests/types/test_numpy.py` | | `x2py/fortran_parser/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | -| `x2py/compiling/` | Native compile objects, compiler command orchestration, shared-library linking, and runtime support installation | `basic.py`, `compilers.py`, `python_wrapper.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | +| `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `basic.py`, `compilers.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | | `x2py/stdlib/` | Native runtime support files copied into generated wrapper builds | `x2py_runtime/` | wrapper runtime tests | | `x2py/utilities/` | Small shared Python utilities | `metaclasses.py`, `strings.py` | tests that exercise callers | diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index 54565310d..26600709b 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -103,7 +103,7 @@ recorded progression, not in the live ledger. | Default and keyword construction, explicit bound construction, exact constructor/method overloads, borrowed-child finalization, extension inheritance, and scalar polymorphic input run through the direct wrapper-plan class path | `derived_types/test_constructors_and_finalizers.py`, `derived_types/test_phase9_bound_constructors.py`, `naming/test_phase9_class_overloads.py`, `derived_types/test_borrowed_finalizers.py`, `derived_types/test_inheritance.py`, `derived_types/test_derived_type_methods.py` | | Reduced derived procedure boundaries replay passing legacy/source behavior through direct plans for required and optional wrapper inputs, exact-type rejection, in-place and caller-supplied output identity, ordinary, `sequence`, and `bind(C)` exact typed native value copies, direct and hidden owned results, checked allocation, conversion cleanup, and exactly-once owner finalization | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_use_canonical_plan`, `derived_types/test_phase8_derived_plan.py::test_value_copy_and_optional_derived_inputs_match_source_oracle`, `derived_types/test_scalar_derived_actual_dummy_matrix.py::test_sequence_derived_value_uses_the_same_typed_opaque_call_path`, `derived_types/test_phase8_derived_plan.py::test_borrowed_child_retains_owner_and_finalizes_exactly_once`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_mixed_derived_results_check_allocation_and_own_every_failure_path_before_scalar_conversion` | | Plain non-target module objects use live typed member proxies while `Aliased` objects use direct addresses; both retain the module, reject module replacement, and expose live scalar, fixed-string, ordinary-array, nested-derived, allocatable-handle, and pointer-handle fields with completed getter/setter and parent-owner behavior. Detached whole-object `Snapshot[T]` is removed | `derived_types/test_phase8_derived_plan.py::test_plain_module_derived_proxy_reads_and_writes_live_members`, `derived_types/test_phase8_derived_plan.py::test_aliased_module_derived_object_uses_direct_live_field_handles`, `derived_types/test_phase8_derived_plan.py::test_fixed_string_fields_use_canonical_plan`, `derived_types/test_phase8_derived_plan.py::test_pointer_field_descriptor_views_use_canonical_plan` | -| Every wrapper build uses completed policy and the canonical wrapper-plan generator; dependency isolation is structural, while unsupported derived shapes retain exact policy blockers | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_use_canonical_plan`, `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `derived_types/test_phase9_bound_constructors.py::test_bound_constructor_replaces_field_initialization_and_reuses_method_plan`, `tests/wrapper_codegen/test_phase0b_contracts.py::test_active_wrapper_build_modules_do_not_import_retired_codegen`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_unsupported_derived_shapes_fail_on_exact_completed_policy_blockers` | +| Every wrapper build uses completed policy and the canonical wrapper-plan generator; dependency isolation is structural, while unsupported derived shapes retain exact policy blockers | `derived_types/test_phase8_derived_plan.py::test_scalar_derived_objects_use_canonical_plan`, `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `derived_types/test_phase9_bound_constructors.py::test_bound_constructor_replaces_field_initialization_and_reuses_method_plan`, `tests/wrapper_codegen/test_phase0b_contracts.py::test_wrapper_build_pipeline_imports_canonical_generator`, `tests/wrapper_codegen/test_phase8_derived_types.py::test_unsupported_derived_shapes_fail_on_exact_completed_policy_blockers` | ## Build From Source From 2b0cca4272c4459ca66624823e067da66948797f Mon Sep 17 00:00:00 2001 From: said Date: Thu, 16 Jul 2026 14:06:00 +0100 Subject: [PATCH 14/30] github-actions From 182a82a3af3c96a95b5f18ba255e89c50cae3bc7 Mon Sep 17 00:00:00 2001 From: said Date: Thu, 16 Jul 2026 14:24:23 +0100 Subject: [PATCH 15/30] github-actions --- .github/workflows/quality.yml | 5 ++--- docs/developer/quality-assurance.md | 12 ++++++------ docs/developer/testing-strategy.md | 8 ++++---- tests/wrapper/fortran/real_libraries/README.md | 12 ++++++------ 4 files changed, 18 insertions(+), 19 deletions(-) diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index ce9d5b46a..1f2d07b66 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -186,8 +186,6 @@ jobs: fail-fast: false matrix: library: [blas, lapack] - env: - X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR: ${{ runner.temp }}/x2py-real-library-native steps: - name: Checkout repository uses: actions/checkout@v4 @@ -216,12 +214,13 @@ jobs: - name: Restore compiled native library cache uses: actions/cache@v4 with: - path: ${{ env.X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR }} + path: ${{ runner.temp }}/x2py-real-library-native key: real-library-${{ runner.os }}-gfortran13-${{ matrix.library }}-${{ hashFiles('tests/data/fortran/blas/**', 'tests/data/fortran/lapack/**') }} - name: Run full ${{ matrix.library }} wrapper test env: PYTHONPATH: . HYPOTHESIS_PROFILE: ci + X2PY_REAL_LIBRARY_NATIVE_CACHE_DIR: ${{ runner.temp }}/x2py-real-library-native run: >- python -m pytest -q --randomly-seed=1 "tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py::test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_library[${{ matrix.library }}]" diff --git a/docs/developer/quality-assurance.md b/docs/developer/quality-assurance.md index fa649931c..aa15032c1 100644 --- a/docs/developer/quality-assurance.md +++ b/docs/developer/quality-assurance.md @@ -228,12 +228,12 @@ reports advisory/manual. issues, Ruff formatting drift, Vulture unused test parameters, and the too-strict Radon policy. -**Native artifact cache:** the full BLAS/LAPACK native-cache preparation is -disabled with the deferred real-library wrapper test during wrapper-plan -migration. Restore the cache job and matrix environment only when Phase 12 of -the migration checklist explicitly re-enables both corpora. Requested coverage -runs still collect Python 3.12 coverage data; a final coverage job combines -that artifact and uploads the XML report. +**Native artifact cache:** dedicated Python 3.12 BLAS and LAPACK jobs restore a +separate runner-local native cache for each library before executing the full +wrapper test. The ordinary pytest matrix excludes that full corpus while +retaining the lighter native-bundle tests. Requested coverage runs still +collect Python 3.12 coverage data; a final coverage job combines that artifact +and uploads the XML report. **Failure reporting:** each pytest matrix invocation writes `pytest-results.xml`; the final failure-only step runs diff --git a/docs/developer/testing-strategy.md b/docs/developer/testing-strategy.md index 9bea6f0d8..a7b10005f 100644 --- a/docs/developer/testing-strategy.md +++ b/docs/developer/testing-strategy.md @@ -69,10 +69,10 @@ python3 -m pytest -q tests/wrapper/fortran \ --ignore=tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py ``` -Do not run the full BLAS or LAPACK real-library wrapper tests locally or in -GitHub Actions during migration. Re-enable both only after every other -wrapper-plan migration row is complete. General native-bundle tests remain -active. +The full LAPACK wrapper test remains CI-only by default. The full BLAS test may +be run locally when its library-scale evidence is needed, and dedicated +GitHub Actions jobs run both BLAS and LAPACK on Python 3.12. General +native-bundle tests remain active in the ordinary matrix. ## Fixtures and generated expectations diff --git a/tests/wrapper/fortran/real_libraries/README.md b/tests/wrapper/fortran/real_libraries/README.md index db52fff3f..4e6307a75 100644 --- a/tests/wrapper/fortran/real_libraries/README.md +++ b/tests/wrapper/fortran/real_libraries/README.md @@ -21,12 +21,12 @@ Fortran bridge compilation. These jobs validate large wrapper generation and import/runtime behavior; they are not runtime-performance benchmarks, so the fast compile override keeps CI focused on wrapper correctness. -GitHub Actions pins the real-library jobs to `ubuntu-24.04` with `gfortran-13`, -warms this cache in a pre-matrix job, and then restores it in each Python -matrix job. The key includes the runner OS, runner architecture, pinned -`gfortran` version, source content hash, and native cache helper code. Native -object files are reusable only for the same platform/compiler/source -combination; a different runner image, compiler, architecture, or BLAS/LAPACK +GitHub Actions runs separate BLAS and LAPACK jobs on `ubuntu-24.04`, Python +3.12, and `gfortran-13`. Each job restores its library-specific cache into the +runner temporary directory and rebuilds it on a cache miss. The key includes +the runner OS, pinned compiler profile, selected library, and BLAS/LAPACK source +content. Native object files are reusable only for the same +platform/compiler/source combination; a different runner image, compiler, or fixture content gets a separate rebuildable cache entry. Contract fixtures: full generated BLAS and LAPACK packages are compared against From 5889cf2464ad6e932f437da45061a24197e672fc Mon Sep 17 00:00:00 2001 From: said Date: Thu, 16 Jul 2026 17:39:53 +0100 Subject: [PATCH 16/30] fix errors and add pass by value --- .../wrapper-generation-pipeline.md | 34 +++++ .../wrapper-plan-migration-checklist.md | 28 ++-- docs/user/guide/wrapping-derived-types.md | 3 +- docs/user/reference/semantic-pyi-format.md | 76 ++++++++-- .../conversion/pyi/test_types_and_values.py | 20 +-- .../fixtures/general/basic_subroutine.json | 1 + .../general/compile_time_all_exprs.json | 27 ++-- .../general/compile_time_shape_exprs.json | 6 +- .../fixtures/general/derived_type.json | 1 + .../general/derived_types_and_methods.json | 2 + .../fixtures/general/modern_pyi_example.json | 10 +- .../fixtures/general/module_vars_use.json | 5 +- .../general/procedures_and_functions.json | 2 + .../scope_name_reuse_combinations.json | 10 +- .../fbind_c_derived_layout_f90.pyi | 5 +- ...scalar_derived_actual_dummy_matrix_f90.pyi | 12 +- .../derived_types/test_phase8_derived_plan.py | 5 +- ...test_scalar_derived_actual_dummy_matrix.py | 3 - .../contracts/lapack/__init__.pyi | 130 ++++++++--------- .../printers/test_types_and_declarations.py | 44 +++++- .../test_phase6b_dense_array_shapes.py | 10 ++ .../test_phase7_native_array_handles.py | 13 ++ .../test_phase8_derived_types.py | 5 +- ...ase8_scalar_derived_actual_dummy_matrix.py | 36 ++--- x2py/contracts/__init__.py | 10 +- x2py/semantics/fortran2ir.py | 15 +- x2py/semantics/ownership.py | 10 +- x2py/semantics/pyi2ir.py | 82 +++++++---- x2py/semantics/wrapper_policy.py | 55 ++------ x2py/wrapper_codegen/c/binding.py | 131 ++++++++++++------ x2py/wrapper_codegen/fortran/bridge.py | 69 --------- x2py/wrapper_codegen/plan.py | 2 - x2py/wrapper_codegen/planner.py | 1 - x2py/wrapper_codegen/printers/pyi_printer.py | 76 +++++----- 34 files changed, 545 insertions(+), 394 deletions(-) diff --git a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md index 114999d3c..36b013c5f 100644 --- a/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md +++ b/docs/maintainer/internal-architecture/wrapper-generation-pipeline.md @@ -26,6 +26,32 @@ then dispatch only from completed selectors into small named lowering methods; they do not reconstruct policy from datatype, `intent`, shape, alias flags, or local memory checks. +Native-source `intent` may be consumed while importing a source declaration to +propose default Python argument/result positions. It is not retained in the +semantic `.pyi` or post-IR ownership context. The editable Python signature, +`Returns[...]` projection, and ordered native-call mapping are authoritative. +Bridge entry dummies omit `intent`, leaving their storage permissive; that +contract controls wrapper copy-in, copy-back, and returned values, while the +compiled native procedure's own interface controls native access. + +Within that contract, an explicit native-call list is exhaustive for native +dummy positions. Matching named `Returns[...]` items attach result positions to +visible `Arg(i)` entries automatically; direct function results remain the first +ordinary Python return item, while hidden native output dummies require explicit +`Return(...)` entries. Descriptor reassociation follows the same rule: +`Pointer(Arg(i))` without a projected return uses a call-local adapter and +discards reassociation, while a matching projected return requires storage that +can preserve association writeback. + +Native transport overrides also live on that mapping. Primitive `Arg(i)` is a +value handoff and `Addr(Arg(i))` selects call-local address handoff. Wrapped +derived `Arg(i)` is a typed reference handoff and `Value(Arg(i))` selects exact +typed value handoff. `Returns[...]` never selects either ABI; it only assigns a +Python result position and writeback expectation. A derived `Value(...)` slot +does not expose aggregate layout at the C boundary: C still supplies an opaque +address, the bridge reconstructs the exact native type, and the Fortran compiler +applies the explicit interface's `VALUE` semantics at the typed call. + The public direct-generation boundary is: ```python @@ -149,6 +175,14 @@ and ordinary arrays use the same source kind with `CodegenAction.COPY_OUT`. data-buffer ABI. Its handoff plan carries data, rank, extents, strides, and itemsize. `PASS_NATIVE_DESCRIPTOR` is reserved for Phase 7 persistent native descriptors and handles. Neither backend may substitute one for the other. +Array handoff shapes are completed bridge extents; native source bounds are +temporary import facts and must not appear in semantic `.pyi` or become extent +dependencies. A source dimension such as `0:LDB-1` therefore completes to +extent `LDB`, while the native procedure keeps control of its own indexing +bounds. When `PASS_NATIVE_DESCRIPTOR` also carries +optional absence, the completed optional mode lowers a valid call-local +placeholder descriptor plus a separate presence role. This keeps the bridge +entry ABI valid while presence dispatch omits the native dummy. `DatatypeFamily` remains useful after object-kind dispatch for primitive element spelling and conversion, such as integer versus real scalar types or diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index a0ae2e09f..43588eada 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -3475,8 +3475,9 @@ itself; Phase 8F/H add the public field surface on this substrate. ### Phase 8G — Exact Native `value` Copies And Opaque Layout - [x] Preserve `bind(C)`/`sequence`/ordinary derived-type facts and native - `value` metadata through generated `Annotated[T, ByValue]`, post-IR policy, - and the derived handoff plan. + `value` transport through generated `Value(Arg(i))`, post-IR policy, and the + derived handoff plan. Do not store this per-call ABI choice on the annotated + Python type. - [x] For every supported exact rank-zero monomorphic native `value` argument, keep Python on the opaque wrapper contract. The Fortran bridge imports the exact native type, reads the typed pointee, and performs the typed call. The @@ -3558,13 +3559,13 @@ forms are: | `P` | `type(item), pointer :: arg` | | `V` | `type(item), value :: arg` | -`OPTIONAL`, rank, qualified type identity, and `INTENT` remain separate facts. -For the `P` column, a nonpointer actual is legal only for an explicitly -`INTENT(IN)` pointer dummy. An absent `INTENT` is reassociable, not read-only. -If x2py lacks the intent but the compiler has the authoritative imported module -interface, select a compiler-validated target adapter; if neither has an -authoritative interface, report an interface error rather than fabricating a -pointer actual. +`OPTIONAL`, rank, and qualified type identity remain separate facts. Source +`INTENT` may propose the initial Python projection, but it is not a completed +matrix selector. For the `P` column, `Pointer(Arg(i))` without a matching +projected return selects a call-local pointer input adapter and discards native +reassociation. A matching `Returns[...]` selects association writeback and +therefore requires persistent pointer storage. x2py never selects between these +paths from native `INTENT`. Use these completed action names. Parenthesized state requirements are runtime preconditions, not alternative fallback actions: @@ -3580,7 +3581,7 @@ preconditions, not alternative fallback actions: | `POINTEE_REFERENCE` | pass the current target of a pointer holder or module pointer to a nonpointer dummy | | `POINTER_HOLDER` | pass a persistent wrapper-owned pointer holder component directly so association writeback updates the same holder | | `MODULE_POINTER_TRANSACTION` | initialize one bridge-local typed pointer holder from the current target and restore its final association through an interoperable holder-address operation | -| `POINTER_INPUT_ADAPTER` | expose a nonpointer actual as a target only for a pointer dummy proved or compiler-validated as `INTENT(IN)` | +| `POINTER_INPUT_ADAPTER` | expose a payload through a call-local pointer carrier because the Python contract does not project pointer association writeback | | `TYPED_VALUE_COPY` | the exact Fortran bridge passes the typed object into the native `VALUE` slot; C never copies aggregate bytes | | `INCOMPATIBLE` | language-level storage mismatch; raise the specified `TypeError` and never enter native code | @@ -3598,7 +3599,7 @@ those preconditions. | `type(item), allocatable :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | | `type(item), allocatable :: var` | module | `SCOPED_REFERENCE [allocated]` | `SCOPED_REFERENCE [allocated]` with call-scoped target | `MODULE_ALLOCATABLE_TRANSACTION` | `MODULE_ALLOCATABLE_TRANSACTION` with call target | scoped `POINTER_INPUT_ADAPTER [allocated]` | scoped `TYPED_VALUE_COPY [allocated]` | | `type(item), allocatable, target :: var` | non-module holder | `HOLDER_REFERENCE [allocated]` | `HOLDER_REFERENCE [allocated]` with holder target lifetime | `ALLOCATABLE_HOLDER` | `ALLOCATABLE_HOLDER` | holder `POINTER_INPUT_ADAPTER [allocated]` | holder `TYPED_VALUE_COPY [allocated]` | -| `type(item), allocatable, target :: var` | module | `MODULE_ADDRESS [allocated]` | `MODULE_ADDRESS [allocated]` with module target lifetime | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `POINTER_INPUT_ADAPTER [allocated]` | module-address `TYPED_VALUE_COPY [allocated]` | +| `type(item), allocatable, target :: var` | module | `MODULE_ADDRESS [allocated]` | `MODULE_ADDRESS` with module target lifetime | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `MODULE_ALLOCATABLE_TRANSACTION` preserving target | `POINTER_INPUT_ADAPTER [allocated]` | module-address `TYPED_VALUE_COPY [allocated]` | | `type(item), pointer :: var` | non-module holder | `POINTEE_REFERENCE [associated]` | `POINTEE_REFERENCE [associated]` with retained target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `POINTER_HOLDER` | pointee `TYPED_VALUE_COPY [associated]` | | `type(item), pointer :: var` | module | module `POINTEE_REFERENCE [associated]` | module `POINTEE_REFERENCE [associated]` with native target owner | `INCOMPATIBLE` | `INCOMPATIBLE` | `MODULE_POINTER_TRANSACTION` | module-pointee `TYPED_VALUE_COPY [associated]` | @@ -3612,6 +3613,11 @@ completed action or one deliberate language-level error before lowering. No backend may infer a different action from datatype, `intent`, module shape, address presence, or local memory checks. +The table's `P` entries show the non-projecting input-adapter form. When the +Python contract projects pointer association writeback, replace every +nonpointer `P` cell with `INCOMPATIBLE`; the two pointer-storage rows retain +`POINTER_HOLDER` and `MODULE_POINTER_TRANSACTION`. + ##### Shared Holder And Callback ABI Define these support types once per qualified native derived type and import diff --git a/docs/user/guide/wrapping-derived-types.md b/docs/user/guide/wrapping-derived-types.md index da140d523..288b9e277 100644 --- a/docs/user/guide/wrapping-derived-types.md +++ b/docs/user/guide/wrapping-derived-types.md @@ -99,7 +99,8 @@ same mutation in place and returns only the other declared results (or `None`). The complete example shows inout mutation and a wrapper-owned function result. A native by-value argument is preserved in generated semantic contracts as -`Annotated[point, ByValue]`. Python still passes an existing `point` wrapper. +`@native_call([Value(Arg(0)), ...])`. Python still passes an existing `point` +wrapper. The generated Fortran bridge imports the exact native type and performs the typed by-value call. The foreign boundary never lays out or byte-copies the aggregate, diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index 8d79cdf8e..3cc82b956 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -917,6 +917,14 @@ is an array whose element length is not fixed in the public contract. The native boundary is always the array data address; `@native_call([Addr(Arg(i))])` is redundant for these arguments. +Array dimensions in the public type are bridge extents, not native lower and +upper bounds. For example, a native dimension `0:LDB-1` has public extent +`LDB`, so an assumed-size contract is written `T[LDB, Flat]`. The bridge may +construct its local view with any lower bound while passing the same base +address and extent; the compiled native procedure applies its own declared +bounds. Native lower bounds, upper bounds, and source-dimension spellings are +not part of the semantic `.pyi` format. + `String[n]` represents a Python `str` at the Python boundary. Its encoded byte length must be exactly `n`; x2py does not pad or truncate the public value. x2py converts it to call-local fixed-width character storage and passes that storage @@ -1065,12 +1073,12 @@ Use local constants or generated `Final[...]` names for shape symbols. ## Metadata With `Annotated` `Annotated[...]` carries storage metadata and semantic constraints. It does -not normally carry source-language argument direction. The one native-call -exception is `ByValue` on a wrapped derived-type argument: it records that the -exact native dummy receives the derived object by value rather than by -reference. The Python API still accepts the same opaque wrapper object; the -generated Fortran bridge performs the typed call, and the binding never -exposes or guesses aggregate layout. +not carry source-language argument direction or per-call value/reference +selection. Native call transport belongs to `@native_call`: a wrapped derived +object uses its normal reference handoff with `Arg(i)` and exact typed value +handoff with `Value(Arg(i))`. The Python API accepts the same opaque wrapper +object in both cases; the generated Fortran bridge performs the typed call, and +the binding never exposes or guesses aggregate layout. ```python from x2py.contracts import Annotated, COPY_F, Float64, ORDER_C, ORDER_F @@ -1091,7 +1099,6 @@ Generated canonical metadata: | `PointerAssociation("runtime")` | pointer association is a runtime state rather than a declaration-time constant | | `Name("native-name")` | source name cannot be represented directly as the Python target name | | `Aliased` | native storage may be exposed across the Python boundary as an alias | -| `ByValue` | a wrapped exact rank-zero monomorphic derived-type argument is passed to a native value dummy by the typed Fortran bridge; the binding never lays out the aggregate, so ordinary, `sequence`, and `bind(C)` types share this path | | `Immutable` | Python-visible value must not be mutated in place; writable native calls require a completed copy-in/copy-out replacement policy or an explicit call-local discarded-mutation policy | | `Ownership("python" | "native" | "wrapper" | "caller" | "temporary" | "unknown")` | explicit owner override for the wrapper ownership policy | | `Transfer("copy_return" | "snapshot_copy" | "borrowed_view" | "call_local" | "in_place" | "by_value" | "wrapper_instance" | "blocked")` | explicit boundary transfer override for the wrapper ownership policy | @@ -1108,8 +1115,6 @@ Loaded compatibility metadata: | --- | --- | | `Contiguous` | source provenance says the array is contiguous | | `ArrayCategory("...")` | source array category provenance | -| `SourceDims(...)` | source declaration dimensions | -| `LowerBounds(...)`, `UpperBounds(...)` | source bound provenance | | `FortranAllocatable` | older scalar character allocatable metadata; generated contracts use `Allocatable[String]` | diff --git a/docs/developer/quality-assurance.md b/docs/developer/quality-assurance.md index aa15032c1..b5c3eb8da 100644 --- a/docs/developer/quality-assurance.md +++ b/docs/developer/quality-assurance.md @@ -70,6 +70,9 @@ deciding a fix. A plain local coverage run can miss subprocess data. GitHub Actions runs ordinary PR tests without coverage overhead. During the wrapper-plan migration, every Python version excludes the full BLAS/LAPACK real-library wrapper test while retaining general native-bundle coverage. +One separate Python 3.12 job runs the full BLAS and LAPACK nodes together. A +pull request may use the `ignore-real-library-wrappers` label to skip that +expensive job without disabling the ordinary Python-version matrix. Pushes to `main` always run the remaining Python 3.12 test job under coverage and publish the coverage report. Add the `run-coverage` PR label, or pass `coverage: true` to the reusable workflow, to request the same coverage gate diff --git a/docs/developer/testing-strategy.md b/docs/developer/testing-strategy.md index a7b10005f..55e35e43d 100644 --- a/docs/developer/testing-strategy.md +++ b/docs/developer/testing-strategy.md @@ -70,9 +70,12 @@ python3 -m pytest -q tests/wrapper/fortran \ ``` The full LAPACK wrapper test remains CI-only by default. The full BLAS test may -be run locally when its library-scale evidence is needed, and dedicated -GitHub Actions jobs run both BLAS and LAPACK on Python 3.12. General -native-bundle tests remain active in the ordinary matrix. +be run locally when its library-scale evidence is needed. One dedicated GitHub +Actions job runs the exact full BLAS and LAPACK nodes together on Python 3.12; +the ordinary Python-version matrix excludes their complete test file. Add the +`ignore-real-library-wrappers` label to a pull request to skip only this +expensive dedicated job. General native-bundle tests remain active in the +ordinary matrix. ## Fixtures and generated expectations diff --git a/docs/maintainer/design/wrapper-design-notes.md b/docs/maintainer/design/wrapper-design-notes.md index c9f8b0ccc..24bc0ccd5 100644 --- a/docs/maintainer/design/wrapper-design-notes.md +++ b/docs/maintainer/design/wrapper-design-notes.md @@ -40,7 +40,7 @@ X2PY_C_DOCS_END --> Merge or move concepts only when their invariants match: @@ -220,8 +220,8 @@ X2PY_C_DOCS_END --> policy-completion ownership decisions, transfer actions, mutability, setter exposure, and release responsibility. - [ ] `docs/maintainer/internal-architecture/ast-design.md`: document parser AST, semantic - IR, codegen AST, what each layer may store, and what must not leak across + IR, completed wrapper plans, generated source syntax, what each layer may + store, and what must not leak across layers. - [ ] `docs/maintainer/internal-architecture/semantic-passes.md`: document semantic pass ordering, completed policy decisions, readiness checks, and handoff to @@ -165,7 +166,7 @@ X2PY_C_DOCS_END --> names. - unresolved typedef or unknown type references; - legacy parser reports carrying macro-dependent declarations; - variadic functions; -- function pointer/callback signatures without edited `.pyi` `Callable` +- function pointer/callback signatures without a resolved named prototype policy; - mutable numeric or `void *` pointer parameters without ownership, scalar-storage, raw-address, or array policy; @@ -439,11 +439,12 @@ storage, NumPy array storage, Python strings, raw address values, and generated wrapper instances. The native barrier distinguishes direct values, call-local addresses, caller/Python-backed storage addresses, raw addresses, packed array descriptors, and wrapper-owned native addresses. These decisions are semantic -policy. `ir2ast.py`, bindings, bridges, and printers may create backend-local -temporaries, but they must not infer or override a barrier action from datatype, -source-declaration direction, array category, aliasing, or memory-storage checks. +policy. Wrapper planning, binding/bridge lowering, and printers may create +backend-local temporaries, but they must not infer or override a barrier action +from datatype, source-declaration direction, array category, aliasing, or +memory-storage checks. -Parser-model conversion and codegen model traversal use the shared +Parser-model conversion and semantic/wrapper-model traversal use the shared `x2py.utilities.visitor.ClassVisitor` dispatcher and one configured `_` protocol. The default prefix is `_visit`; specialized visitors may choose clearer names such as `_print` or `_parse` while still using diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index 3cc82b956..f4d181de9 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -156,7 +156,7 @@ or an explicit call-local copy whose native mutation is discarded. A replacement requires a projected return such as `Returns["values", Float64[:]]`; the bridge and binding then emit the already-selected action without reconsidering the datatype, mutability, ownership, or storage mode. Unsupported combinations block -before `ir2ast.py`. +before wrapper planning and direct lowering. `@native_call(...)` and `Returns[...]` describe projection and native placement; they do not ask the backend to rediscover conversion policy. After `.pyi` @@ -342,11 +342,12 @@ from x2py.contracts import Float64, Int32, external def dgesv(a: Float64[:, :], b: Float64[:, :]) -> Int32: ... ``` -`@external` is immutable native-placement metadata. The bridge must generate a -matching explicit Fortran interface and call the external procedure without a -`use ` statement. The procedure therefore needs no Fortran `.mod` file, -but its defining object, archive, or shared library must be supplied to the -link. +`@external` is immutable native-placement metadata. The bridge calls the +external procedure without a `use ` statement. Classic +implicit-interface-compatible procedures use a compact `external` declaration; +features that require an explicit interface retain one. The procedure needs no +Fortran `.mod` file, but its defining object, archive, or shared library must be +supplied to the link. Python-visible renaming is separate from placement. `@bind` retains the native Fortran procedure name while the declaration uses a wrapper name: @@ -485,7 +486,8 @@ def DAXPY( not a request to collapse the whole array to rank one: `Float64[:, Flat]` remains a rank-two Python and bridge contract. Because `real :: a(:, *)` is not a legal Fortran assumed-size declaration, an external interface whose prefix -extent is known only at runtime uses the sequence-associated `a(*)` spelling; +extent is known only at runtime uses the sequence-associated `a(*)` spelling +when that procedure requires an explicit interface; the bridge view still has rank two and receives both runtime extents. The Python-visible flat dimension remains unconstrained. @@ -573,7 +575,8 @@ files while the generated bridge is compiled: Archives do not normally contain `.mod` files, so module directories remain separate inputs. Standalone `@external` procedures require no `.mod` file because -the bridge emits their interface from the semantic contract. +the semantic contract supplies either their implicit external declaration or +their required explicit interface. Required link cases are: @@ -586,7 +589,7 @@ Required link cases are: | Vendor shared implementation | direct `.so` path or `--native-library NAME` plus search directory | | Mixed implementation | objects, archives, direct shared libraries, and named libraries in one ordered plan | | Module procedures | native artifacts plus every required `.mod` search directory | -| Standalone procedures | native artifacts only; interfaces come from `@external` declarations | +| Standalone procedures | native artifacts only; declaration mode comes from the completed `@external` contract | Static link order is semantically significant: dependent objects precede the archives or libraries that satisfy them, and dependent libraries precede their @@ -750,7 +753,7 @@ extension binding surface. Declarations in imported leaf files that are not reachable from that policy do not get standalone public wrapper bindings; they remain native contract facts only when an exported declaration depends on them. -Absolute support imports such as `from x2py.contracts import Callable` or +Absolute support imports such as `from x2py.contracts import prototype` or `from types import SimpleNamespace` may support annotation parsing, but they are not contract graph edges and never create runtime exports. Generated references to declarations in another contract package file use relative imports. @@ -775,55 +778,60 @@ X2PY_C_DOCS_END --> | Complex | `Complex64`, `Complex128`, `Complex256` | | Text | `String` | | User types | class names and imported type names | -| Callables | `Callable`, `Callable[..., T]`, `Callable[[A, B], T]` | -| Callback argument interface wrappers | `PassByRef(T)`, `In(T)`, `Out(T)`, `InOut(T)` inside `Callable[[...], T]` only | +| Named callable prototypes | `@prototype` function declarations referenced by name | +| Prototype value override | `Value(T)` inside a `@prototype` declaration only | - -Projection/runtime roadmap: - -1. Lower `@native_call` mappings into executable wrapper calls. -2. Add validation and coercion contracts for dtype, rank, shape, order, - strides, alignment, mutability and aliasing. -3. Add ownership and lifetime contracts for opaque handles, pointer returns, - allocatable/pointer reassociation, callbacks and work buffers. -4. Decide how to emit clean IDE/type-checker stubs from semantic `.pyi` files - without losing the native wrapper contract. diff --git a/tests/_shared/pyi_conversion_support.py b/tests/_shared/pyi_conversion_support.py index 3f858a661..2b4e0d06a 100644 --- a/tests/_shared/pyi_conversion_support.py +++ b/tests/_shared/pyi_conversion_support.py @@ -34,7 +34,6 @@ ) from x2py.semantics.models import ( - CALLBACK_DECLARATION_ACCESS_METADATA, ProjectionMapping, PYTHON_VALUE_IMMUTABLE, PYTHON_VALUE_MUTABILITY_METADATA, @@ -127,7 +126,6 @@ def _semantic_modules_for_source(path: Path): "ADDRESS_ROLE_PROJECTION", "ADDRESS_ROLE_RAW", "BIND_TARGET_METADATA", - "CALLBACK_DECLARATION_ACCESS_METADATA", "CONTRACT_IMPORT", "CONTRACT_SYMBOLS", "FORTRAN_PYI_COMPARE_FIXTURES", diff --git a/tests/architecture/test_test_suite_layout.py b/tests/architecture/test_test_suite_layout.py index bfed2b928..637d1b755 100644 --- a/tests/architecture/test_test_suite_layout.py +++ b/tests/architecture/test_test_suite_layout.py @@ -10,6 +10,8 @@ TEST_ROOT = REPO_ROOT / "tests" TEST_INDEX = TEST_ROOT / "README.md" MIGRATION_CHECKLIST = REPO_ROOT / "docs/maintainer/roadmap/test-suite-organization-checklist.md" +QUALITY_WORKFLOW = REPO_ROOT / ".github/workflows/quality.yml" +FULL_REAL_LIBRARY_TEST = "tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py" STAGE_DIRECTORIES = { "architecture", @@ -158,3 +160,21 @@ def test_maintained_docs_do_not_name_deprecated_pytest_locations() -> None: if pattern in text: stale.append(f"{path.relative_to(REPO_ROOT)}: {pattern}") assert stale == [] + + +def test_full_real_library_nodes_have_one_dedicated_quality_job() -> None: + text = QUALITY_WORKFLOW.read_text(encoding="utf-8") + ordinary_jobs, dedicated_and_later = text.split(" real-library-wrappers:", maxsplit=1) + dedicated_job, _later_jobs = dedicated_and_later.split("\n coverage-report:", maxsplit=1) + + assert f"--ignore={FULL_REAL_LIBRARY_TEST}" in ordinary_jobs + assert ( + f'"{FULL_REAL_LIBRARY_TEST}::test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_library[blas]"' + in dedicated_job + ) + assert ( + f'"{FULL_REAL_LIBRARY_TEST}::test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_library[lapack]"' + in dedicated_job + ) + assert "ignore-real-library-wrappers" in dedicated_job + assert "matrix.library" not in dedicated_job diff --git a/tests/cli/test_readiness_reports.py b/tests/cli/test_readiness_reports.py index d4c6ecd85..d8c0894b6 100644 --- a/tests/cli/test_readiness_reports.py +++ b/tests/cli/test_readiness_reports.py @@ -116,7 +116,7 @@ def test_x2py_readiness_formatting_and_compiler_without_requirements(): "callback_signature_incomplete", {"owner": "handler", "needs": ["arguments", "return type"]}, ) - == "handler needs Callable[[...], ...] metadata (arguments, return type)" + == "handler needs a complete named @prototype (arguments, return type)" ) assert x2py_cli._format_semantic_blocker_item("c_unknown_type", {"owner": "api", "type": "widget"}) == "api: widget" assert x2py_cli._format_semantic_blocker_item("c_unknown_type", {"type": "widget"}) == ": widget" @@ -707,7 +707,7 @@ def test_x2py_format_semantic_readiness_reports_wrappable_and_blocked_sources(): - unresolved_semantic_types: unresolved external type * api_mod.solve uses unresolved type external_t - callback_signature_incomplete: callback metadata incomplete - * api_mod.apply needs Callable[[...], ...] metadata (arguments) + * api_mod.apply needs a complete named @prototype (arguments) File: interface.pyi Source: pyi @@ -738,7 +738,7 @@ def test_x2py_format_semantic_readiness_reports_wrappable_and_blocked_sources(): assert " Why not wrappable:" in text assert " - unresolved_semantic_types: unresolved external type" in text assert " * api_mod.solve uses unresolved type external_t" in text - assert " * api_mod.apply needs Callable[[...], ...] metadata (arguments)" in text + assert " * api_mod.apply needs a complete named @prototype (arguments)" in text assert "File: interface.pyi" in text assert " Source: pyi" in text assert " Semantic modules: " in text diff --git a/tests/cli/test_wrap_readiness.py b/tests/cli/test_wrap_readiness.py index c6cd07700..5d5022eef 100644 --- a/tests/cli/test_wrap_readiness.py +++ b/tests/cli/test_wrap_readiness.py @@ -145,7 +145,7 @@ def test_x2py_main_semantic_readiness_blocker_formatting(): assert "step uses unresolved type sim_state" in text assert "fill shape 'n' uses unresolved symbol n" in text assert "fill needs literal value for Final constant n" in text - assert "integrate.objective needs Callable[[...], ...] metadata (callback argument types)" in text + assert "integrate.objective needs a complete named @prototype (callback argument types)" in text assert "empty needs public functions" in text assert "{'payload': 1}" in text diff --git a/tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py b/tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py index 3c4c61786..5210afd3c 100644 --- a/tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py +++ b/tests/semantics/conversion/fortran/test_fortran_conversion_procedures_and_interfaces.py @@ -631,7 +631,7 @@ def test_dummy_procedure_interfaces_become_complete_callable_contracts(): module = FortranToIRConverter().visit(parse_fortran_source(source).modules[0]) abstract_callback = get_function(module, "abstract_case").arguments[0].semantic_type - assert abstract_callback.name == "Callable" + assert abstract_callback.name == "transform_iface" assert [argument.name for argument in abstract_callback.metadata["callback_arguments"]] == [ "count", "values", @@ -648,13 +648,12 @@ def test_dummy_procedure_interfaces_become_complete_callable_contracts(): assert abstract_callback.metadata["callback_lifetime"] == "call" assert abstract_callback.metadata["callback_thread"] == "entering_thread" assert abstract_callback.metadata["callback_exception"] == "print_traceback_and_abort" - assert [ - argument.metadata[semantic_models.CALLBACK_DECLARATION_ACCESS_METADATA] - for argument in abstract_callback.metadata["callback_arguments"] - ] == ["read", "read", "read"] + assert all( + argument.semantic_type.storage is not None for argument in abstract_callback.metadata["callback_arguments"] + ) explicit_callback = get_function(module, "explicit_case").arguments[0].semantic_type - assert explicit_callback.name == "Callable" + assert explicit_callback.name == "callback" assert [argument.name for argument in explicit_callback.metadata["arguments"]] == ["Int32"] assert explicit_callback.metadata["return"].name == "Int32" @@ -662,35 +661,28 @@ def test_dummy_procedure_interfaces_become_complete_callable_contracts(): assert notify_callback.metadata["return"].name == "None" no_intent_callback = get_function(module, "no_intent_case").arguments[0].semantic_type - assert [ - argument.metadata[semantic_models.CALLBACK_DECLARATION_ACCESS_METADATA] - for argument in no_intent_callback.metadata["callback_arguments"] - ] == ["unspecified", "unspecified"] + assert all(argument.semantic_type.storage.mutable for argument in no_intent_callback.metadata["callback_arguments"]) value_callback = get_function(module, "value_case").arguments[0].semantic_type assert [argument.name for argument in value_callback.metadata["callback_arguments"]] == ["value", "ref"] - assert [ - argument.metadata[semantic_models.CALLBACK_DECLARATION_ACCESS_METADATA] - for argument in value_callback.metadata["callback_arguments"] - ] == ["read", "unspecified"] assert [argument.origin.metadata["value"] for argument in value_callback.metadata["callback_arguments"]] == [ True, False, ] string_callback = get_function(module, "string_case").arguments[0].semantic_type - assert [ - argument.metadata[semantic_models.CALLBACK_DECLARATION_ACCESS_METADATA] + assert all( + argument.semantic_type.storage.array.category == "scalar_storage" for argument in string_callback.metadata["callback_arguments"] - ] == ["read", "write", "readwrite"] + ) emitted = emit_module(module) - assert "FortranCallback" not in emitted - assert "Callable[[" in emitted - assert "Callable[[In(Int32), In(Float64[count]), In(point_t)], Float64[count]]" in emitted - assert "Callable[[PassByRef(Int32), Float64[count]], None]" in emitted - assert "Callable[[Int32, PassByRef(Float64)], None]" in emitted - assert "Callable[[In(String[8]), Out(String[8][()]), InOut(String[8][()])], None]" in emitted + assert "@prototype\ndef transform_iface(" in emitted + assert "callback: transform_iface" in emitted + assert "@prototype\ndef value_iface(" in emitted + assert "value: Value(Int32)" in emitted + assert "@prototype\ndef string_iface(" in emitted + assert "read_label: String[8]" in emitted assert native_contract_issues(parse_pyi_text(emitted, module_name=module.name)) == [] project = parse_fortran_project( @@ -719,7 +711,7 @@ def test_dummy_procedure_interfaces_become_complete_callable_contracts(): ) modules = {item.name: item for item in FortranToIRConverter().visit(project)} imported_callback = get_function(modules["callback_user"], "apply").arguments[0].semantic_type - assert imported_callback.name == "Callable" + assert imported_callback.name == "renamed" assert [argument.name for argument in imported_callback.metadata["arguments"]] == ["Int32"] assert imported_callback.metadata["return"].name == "Int32" @@ -736,6 +728,6 @@ def test_dummy_procedure_interfaces_become_complete_callable_contracts(): ) standalone_module = FortranToIRConverter().visit(standalone)[0] standalone_callback = get_function(standalone_module, "standalone_case").arguments[0].semantic_type - assert standalone_callback.name == "Callable" + assert standalone_callback.name == "callback" assert [argument.name for argument in standalone_callback.metadata["arguments"]] == ["Int32"] assert standalone_callback.metadata["return"].name == "Int32" diff --git a/tests/semantics/conversion/pyi/test_calls_and_projections.py b/tests/semantics/conversion/pyi/test_calls_and_projections.py index 50fb3766b..9f941d508 100644 --- a/tests/semantics/conversion/pyi/test_calls_and_projections.py +++ b/tests/semantics/conversion/pyi/test_calls_and_projections.py @@ -25,26 +25,29 @@ ) -def test_convert_pyi_to_ir_preserves_callable_signature_metadata(): +def test_convert_pyi_to_ir_preserves_named_prototype_signature_metadata(): module = parse_pyi_text( """ -from x2py.contracts import Callable, Float64, Int32 +from x2py.contracts import Float64, Int32, prototype class sim_state: n: Int32 +@prototype +def objective(state: sim_state, value: Float64) -> Float64: ... + def integrate( state: sim_state, - objective: Callable[[sim_state, Float64], Float64] + callback: objective ) -> Float64: ... """, module_name="callbacks", ) callback_type = module.functions[0].arguments[1].semantic_type - assert callback_type.name == "Callable" - assert callback_type.dtype == "Callable" - assert [arg.name for arg in callback_type.metadata["arguments"]] == ["sim_state", "Float64"] + assert callback_type.name == "objective" + assert callback_type.dtype == "Prototype" + assert [arg.name for arg in callback_type.metadata["callback_arguments"]] == ["state", "value"] assert callback_type.metadata["return"].name == "Float64" @@ -915,11 +918,11 @@ def convert(value: F64 | None) -> F64 | None: ... [ ( "def consume(value: Allocatable[Float64]) -> None: ...\n", - "Callable scalar descriptors use nullable value annotations", + "Procedure scalar descriptors use nullable value annotations", ), ( "def produce() -> Pointer[Float64]: ...\n", - "Callable scalar descriptor results use a nullable value annotation", + "Procedure scalar descriptor results use a nullable value annotation", ), ( "@native_call([Allocatable(Arg(0))])\ndef consume(value: Float64) -> None: ...\n", @@ -940,12 +943,9 @@ def test_convert_pyi_to_ir_rejects_legacy_or_incomplete_scalar_descriptor_callab parse_pyi_text(source, module_name="invalid_descriptor_projection") -def test_convert_pyi_to_ir_handles_callable_and_pointer_storage_variants(): +def test_convert_pyi_to_ir_handles_pointer_and_array_storage_variants(): module = parse_pyi_text( """ -plain_callback: Callable -second_callback: Callable -opaque_callback: Callable[..., Float64] constant: Int32 deep: Addr[3](Float64) rank_any: Float64[...] @@ -957,16 +957,7 @@ def test_convert_pyi_to_ir_handles_callable_and_pointer_storage_variants(): module_name="storage", ) - plain, qualified, callback, constant, deep, rank_any, strided, computed, bounded, nested = [ - var.semantic_type for var in module.variables - ] - assert plain.name == "Callable" - assert plain.dtype == "Callable" - assert qualified.name == "Callable" - assert qualified.dtype == "Callable" - assert callback.metadata["arguments"] is None - assert callback.dtype == "Callable" - assert callback.metadata["return"].name == "Float64" + constant, deep, rank_any, strided, computed, bounded, nested = [var.semantic_type for var in module.variables] assert constant.storage is None assert deep.storage.kind == "pointer" assert deep.storage.pointer_depth == 3 @@ -986,7 +977,7 @@ def test_convert_pyi_to_ir_handles_callable_and_pointer_storage_variants(): assert nested.constraints == [SemanticConstraint("Constant")] -def test_convert_pyi_to_ir_preserves_module_fields_and_private_callable_arguments(): +def test_convert_pyi_to_ir_preserves_module_fields_and_private_function_arguments(): module = parse_pyi_text( """ output: Float64[:] = ... diff --git a/tests/semantics/conversion/pyi/test_types_and_values.py b/tests/semantics/conversion/pyi/test_types_and_values.py index 09595e287..8e90d4c8a 100644 --- a/tests/semantics/conversion/pyi/test_types_and_values.py +++ b/tests/semantics/conversion/pyi/test_types_and_values.py @@ -3,7 +3,6 @@ from tests._shared.pyi_conversion_support import ( ADDRESS_ROLE_METADATA, ADDRESS_ROLE_RAW, - CALLBACK_DECLARATION_ACCESS_METADATA, NATIVE_ARRAY_DESCRIPTOR_METADATA, OPTIONAL_ABSENT_HANDLE_METADATA, PYTHON_VALUE_IMMUTABLE, @@ -35,21 +34,19 @@ def test_convert_pyi_to_ir_dispatches_nested_and_qualified_semantic_types(): """ public_value: Int32 bounded: Final[Annotated[Int32, Bounded(1, 8)]] -callback: Callable pointer: Addr(Float64) raw_pointer: Addr(Float64) """, module_name="dispatch", ) - public_value, bounded, callback, pointer, raw_pointer = module.variables + public_value, bounded, pointer, raw_pointer = module.variables assert isinstance(public_value, SemanticVariable) assert public_value.visibility == "public" assert bounded.semantic_type.constraints == [ SemanticConstraint("Bounded", [1, 8]), SemanticConstraint("Constant"), ] - assert callback.semantic_type.name == "Callable" assert pointer.semantic_type.storage.kind == "address" assert pointer.semantic_type.storage.metadata[ADDRESS_ROLE_METADATA] == ADDRESS_ROLE_RAW assert raw_pointer.semantic_type.storage.read_only is False @@ -206,25 +203,25 @@ def set_status( assert parse_pyi_text(emitted, module_name="status_api") == module -def test_convert_pyi_to_ir_preserves_callback_argument_abi_wrappers(): +def test_convert_pyi_to_ir_uses_reference_default_and_explicit_value_callbacks(): module = parse_pyi_text( """ class particle: mass: Float64 +@prototype +def callback_shape( + value: Value(Int32), + values: Float64[:], + scalar_storage: Float64[()], + scalar: Float64, + count: Int32, + output: Float64[:], + result_storage: Float64[()], +) -> None: ... + def register( - callback: Callable[ - [ - Int32, - Float64[:], - Float64[()], - PassByRef(Float64), - In(Int32), - Out(Float64[:]), - InOut(Float64[()]), - ], - None, - ] + callback: callback_shape ) -> None: ... """, module_name="callbacks", @@ -233,15 +230,6 @@ def register( callback_type = module.functions[0].arguments[0].semantic_type callback_arguments = callback_type.metadata["callback_arguments"] - assert [arg.metadata[CALLBACK_DECLARATION_ACCESS_METADATA] for arg in callback_arguments] == [ - "unspecified", - "unspecified", - "unspecified", - "unspecified", - "read", - "write", - "readwrite", - ] assert [arg.origin.metadata["value"] for arg in callback_arguments] == [ True, False, @@ -255,7 +243,7 @@ def register( assert callback_arguments[1].semantic_type.storage.kind == "array" assert callback_arguments[2].semantic_type.storage.kind == "array" assert callback_arguments[3].semantic_type.storage.kind == "reference" - assert callback_arguments[4].semantic_type.storage.read_only is True + assert callback_arguments[4].semantic_type.storage.mutable is True assert callback_arguments[5].semantic_type.storage.mutable is True assert callback_arguments[6].semantic_type.storage.mutable is True @@ -263,32 +251,18 @@ def register( @pytest.mark.parametrize( "annotation", [ - "Callable[[Out(String[8])], None]", - "Callable[[InOut(String[8])], None]", - "Callable[[PassByRef(String[8])], None]", - ], -) -def test_callback_writable_plain_string_requires_scalar_storage(annotation: str): - module = parse_pyi_text( - f"def register(callback: {annotation}) -> None: ...", - module_name="callbacks", - ) - - with pytest.raises(ValueError, match="Writable callback strings require mutable scalar character storage"): - complete_semantic_policies(module) - - -@pytest.mark.parametrize( - "annotation", - [ - "Callable[[In(String[8])], None]", - "Callable[[Out(String[8][()])], None]", - "Callable[[InOut(String[8][()])], None]", + "String[8]", + "String[8][()]", ], ) def test_callback_string_storage_contracts_complete(annotation: str): module = parse_pyi_text( - f"def register(callback: {annotation}) -> None: ...", + f""" +@prototype +def string_callback(value: {annotation}) -> None: ... + +def register(callback: string_callback) -> None: ... +""", module_name="callbacks", ) @@ -298,11 +272,17 @@ def test_callback_string_storage_contracts_complete(annotation: str): assert callback_argument.semantic_type.name == "String" -def test_convert_pyi_to_ir_infers_callback_dimension_argument_names(): +def test_convert_pyi_to_ir_preserves_prototype_argument_names_and_dimensions(): module = parse_pyi_text( """ +@prototype +def transform_callback( + count: Int32, + values: Float64[count], +) -> Float64[count]: ... + def apply_transform( - callback: Callable[[PassByRef(Int32), Float64[count]], Float64[count]] + callback: transform_callback ) -> None: ... """, module_name="callbacks", @@ -310,8 +290,47 @@ def apply_transform( callback_type = module.functions[0].arguments[0].semantic_type callback_arguments = callback_type.metadata["callback_arguments"] - assert [arg.name for arg in callback_arguments] == ["count", "arg_1"] + assert [arg.name for arg in callback_arguments] == ["count", "values"] assert callback_type.metadata["return"].shape == ["count"] + assert callback_type.metadata["prototype_ref"]["name"] == "transform_callback" + + +def test_imported_prototype_resolves_as_module_interface_definition(tmp_path): + from x2py.pipeline.pyi import pyi_paths_to_semantic_modules + from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner + + (tmp_path / "callback_shapes.pyi").write_text( + """from x2py.contracts import Float64, Int32, prototype + +@prototype +def transform(count: Int32, values: Float64[count]) -> Float64[count]: ... +""", + encoding="utf-8", + ) + (tmp_path / "api.pyi").write_text( + """from x2py.contracts import Float64, Int32 +from .callback_shapes import transform + +def apply(callback: transform, count: Int32, values: Float64[count]) -> None: ... +""", + encoding="utf-8", + ) + + modules = {module.name: module for module in pyi_paths_to_semantic_modules(tmp_path)} + api = modules["api"] + callback_type = api.functions[0].arguments[0].semantic_type + assert callback_type.metadata["prototype_ref"] == { + "name": "transform", + "local_name": "transform", + "origin_module": "callback_shapes", + } + + complete_semantic_policies(api) + artifacts = WrapperCodeGenerator().generate(WrapperPlanner().build(api)) + bridge = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + assert "use callback_shapes, only:" in bridge + assert "_prototype => transform" in bridge + assert "procedure(x2py_callback_adapter_callback_" in bridge def test_convert_pyi_to_ir_forwards_filename_to_syntax_errors(): @@ -439,19 +458,19 @@ def invalid(value: String[:]) -> None: ... @pytest.mark.parametrize( - ("annotation", "message"), + "annotation", [ - ("Callable[[In(Addr(Int32))], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), - ("Callable[[Out(Addr(Float64))], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), - ("Callable[[InOut(Addr(Float64))], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), - ("Callable[[Addr(Float64)], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), - ("Callable[[Addr(Float64[n])], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), - ("Callable[[Addr[2](Float64)], None]", r"Addr\(\.\.\.\) is not valid inside Callable"), + "Addr(Float64)", + "Addr(Float64[n])", + "Addr[2](Float64)", ], ) -def test_convert_pyi_to_ir_rejects_invalid_callback_reference_wrappers(annotation: str, message: str): - with pytest.raises(ValueError, match=message): - parse_pyi_text(f"def register(callback: {annotation}) -> None: ...", module_name="callbacks") +def test_convert_pyi_to_ir_rejects_unnecessary_prototype_address_wrappers(annotation: str): + with pytest.raises(ValueError, match=r"Addr\(\.\.\.\) is unnecessary inside prototype declarations"): + parse_pyi_text( + f"@prototype\ndef callback(value: {annotation}) -> None: ...", + module_name="callbacks", + ) def test_convert_pyi_to_ir_preserves_explicit_array_source_dimensions(): @@ -792,8 +811,6 @@ def helper(value: Int32) -> None: ... [ ("value: Addr(Int32, Float64)\n", "Addr type expects one argument: 'Addr(Int32, Float64)'"), ("value: Addr[1](Int32)\n", "Addr[1](...) is invalid; use Addr(...)"), - ("value: Callable[Int32]\n", "Callable expects argument types and a return type: 'Callable[Int32]'"), - ("value: Callable[Int32, Float64]\n", "Callable arguments must be a list: 'Callable[Int32, Float64]'"), ( "value: Float64[ORDER_F]\n", "Non-dimensional type subscriptions are not supported; use Final[...] for constants and " diff --git a/tests/semantics/fixtures/general/basic_subroutine.json b/tests/semantics/fixtures/general/basic_subroutine.json index 6f92c5987..dc81b838a 100644 --- a/tests/semantics/fixtures/general/basic_subroutine.json +++ b/tests/semantics/fixtures/general/basic_subroutine.json @@ -220,6 +220,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/compile_time_all_exprs.json b/tests/semantics/fixtures/general/compile_time_all_exprs.json index 3b14b6f06..2b1d20c3a 100644 --- a/tests/semantics/fixtures/general/compile_time_all_exprs.json +++ b/tests/semantics/fixtures/general/compile_time_all_exprs.json @@ -1072,6 +1072,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [ diff --git a/tests/semantics/fixtures/general/compile_time_shape_exprs.json b/tests/semantics/fixtures/general/compile_time_shape_exprs.json index f663fbcd0..f7bf83b67 100644 --- a/tests/semantics/fixtures/general/compile_time_shape_exprs.json +++ b/tests/semantics/fixtures/general/compile_time_shape_exprs.json @@ -260,6 +260,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [ diff --git a/tests/semantics/fixtures/general/contract_import_graph.json b/tests/semantics/fixtures/general/contract_import_graph.json index cd75c54fb..3573b99bc 100644 --- a/tests/semantics/fixtures/general/contract_import_graph.json +++ b/tests/semantics/fixtures/general/contract_import_graph.json @@ -140,6 +140,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], @@ -295,6 +296,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/contract_mixed_module_external.json b/tests/semantics/fixtures/general/contract_mixed_module_external.json index 64839a370..5c645128e 100644 --- a/tests/semantics/fixtures/general/contract_mixed_module_external.json +++ b/tests/semantics/fixtures/general/contract_mixed_module_external.json @@ -140,6 +140,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/contract_multi_module.json b/tests/semantics/fixtures/general/contract_multi_module.json index 1d85ffb6e..c7e8e1b3c 100644 --- a/tests/semantics/fixtures/general/contract_multi_module.json +++ b/tests/semantics/fixtures/general/contract_multi_module.json @@ -140,6 +140,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], @@ -295,6 +296,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/contract_same_name.json b/tests/semantics/fixtures/general/contract_same_name.json index f85fdd331..81bced500 100644 --- a/tests/semantics/fixtures/general/contract_same_name.json +++ b/tests/semantics/fixtures/general/contract_same_name.json @@ -24,6 +24,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/derived_type.json b/tests/semantics/fixtures/general/derived_type.json index 649c3c968..98e001f60 100644 --- a/tests/semantics/fixtures/general/derived_type.json +++ b/tests/semantics/fixtures/general/derived_type.json @@ -106,6 +106,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [ { diff --git a/tests/semantics/fixtures/general/derived_types_and_methods.json b/tests/semantics/fixtures/general/derived_types_and_methods.json index 7c5fdaf31..4a17cd54d 100644 --- a/tests/semantics/fixtures/general/derived_types_and_methods.json +++ b/tests/semantics/fixtures/general/derived_types_and_methods.json @@ -3,6 +3,7 @@ { "name": "mesh_mod", "functions": [], + "prototypes": [], "overload_sets": [], "classes": [ { diff --git a/tests/semantics/fixtures/general/modern_pyi_example.json b/tests/semantics/fixtures/general/modern_pyi_example.json index 545d4d081..6900956e9 100644 --- a/tests/semantics/fixtures/general/modern_pyi_example.json +++ b/tests/semantics/fixtures/general/modern_pyi_example.json @@ -1736,6 +1736,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [ { diff --git a/tests/semantics/fixtures/general/module_vars_use.json b/tests/semantics/fixtures/general/module_vars_use.json index 20748a89d..3b38dc9f6 100644 --- a/tests/semantics/fixtures/general/module_vars_use.json +++ b/tests/semantics/fixtures/general/module_vars_use.json @@ -3,6 +3,7 @@ { "name": "constants_mod", "functions": [], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [ diff --git a/tests/semantics/fixtures/general/procedures_and_functions.json b/tests/semantics/fixtures/general/procedures_and_functions.json index f61099ff5..c573bec3c 100644 --- a/tests/semantics/fixtures/general/procedures_and_functions.json +++ b/tests/semantics/fixtures/general/procedures_and_functions.json @@ -390,6 +390,7 @@ } } ], + "prototypes": [], "overload_sets": [], "classes": [], "variables": [], diff --git a/tests/semantics/fixtures/general/scope_name_reuse_combinations.json b/tests/semantics/fixtures/general/scope_name_reuse_combinations.json index c114caf19..6a721408a 100644 --- a/tests/semantics/fixtures/general/scope_name_reuse_combinations.json +++ b/tests/semantics/fixtures/general/scope_name_reuse_combinations.json @@ -926,6 +926,7 @@ } } ], + "prototypes": [], "overload_sets": [ { "name": "do_work", diff --git a/tests/semantics/fixtures/wrap_readiness_messages.json b/tests/semantics/fixtures/wrap_readiness_messages.json index 5bde646a8..02570160d 100644 --- a/tests/semantics/fixtures/wrap_readiness_messages.json +++ b/tests/semantics/fixtures/wrap_readiness_messages.json @@ -23144,12 +23144,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23766,12 +23766,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23784,12 +23784,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23822,12 +23822,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 3 } ] @@ -23841,7 +23841,7 @@ "n_variables": 0, "messages": [ "Some shape expressions refer to symbols not supplied by the semantic interface.", - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { @@ -23851,7 +23851,7 @@ }, { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23864,13 +23864,13 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "Some shape expressions refer to symbols not supplied by the semantic interface." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 }, { @@ -23956,12 +23956,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23974,12 +23974,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -23992,12 +23992,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24010,12 +24010,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24682,12 +24682,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24700,12 +24700,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24718,12 +24718,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24736,12 +24736,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24764,12 +24764,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -24782,12 +24782,12 @@ "n_classes": 0, "n_variables": 0, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -25036,12 +25036,12 @@ "n_classes": 0, "n_variables": 3, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 1 } ] @@ -25054,12 +25054,12 @@ "n_classes": 0, "n_variables": 5, "messages": [ - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file." + "Some callback or procedure arguments need a complete named prototype in the .pyi file." ], "blockers": [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "n_items": 2 } ] diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index c9300581c..79a28bab6 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -34,6 +34,7 @@ BridgeDataAction, CallbackABIKind, CallbackTransferAction, + ExternalDeclarationMode, FunctionWrapperPolicy, ModuleGetterAction, ModuleVariablePolicy, @@ -232,22 +233,41 @@ def test_source_hidden_scalar_output_completes_call_local_address_before_plannin assert policy.native_call_slots[1].native_barrier_action is hidden.native_barrier_action -def test_source_callback_value_and_read_access_are_completed_as_independent_facts(): +def test_source_callback_value_override_and_reference_default_are_completed(): module = _source_semantic_module("fcallback_all_f90.f90", module_name="fcallback_all_f90") function = next(item for item in module.functions if item.name == "apply_value_callback") policy = completed_function_wrapper_policy(function) transfer = policy.arguments[0].callback.arguments[0] assert transfer.abi is CallbackABIKind.VALUE - assert transfer.access == "read" + assert transfer.passed_by_value is True assert transfer.adapter_action is CallbackTransferAction.COPY_IN array_function = next(item for item in module.functions if item.name == "apply_array_storage_callback") array_policy = completed_function_wrapper_policy(array_function) extent = array_policy.arguments[0].callback.arguments[0] assert extent.abi is CallbackABIKind.REFERENCE - assert extent.access == "read" - assert extent.adapter_action is CallbackTransferAction.COPY_IN + assert extent.passed_by_value is False + assert extent.adapter_action is CallbackTransferAction.COPY_IN_OUT + + +def test_external_declaration_mode_is_completed_from_native_abi_requirements(): + module = parse_pyi_text( + """ +@external +def classic(n: Int32, values: Float64[n]) -> Float64: ... + +@external +def optional(value: Annotated[Float64, Immutable] | None = ...) -> None: ... +""", + module_name="external_modes", + ) + complete_semantic_policies(module) + + classic = completed_function_wrapper_policy(module.functions[0]) + optional = completed_function_wrapper_policy(module.functions[1]) + assert classic.external_declaration is ExternalDeclarationMode.IMPLICIT_EXTERNAL + assert optional.external_declaration is ExternalDeclarationMode.EXPLICIT_INTERFACE def test_hidden_scalar_descriptor_result_keeps_descriptor_policy_instead_of_plain_address_storage(): diff --git a/tests/semantics/readiness/test_c_readiness.py b/tests/semantics/readiness/test_c_readiness.py index b9fc2a7f8..9351287f9 100644 --- a/tests/semantics/readiness/test_c_readiness.py +++ b/tests/semantics/readiness/test_c_readiness.py @@ -72,11 +72,14 @@ def test_completed_pyi_callback_policy_can_make_c_api_semantically_ready(): module = parse_pyi_text( """ -from x2py.contracts import Addr, Callable, Int8 +from x2py.contracts import Addr, Int8, prototype + +@prototype +def item_visitor(item: Int8, userdata: Int8) -> None: ... def each_item( items: Addr(Int8), - visit: Callable[[Int8, Int8], None], + visit: item_visitor, userdata: Addr(Int8), ) -> None: ... """, diff --git a/tests/semantics/readiness/test_pyi_readiness.py b/tests/semantics/readiness/test_pyi_readiness.py index 8dd855e55..68d97e640 100644 --- a/tests/semantics/readiness/test_pyi_readiness.py +++ b/tests/semantics/readiness/test_pyi_readiness.py @@ -8,6 +8,7 @@ SemanticConstraint, SemanticFunction, SemanticModule, + SemanticStorageContract, SemanticType, _blocker_codes, _readiness_from_pyi, @@ -20,7 +21,7 @@ def test_completed_pyi_interface_is_semantically_ready(): report = _readiness_from_pyi( """ -from x2py.contracts import Callable, Final, Float64, Int32, Returns +from x2py.contracts import Final, Float64, Int32, Returns, prototype rk: Final[Int32] = 8 nmax: Final[Int32] = 32 @@ -29,10 +30,13 @@ class sim_state: n: Int32 values: Float64[n] +@prototype +def score_callback(state: sim_state, t: Float64) -> Float64: ... + def step( state: sim_state, t: Float64, - objective: Callable[[sim_state, Float64], Float64], + objective: score_callback, scratch: Float64[nmax] ) -> tuple[Returns["state", sim_state], Returns["score", Float64]]: ... """ @@ -54,7 +58,7 @@ def step(state: sim_state) -> Returns["state", sim_state]: ... assert report["wrappable"] is True -def test_callback_placeholder_blocks_until_callable_signature_is_supplied(): +def test_callback_placeholder_blocks_until_named_prototype_is_supplied(): report = _readiness_from_pyi( """ def integrate(objective: Procedure, x0: Float64) -> Float64: ... @@ -71,29 +75,19 @@ def integrate(objective: Procedure, x0: Float64) -> Float64: ... ] -def test_callable_with_signature_makes_callback_ready(): +def test_named_prototype_makes_callback_ready(): report = _readiness_from_pyi( """ -from x2py.contracts import Callable, Float64 - -def integrate(objective: Callable[[Float64], Float64], x0: Float64) -> Float64: ... -""" - ) - - assert report["wrappable"] is True +from x2py.contracts import Float64, prototype +@prototype +def objective(value: Float64) -> Float64: ... -def test_callable_without_argument_list_is_not_enough_for_readiness(): - report = _readiness_from_pyi( - """ -from x2py.contracts import Callable, Float64 - -def integrate(objective: Callable[..., Float64], x0: Float64) -> Float64: ... +def integrate(callback: objective, x0: Float64) -> Float64: ... """ ) - assert report["wrappable"] is False - assert "callback_signature_incomplete" in _blocker_codes(report) + assert report["wrappable"] is True def test_assess_pyi_wrap_readiness_expands_directory_and_uses_leaf_filenames(tmp_path: Path): @@ -139,14 +133,21 @@ def step(a: state_mod.state_t, b: mesh.mesh_t, c: imported_value) -> None: ... assert report["wrappable"] is True -def test_readiness_reports_incomplete_callable_payload(): +def test_readiness_reports_incomplete_prototype_payload(): module = SemanticModule( "callbacks", functions=[ SemanticFunction( "run", arguments=[ - SemanticArgument("cb", SemanticType("Callable", metadata={"return": SemanticType("Int32")})) + SemanticArgument( + "cb", + SemanticType( + "incomplete_prototype", + metadata={"return": SemanticType("Int32")}, + storage=SemanticStorageContract(kind="callback"), + ), + ) ], ) ], @@ -157,12 +158,12 @@ def test_readiness_reports_incomplete_callable_payload(): assert report["wrappability_blockers"] == [ { "code": "callback_signature_incomplete", - "message": "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "message": "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "items": [ { "owner": "callbacks.run.cb", "item": "cb", - "type": "Callable", + "type": "incomplete_prototype", "needs": [ "callback argument order", "callback argument types", @@ -183,7 +184,7 @@ def test_readiness_reports_incomplete_callable_payload(): def test_readiness_propagates_context_through_imports_callbacks_and_class_fields(): nested_callback = SemanticType( - "Callable", + "nested_prototype", metadata={ "arguments": [ SemanticType("MissingCallbackArg"), @@ -192,6 +193,7 @@ def test_readiness_propagates_context_through_imports_callbacks_and_class_fields ], "return": SemanticType("types_mod.callback_result_t"), }, + storage=SemanticStorageContract(kind="callback"), ) module = SemanticModule( "edge", diff --git a/tests/semantics/readiness/test_reports.py b/tests/semantics/readiness/test_reports.py index 59628263c..bd3266179 100644 --- a/tests/semantics/readiness/test_reports.py +++ b/tests/semantics/readiness/test_reports.py @@ -427,11 +427,12 @@ def test_readiness_metadata_defaults_and_duplicate_unit_blockers_are_stable(): def test_readiness_preserves_ordering_and_nested_type_context(): nested_callback = SemanticType( - "Callable", + "nested_prototype", metadata={ "arguments": [SemanticType("types_mod.input_t")], "return": SemanticType("Float64", shape=["n + missing"]), }, + storage=SemanticStorageContract(kind="callback"), ) module = SemanticModule( "ordered", @@ -534,11 +535,12 @@ def test_readiness_report_preserves_blocker_payloads_and_unit_ownership(): metadata={"readiness_blockers": [{"code": "type_policy", "message": "type message", "item": {"detail": "t"}}]}, ) callback_type = SemanticType( - "Callable", + "callback_prototype", metadata={ "arguments": [SemanticType("MissingCallbackArg")], "return": SemanticType("MissingCallbackReturn"), }, + storage=SemanticStorageContract(kind="callback"), ) module = SemanticModule( name="api", @@ -654,7 +656,7 @@ def test_readiness_report_preserves_blocker_payloads_and_unit_ownership(): assert units["api.unknown"]["kind"] == "function" assert set(units) == {"api", "api.State", "api.State.apply", "api.hook", "api.unknown"} assert set(report["why_not_wrappable"]) == { - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "Some callback or procedure arguments need a complete named prototype in the .pyi file.", "Some compile-time constants are declared but do not have literal .pyi values.", "Some semantic type references are not declared by the .pyi interface or its imports.", "argument message", diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index 26600709b..31e21941b 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -44,7 +44,7 @@ recorded progression, not in the live ledger. | --- | --- | | Shared source/generated fixture pattern for standalone externals | `external_routines/test_external_procedures.py::test_fixed_form_standalone_external_runtime_parity`, `external_routines/test_external_procedures.py::test_free_form_standalone_external_runtime_parity`, `external_routines/test_external_procedures.py::test_one_source_with_several_standalone_externals_exports_each_at_root` | | Fixed-form, free-form, multi-procedure, and compact BLAS-like external contracts | `external_routines/test_external_procedures.py::test_generated_external_contracts_are_non_empty_root_fragments`, `external_routines/test_external_procedures.py::test_compact_blas_like_folder_generates_one_external_entry_and_preserves_separate_objects` | -| External bridge placement and module-procedure contrast | `external_routines/test_external_procedures.py::test_external_bridge_uses_explicit_interface_and_no_module_use`, `external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | +| External bridge placement and module-procedure contrast | `external_routines/test_external_procedures.py::test_classic_external_bridge_uses_implicit_declaration_and_no_module_use`, `external_routines/test_external_procedures.py::test_module_procedure_bridge_uses_native_module_scope` | | `@external` with `@bind` and handwritten source-free contracts | `external_routines/test_external_procedures.py::test_external_bind_renames_python_export_without_changing_native_call` | | C-order flat storage over assumed-size native external buffers | `external_routines/test_external_procedures.py::test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view` | | Invalid root/module placement edits fail before code generation | `external_routines/test_external_procedures.py::test_package_entry_rejects_non_external_root_declaration_before_codegen`, `external_routines/test_external_procedures.py::test_namespace_imported_module_rejects_external_marker_before_codegen` | @@ -58,7 +58,7 @@ recorded progression, not in the live ledger. | Array contracts rebuild from generated `.pyi` fixtures with the same dtype, rank, shape, order, stride, lower-bound, writeability, byte-order, alignment, zero-extent, assumed-rank dispatch, ordinary Python-owned result behavior, and allocatable result-handle behavior as source builds | `arrays/test_array_contracts.py::test_remaining_array_contracts_are_validated_before_fortran_calls`, `arrays/test_array_results.py::test_array_results_follow_data_buffer_and_descriptor_handle_contracts`, `arrays/test_array_results.py::test_owned_allocatable_results_preserve_handle_state`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_arguments_dispatch_to_runtime_rank`, `arrays/test_assumed_rank_arrays.py::test_assumed_rank_bridge_dispatches_each_runtime_rank_argument`, `arrays/test_multidimensional_arrays.py::test_rank2_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_accepts_fortran_ordered_strided_views`, `arrays/test_multidimensional_arrays.py::test_rank2_assumed_shape_rejects_non_positive_strides`, `arrays/test_multidimensional_arrays.py::test_rank2_explicit_shape_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_contiguous_contract_requires_fortran_contiguous`, `arrays/test_multidimensional_arrays.py::test_rank3_assumed_shape_accepts_fortran_ordered_strided_views` | | Character contracts rebuild from generated `.pyi` fixtures with the same fixed-length buffers, assumed-length strings, nullable deferred scalar results, deferred-width native handles, copy-in/copy-out behavior, optional strings, Unicode handling, embedded-NUL validation, and raw fixed-width array addresses as source builds | `strings/test_character_arguments.py::test_legacy_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_modern_fortran_character_arguments_and_results`, `strings/test_character_arguments.py::test_deferred_allocatable_string_results_use_canonical_plan`, `strings/test_character_arguments.py::test_deferred_character_array_handles_use_canonical_plan`, `strings/test_character_arguments.py::test_raw_fixed_width_character_arrays_use_canonical_plan`, `strings/test_character_edge_cases.py::test_fortran_character_edge_cases_follow_copy_in_copy_out_policy`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_projected_replacement_without_native_call_keeps_writable_argument_storage`, `tests/semantics/conversion/pyi/test_calls_and_projections.py::test_native_call_projected_output_keeps_visible_storage_writable` | | Derived-type contracts rebuild from generated `.pyi` fixtures with the same fields, methods, type-bound root targets, constructors, finalizers, borrowed child lifetime, scalar object boundaries, inheritance, polymorphic dispatch, complete scalar actual/dummy compatibility, descriptor-backed scalar module proxies, wrapper-owned allocatable/pointer holders, and exact readiness blockers as source builds | `derived_types/test_derived_type_methods.py::test_modern_fortran_derived_type_exposes_class_and_type_bound_methods`, `derived_types/test_constructors_and_finalizers.py::test_fortran_default_constructor_keywords_and_finalization`, `derived_types/test_borrowed_finalizers.py::test_borrowed_child_wrapper_never_finalizes_native_component`, `derived_types/test_derived_layout.py::test_bind_c_derived_types_use_accessors_and_fortran_value_copy`, `derived_types/test_derived_type_boundaries.py::test_scalar_derived_types_cross_procedure_boundaries`, `derived_types/test_phase8_derived_plan.py`, `derived_types/test_scalar_derived_actual_dummy_matrix.py`, `derived_types/test_inheritance.py::test_fortran_extension_types_generate_python_inheritance`, `derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy`, `tests/wrapper_codegen/test_phase8_scalar_derived_actual_dummy_matrix.py`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py::test_type_bound_method_declarations_restore_root_target_metadata` | -| Callback contracts route through wrapper-plan generation without legacy lowering and rebuild from generated `.pyi` fixtures with the same value, scalar-storage, array, character-storage, and derived callback conversions, call-scoped lifetime, nested same-thread entry, GIL handling, reference cleanup, and fatal exception behavior as source builds | `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `callbacks/test_scalar_callbacks.py::test_immediate_scalar_dummy_procedure_calls_python_callback`, `callbacks/test_scalar_callbacks.py::test_callback_exception_prints_traceback_and_aborts_host_process`, `callbacks/test_array_callbacks.py::test_immediate_dummy_procedure_converts_array_arguments_and_results`, `callbacks/test_derived_callbacks.py::test_immediate_dummy_procedure_converts_derived_arguments_and_results`, `tests/wrapper_codegen/test_phase10_callbacks.py`, `tests/semantics/conversion/pyi/test_types_and_values.py::test_convert_pyi_to_ir_infers_callback_dimension_argument_names` | +| Callback contracts route through wrapper-plan generation without legacy lowering and rebuild from generated `.pyi` fixtures with the same value, scalar-storage, array, character-storage, and derived callback conversions, call-scoped lifetime, nested same-thread entry, GIL handling, reference cleanup, and fatal exception behavior as source builds | `callbacks/test_all_callback_shapes.py::test_immediate_callbacks_cover_all_supported_argument_shapes`, `callbacks/test_scalar_callbacks.py::test_immediate_scalar_dummy_procedure_calls_python_callback`, `callbacks/test_scalar_callbacks.py::test_callback_exception_prints_traceback_and_aborts_host_process`, `callbacks/test_array_callbacks.py::test_immediate_dummy_procedure_converts_array_arguments_and_results`, `callbacks/test_derived_callbacks.py::test_immediate_dummy_procedure_converts_derived_arguments_and_results`, `tests/wrapper_codegen/test_phase10_callbacks.py`, `tests/semantics/conversion/pyi/test_types_and_values.py::test_convert_pyi_to_ir_preserves_prototype_argument_names_and_dimensions` | | Module-state contracts rebuild from generated `.pyi` fixtures with the same scalar module attributes, parameter behavior, saved native state, plain and `Aliased` live allocatable module views, borrowed field handles, owned allocatable result handles, same-handle allocatable descriptor mutation, explicit-copy independence, fresh extraction after state changes, rank-zero descriptor copying/nullability, and common-block encapsulation as source builds; isolated scalar and native-handle owners also replay legacy and wrapper-plan routes | `module_state/test_module_state.py::test_scalar_module_variables_use_attributes_and_parameters_have_no_native_setter`, `module_state/test_scalar_module_variable_plan.py::test_whole_scalar_module_variable_behavior_uses_canonical_plan`, `module_state/test_allocatable_views.py::test_allocatable_module_fields_and_results_expose_lifetime_safe_handles`, `module_state/test_allocatable_views.py::test_plain_allocatable_module_array_exposes_current_live_view`, `module_state/test_allocatable_views.py::test_scalar_descriptor_module_variables_return_copied_optional_values`, `module_state/test_allocatable_replacement.py::test_allocatable_inout_arrays_mutate_and_return_the_same_handle`, `module_state/test_allocatable_replacement.py::test_projected_allocatable_descriptor_preserves_same_handle_identity`, `derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan`, `module_state/test_common_blocks.py::test_common_block_storage_stays_internal_to_wrapped_fortran` | | Runtime behavior contracts rebuild from generated or edited `.pyi` fixtures with the same recursion/reentrancy behavior, `@hold_gil` GIL policy, `@raises(...)` status projection, and generated wrapper policy code as source-backed builds | `runtime_behavior/test_runtime_recursion.py::test_recursive_native_runtime_calls`, `runtime_behavior/test_runtime_policies.py::test_pyi_runtime_policies_release_gil_and_project_native_errors`, `runtime_behavior/test_runtime_policies.py::test_compiled_runtime_policies_release_gil_and_project_native_errors`, `runtime_behavior/test_openmp_runtime.py::test_openmp_enabled_procedure_builds_with_explicit_gnu_flags` | | Naming and generic-interface contracts rebuild from generated `.pyi` fixtures with the same public-name normalization, visibility filtering, keyword/collision policy, public generic dispatch, type-bound binding names, defined operators, comparisons, named operators, and assignment behavior as source builds | `naming/test_visibility_naming.py::test_visibility_and_default_python_name_fixing_policy`, `naming/test_generic_interfaces.py::test_fortran_generic_interfaces_dispatch_in_generated_c_extension`, `naming/test_generic_interfaces.py::test_fixed_form_fortran_generic_interface_dispatches_in_generated_c_extension`, `naming/test_defined_operators.py::test_fortran_defined_operators_and_assignment_dispatch_in_generated_c_extension`, `tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py::test_fortran_generated_contracts_reserve_colliding_public_names_by_namespace`, `tests/semantics/conversion/pyi/test_classes_and_overloads.py::test_pyi_keyword_normalized_type_bound_method_keeps_native_binding_name` | diff --git a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py index 8e517d17f..71dca7f29 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py +++ b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py @@ -233,7 +233,7 @@ def test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support nested.mkdir(parents=True) entry.write_text( "from types import SimpleNamespace\n" - "from x2py.contracts import Callable\n" + "from x2py.contracts import prototype\n" "from . import facade as m2\n" "from .m1 import func as f\n", encoding="utf-8", @@ -257,7 +257,6 @@ def test_recursive_graph_preserves_module_and_symbol_aliases_and_ignores_support assert not hasattr(module, "m1") assert not hasattr(module, "func") assert not hasattr(module, "SimpleNamespace") - assert not hasattr(module, "Callable") assert module.f(np.int32(2)) == np.int32(3) assert module.m2.branch.deep_func(np.int32(3)) == np.int32(6) diff --git a/tests/wrapper/fortran/callbacks/contracts/fcallback_all_f90/fcallback_all_f90.pyi b/tests/wrapper/fortran/callbacks/contracts/fcallback_all_f90/fcallback_all_f90.pyi index 0af4621b8..612630adc 100644 --- a/tests/wrapper/fortran/callbacks/contracts/fcallback_all_f90/fcallback_all_f90.pyi +++ b/tests/wrapper/fortran/callbacks/contracts/fcallback_all_f90/fcallback_all_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Arg, Callable, Float64, In, InOut, Int32, Out, PassByRef, Return, Returns, String, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Return, Returns, String, Value, native_call, prototype class point_t: def __init__( @@ -11,22 +11,53 @@ class point_t: x: Float64 y: Float64 +@prototype +def value_callback( + value: Value(Int32) +) -> Int32: ... + +@prototype +def scalar_storage_callback( + value: Float64, + output: Float64, + missing: Float64 +) -> None: ... + +@prototype +def array_storage_callback( + count: Int32, + values: Float64[count], + output: Float64[count] +) -> None: ... + +@prototype +def string_storage_callback( + read_label: String[8], + write_label: String[8], + update_label: String[8] +) -> None: ... + +@prototype +def point_callback( + value: point_t +) -> point_t: ... + @native_call([Arg(0), Addr(Arg(1))]) def apply_value_callback( - callback: Callable[[Int32], Int32], + callback: value_callback, value: Int32 ) -> Int32: ... @native_call([Arg(0), Addr(Arg(1)), Addr(Arg(2)), Return('output', 0)]) def apply_scalar_storage_callback( - callback: Callable[[InOut(Float64), Out(Float64), PassByRef(Float64)], None], + callback: scalar_storage_callback, value: Float64, missing: Float64 ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1)), Arg(2), Arg(3)]) def apply_array_storage_callback( - callback: Callable[[In(Int32), In(Float64[count]), Out(Float64[count])], None], + callback: array_storage_callback, count: Int32, values: Float64[count], output: Float64[count] @@ -34,12 +65,12 @@ def apply_array_storage_callback( @native_call([Arg(0), Arg(1), Return('write_label', 1)]) def apply_string_storage_callback( - callback: Callable[[In(String[8]), Out(String[8][()]), InOut(String[8][()])], None], + callback: string_storage_callback, update_label: String[8] ) -> tuple[Returns["update_label", String[8]], String[8]]: ... @native_call([Arg(0), Arg(1), Return('output', 0)]) def apply_point_callback( - callback: Callable[[In(point_t)], point_t], + callback: point_callback, value: point_t ) -> point_t: ... diff --git a/tests/wrapper/fortran/callbacks/contracts/fcallback_array_f90/fcallback_array_f90.pyi b/tests/wrapper/fortran/callbacks/contracts/fcallback_array_f90/fcallback_array_f90.pyi index d52d11c71..dad878c74 100644 --- a/tests/wrapper/fortran/callbacks/contracts/fcallback_array_f90/fcallback_array_f90.pyi +++ b/tests/wrapper/fortran/callbacks/contracts/fcallback_array_f90/fcallback_array_f90.pyi @@ -1,15 +1,27 @@ -from x2py.contracts import Addr, Arg, Callable, Float64, In, Int32, Returns, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call, prototype + +@prototype +def reduce_callback( + count: Int32, + values: Float64[count] +) -> Float64: ... + +@prototype +def transform_callback( + count: Int32, + values: Float64[count] +) -> Float64[count]: ... @native_call([Arg(0), Addr(Arg(1)), Arg(2)]) def apply_reduce( - callback: Callable[[In(Int32), In(Float64[count])], Float64], + callback: reduce_callback, count: Int32, values: Float64[count] ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1)), Arg(2), Arg(3)]) def apply_transform( - callback: Callable[[In(Int32), In(Float64[count])], Float64[count]], + callback: transform_callback, count: Int32, values: Float64[count], output: Float64[count] diff --git a/tests/wrapper/fortran/callbacks/contracts/fcallback_derived_f90/fcallback_derived_f90.pyi b/tests/wrapper/fortran/callbacks/contracts/fcallback_derived_f90/fcallback_derived_f90.pyi index d7b9f3bf6..86124cf2a 100644 --- a/tests/wrapper/fortran/callbacks/contracts/fcallback_derived_f90/fcallback_derived_f90.pyi +++ b/tests/wrapper/fortran/callbacks/contracts/fcallback_derived_f90/fcallback_derived_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Arg, Callable, Float64, In, Return, native_call +from x2py.contracts import Arg, Float64, Return, native_call, prototype class point_t: def __init__( @@ -11,8 +11,13 @@ class point_t: x: Float64 y: Float64 +@prototype +def point_callback( + value: point_t +) -> point_t: ... + @native_call([Arg(0), Arg(1), Return('output', 0)]) def apply_point( - callback: Callable[[In(point_t)], point_t], + callback: point_callback, value: point_t ) -> point_t: ... diff --git a/tests/wrapper/fortran/callbacks/contracts/fcallback_scalar_f90/fcallback_scalar_f90.pyi b/tests/wrapper/fortran/callbacks/contracts/fcallback_scalar_f90/fcallback_scalar_f90.pyi index f619bc77d..522d6b7aa 100644 --- a/tests/wrapper/fortran/callbacks/contracts/fcallback_scalar_f90/fcallback_scalar_f90.pyi +++ b/tests/wrapper/fortran/callbacks/contracts/fcallback_scalar_f90/fcallback_scalar_f90.pyi @@ -1,19 +1,34 @@ -from x2py.contracts import Addr, Arg, Callable, Float64, In, native_call +from x2py.contracts import Addr, Arg, Float64, native_call, prototype + +@prototype +def scalar_callback( + value: Float64 +) -> Float64: ... + +@prototype +def notify_callback( + value: Float64 +) -> None: ... + +@prototype +def callback( + value: Float64 +) -> Float64: ... @native_call([Arg(0), Addr(Arg(1))]) def apply_scalar( - callback: Callable[[In(Float64)], Float64], + callback: scalar_callback, value: Float64 ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1))]) def apply_explicit( - callback: Callable[[In(Float64)], Float64], + callback: callback, value: Float64 ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1))]) def call_notify( - callback: Callable[[In(Float64)], None], + callback: notify_callback, value: Float64 ) -> None: ... diff --git a/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py b/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py index a05702c75..59233eeef 100644 --- a/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py +++ b/tests/wrapper/fortran/callbacks/test_all_callback_shapes.py @@ -43,8 +43,10 @@ def scalar_callback(value, output, missing): output = np.empty_like(values) def array_callback(count, input_values, output_values): + assert count.shape == () + assert count.flags.writeable assert input_values.flags.f_contiguous - assert not input_values.flags.writeable + assert input_values.flags.writeable assert output_values.flags.writeable output_values[:count] = input_values[:count] + 1.5 @@ -53,11 +55,13 @@ def array_callback(count, input_values, output_values): np.testing.assert_allclose(output, np.array([2.5, 3.5, 4.5], dtype=np.float64)) def string_callback(read_label, write_label, update_label): - assert read_label == "READONLY" + assert read_label.shape == () assert write_label.shape == () assert update_label.shape == () + assert read_label.dtype.itemsize == 8 assert write_label.dtype.itemsize == 8 assert update_label.dtype.itemsize == 8 + assert read_label[()] == b"READONLY" assert update_label[()] == b"OLD " write_label[...] = b"WRITTEN!" update_label[...] = b"UPDATED!" diff --git a/tests/wrapper/fortran/external_routines/test_external_procedures.py b/tests/wrapper/fortran/external_routines/test_external_procedures.py index dc02dfd19..47a7512ec 100644 --- a/tests/wrapper/fortran/external_routines/test_external_procedures.py +++ b/tests/wrapper/fortran/external_routines/test_external_procedures.py @@ -205,7 +205,7 @@ def test_generated_external_contracts_are_non_empty_root_fragments(tmp_path: Pat assert sorted(path.name for path in entry.parent.glob("*.pyi")) == ["__init__.pyi"] -def test_external_bridge_uses_explicit_interface_and_no_module_use(tmp_path: Path): +def test_classic_external_bridge_uses_implicit_declaration_and_no_module_use(tmp_path: Path): sources = _copy_sources((FREE_EXTERNAL,), tmp_path / "sources") module, result, entry = _build_generated_contract( sources, @@ -219,8 +219,9 @@ def test_external_bridge_uses_explicit_interface_and_no_module_use(tmp_path: Pat assert entry.read_text(encoding="utf-8").startswith( "from x2py.contracts import Addr, Arg, Int32, external, native_call\n\n@external\n" ) - assert "function free_square(" in bridge - assert "end function free_square" in bridge + assert "integer(c_int32_t), external :: free_square" in bridge + assert "function free_square(" not in bridge + assert "result = free_square(value)" in bridge assert "private\n" not in bridge assert "public :: bind_c_free_square" not in bridge assert "use free_external" not in bridge @@ -256,7 +257,8 @@ def test_external_bind_renames_python_export_without_changing_native_call(tmp_pa assert module.renamed_increment(np.int32(4)) == np.int32(5) assert not hasattr(module, "fixed_add") - assert "function fixed_add(" in bridge + assert "integer(c_int32_t), external :: fixed_add" in bridge + assert "result = fixed_add(value)" in bridge assert "use fixed_add" not in bridge @@ -279,8 +281,8 @@ def test_handwritten_c_order_flat_contract_passes_rank_preserving_bridge_view(tm module.row_sums_c(np.int32(values.shape[0]), values, result_values) np.testing.assert_allclose(result_values, [6.0, 60.0]) - assert "values(*)" in compact_bridge - assert "values(*,3)" not in compact_bridge + assert "external::row_sums_c" in compact_bridge + assert "real(c_double),pointer,dimension(:,:)::values" in compact_bridge with pytest.raises(TypeError, match=r"expected ordering \(C\)"): module.row_sums_c(np.int32(values.shape[0]), np.asfortranarray(values), result_values) diff --git a/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py b/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py index 64de7e211..a75ccebbd 100644 --- a/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py +++ b/tests/wrapper/fortran/real_libraries/test_real_blas_lapack.py @@ -402,7 +402,8 @@ def test_full_library_wrapper_imports_every_root_procedure_from_cached_shared_li if library == "blas": bridge = (result.output_dir / "bind_c_full_blas_wrapper.f90").read_text(encoding="utf-8").lower() assert "use full_blas_interfaces" not in bridge - assert "subroutine daxpy(" in bridge + assert "external :: daxpy" in bridge + assert "subroutine daxpy(" not in bridge assert "private\n" not in bridge assert "public :: bind_c_daxpy" not in bridge _assert_blas_runtime_smoke(module) diff --git a/tests/wrapper_codegen/printers/_support.py b/tests/wrapper_codegen/printers/_support.py index 60d17a249..12c6b93aa 100644 --- a/tests/wrapper_codegen/printers/_support.py +++ b/tests/wrapper_codegen/printers/_support.py @@ -24,7 +24,7 @@ from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner from x2py.semantics.models import ( - CALLBACK_DECLARATION_ACCESS_METADATA, + PROTOTYPE_REF_METADATA, ProjectionMapping, RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA, @@ -36,6 +36,7 @@ SemanticMethod, SemanticModule, SemanticOrigin, + SemanticPrototype, SemanticFunction, SemanticField, SemanticStorageContract, @@ -84,8 +85,8 @@ def normalize(text: str) -> str: __all__ = ( - "CALLBACK_DECLARATION_ACCESS_METADATA", "OPERATOR_F90_SOURCE", + "PROTOTYPE_REF_METADATA", "RUNTIME_HOLD_GIL_METADATA", "RUNTIME_STATUS_ERROR_METADATA", "Path", @@ -101,6 +102,7 @@ def normalize(text: str) -> str: "SemanticMethod", "SemanticModule", "SemanticOrigin", + "SemanticPrototype", "SemanticStorageContract", "SemanticType", "SemanticVariable", diff --git a/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py b/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py index 7a9ff837e..a1e0a0857 100644 --- a/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py +++ b/tests/wrapper_codegen/printers/test_calls_and_policy_metadata.py @@ -1,7 +1,7 @@ """Tests split by stable ownership concept from `test_imports_and_packages.py`.""" from tests.wrapper_codegen.printers._support import ( - CALLBACK_DECLARATION_ACCESS_METADATA, + PROTOTYPE_REF_METADATA, ProjectionMapping, PyiPrinter, RUNTIME_HOLD_GIL_METADATA, @@ -12,6 +12,7 @@ SemanticFunction, SemanticModule, SemanticOrigin, + SemanticPrototype, SemanticStorageContract, SemanticType, SemanticVariable, @@ -240,7 +241,10 @@ def serialized(x: Float64) -> Float64: ... def test_callback_contract_holds_gil_and_release_gil_is_removed(): loaded = parse_pyi_text( """ -def apply(callback: Callable[[Float64], Float64], x: Float64) -> Float64: ... +@prototype +def scalar_callback(value: Float64) -> Float64: ... + +def apply(callback: scalar_callback, x: Float64) -> Float64: ... """, module_name="callback_policy", ) @@ -414,14 +418,6 @@ def test_printer_emits_extended_storage_and_callable_forms(): ), ), ) - full_callback = SemanticType( - "Callable", - metadata={ - "arguments": [SemanticType("Int32"), SemanticType("Float64")], - "return": SemanticType("Float64"), - }, - ) - any_callback = SemanticType("Callable", metadata={"return": SemanticType("Float64")}) character = SemanticType( "String", metadata={"fortran_character_length": "16"}, @@ -458,9 +454,6 @@ def test_printer_emits_extended_storage_and_callable_forms(): assert printer.emit(character) == "String[16]" assert printer.emit(allocatable_character) == "Allocatable[String]" assert printer.emit(pointer_scalar) == "Pointer[Int32]" - assert printer.emit(full_callback) == "Callable[[Int32, Float64], Float64]" - assert printer.emit(any_callback) == "Callable[..., Float64]" - assert printer.emit(SemanticType("Callable")) == "Callable" @pytest.mark.parametrize( @@ -537,7 +530,7 @@ def update(scale: Float64 | None = ..., target: Float64 | None = ...) -> None: . assert "Default is None." not in c_wrapper -def test_printer_emits_callback_argument_abi_wrappers(): +def test_printer_emits_named_prototype_and_reference_with_value_override(): printer = PyiPrinter() missing_reference = SemanticType( "Float64", @@ -581,52 +574,59 @@ def test_printer_emits_callback_argument_abi_wrappers(): SemanticArgument( "value", SemanticType("Int32"), - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "read"}, origin=SemanticOrigin(metadata={"value": True}), ), SemanticArgument( "missing", missing_reference, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "unspecified"}, origin=SemanticOrigin(metadata={"value": False}), ), SemanticArgument( "missing_array", missing_array, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "unspecified"}, origin=SemanticOrigin(metadata={"value": False}), ), SemanticArgument( "read", input_reference, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "read"}, origin=SemanticOrigin(metadata={"value": False}), ), SemanticArgument( "write", output_array, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "write"}, origin=SemanticOrigin(metadata={"value": False}), ), SemanticArgument( "readwrite", inout_array, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: "readwrite"}, origin=SemanticOrigin(metadata={"value": False}), ), ] + prototype = SemanticPrototype( + name="update_values", + native_name="update_values", + arguments=callback_arguments, + return_type=SemanticType("None", dtype="None"), + ) callback = SemanticType( - "Callable", + "update_values", + dtype="Prototype", metadata={ "arguments": [argument.semantic_type for argument in callback_arguments], "callback_arguments": callback_arguments, "return": SemanticType("None"), + PROTOTYPE_REF_METADATA: { + "name": "update_values", + "local_name": "update_values", + "origin_module": "callbacks", + }, }, + storage=SemanticStorageContract(kind="callback"), ) - assert printer.emit(callback) == ( - "Callable[[Int32, PassByRef(Float64), Float64[:], In(Int32), Out(Float64[:]), InOut(Float64[:])], None]" - ) + assert printer.emit(callback) == "update_values" + assert "@prototype\ndef update_values(" in printer.emit(prototype) + assert "value: Value(Int32)" in printer.emit(prototype) def test_printer_projection_return_helpers_and_keyword_data_members(): diff --git a/tests/wrapper_codegen/test_phase10_callbacks.py b/tests/wrapper_codegen/test_phase10_callbacks.py index 74c320104..bed4117dd 100644 --- a/tests/wrapper_codegen/test_phase10_callbacks.py +++ b/tests/wrapper_codegen/test_phase10_callbacks.py @@ -14,6 +14,7 @@ CallbackResultAction, CallbackThreadAction, CallbackTransferAction, + ExternalDeclarationMode, ) from x2py.wrapper_codegen import WrapperCodeGenerator, WrapperPlanner, WrapperPlanSupportAnalyzer from x2py.wrapper_codegen.plan import DatatypeFamily @@ -27,6 +28,7 @@ / "fcallback_all_f90" / "fcallback_all_f90.pyi" ) +ARRAY_CONTRACT = CONTRACT.parents[1] / "fcallback_array_f90" / "fcallback_array_f90.pyi" def _module(): @@ -56,7 +58,7 @@ def _sources(plan): return c_source, bridge -def test_callback_policy_completes_every_legacy_observed_transfer_before_planning(): +def test_callback_policy_completes_reference_default_and_value_override_before_planning(): module = _module() policies = { function.name: function.metadata[models.RESOLVED_FUNCTION_WRAPPER_POLICY_METADATA] @@ -68,15 +70,11 @@ def test_callback_policy_completes_every_legacy_observed_transfer_before_plannin assert scalar.thread_action is CallbackThreadAction.REQUIRE_ENTERING_THREAD assert scalar.gil_actions == (CallbackGILAction.ACQUIRE_GIL, CallbackGILAction.RELEASE_GIL) assert tuple(transfer.abi for transfer in scalar.arguments) == (CallbackABIKind.REFERENCE,) * 3 - assert tuple(transfer.adapter_action for transfer in scalar.arguments) == ( - CallbackTransferAction.COPY_IN_OUT, - CallbackTransferAction.COPY_OUT, - CallbackTransferAction.COPY_IN_OUT, - ) + assert tuple(transfer.adapter_action for transfer in scalar.arguments) == (CallbackTransferAction.COPY_IN_OUT,) * 3 array = policies["apply_array_storage_callback"].arguments[0].callback assert array.arguments[0].abi is CallbackABIKind.REFERENCE - assert array.arguments[0].adapter_action is CallbackTransferAction.COPY_IN + assert array.arguments[0].adapter_action is CallbackTransferAction.COPY_IN_OUT assert array.arguments[1].abi is CallbackABIKind.DATA_AND_SHAPE assert array.arguments[1].array.shape == ("count",) @@ -165,18 +163,34 @@ def test_callback_artifacts_use_linear_context_adapter_and_trampoline_paths(): assert "Py_BEGIN_ALLOW_THREADS" not in c_source assert "Py_END_ALLOW_THREADS" not in c_source - assert "abstract interface" in bridge - assert "integer(c_int32_t), intent(in), value :: arg_0" in bridge - assert "integer(c_int32_t), intent(in) :: count" in bridge - assert "procedure(x2py_callback_trampoline" in bridge - assert "size(arg_1_callback_storage, dim=1, kind=c_int64_t)" in bridge - assert "int(len(arg_0_callback_storage), kind=c_int64_t)" in bridge - assert "Int32_to_PyLong((int32_t *)count_data)" in c_source + assert "integer(c_int32_t), value :: value" in bridge + assert "integer(c_int32_t) :: count" in bridge + assert "external :: x2py_callback_adapter" in bridge + assert 'bind(c, name="x2py_callback_trampoline' in bridge + assert "size(values_callback_storage, dim=1, kind=c_int64_t)" in bridge + assert "int(len(read_label_callback_storage), kind=c_int64_t)" in bridge + assert "NPY_INT32, NULL, count_data, 0, NPY_ARRAY_ALIGNED | NPY_ARRAY_WRITEABLE" in c_source assert bridge.count("call native_apply_array_storage_callback(") == 1 - assert bridge.count("call callback(") == 3 + assert "call callback(" not in bridge assert max(map(len, bridge.splitlines())) <= 132 +def test_callback_declaration_uses_external_unless_prototype_requires_explicit_interface(): + module = pyi_file_to_semantic_module(ARRAY_CONTRACT, module_name="fcallback_array_f90") + complete_semantic_policies(module) + plan = WrapperPlanner().build(module) + + reduce = _callback_argument(plan, "apply_reduce").callback + transform = _callback_argument(plan, "apply_transform").callback + assert reduce.declaration_mode is ExternalDeclarationMode.IMPLICIT_EXTERNAL + assert transform.declaration_mode is ExternalDeclarationMode.EXPLICIT_INTERFACE + + _, bridge = _sources(plan) + assert f"real(c_double), external :: {reduce.adapter_symbol}" in bridge + assert f"procedure({transform.adapter_symbol}_prototype) :: {transform.adapter_symbol}" in bridge + assert f"{transform.adapter_symbol}_prototype => transform_callback" in bridge + + def test_optional_callback_retains_one_exact_policy_blocker(): module = pyi_file_to_semantic_module(CONTRACT, module_name="fcallback_all_f90") function = next(item for item in module.functions if item.name == "apply_value_callback") diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index caafcfb9c..4f9d85475 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -66,8 +66,8 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... assert "static PyObject * wrap_swap_args" in c_header assert "module bind_c_render_demo_wrapper" in fortran_source assert 'function bind_c_swap_args(y, x) result(result) bind(c, name="bind_c_swap_args")' in fortran_source - assert "function SWAP_ARGS(y, x) result(native_result)" in fortran_source - assert "real(c_double) :: native_result" in fortran_source + assert "real(c_double), external :: SWAP_ARGS" in fortran_source + assert "function SWAP_ARGS(" not in fortran_source assert "result = SWAP_ARGS(y, x)" in fortran_source diff --git a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py index 1f7f58865..688128cc3 100644 --- a/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py +++ b/tests/wrapper_codegen/test_phase2b_hidden_scalar_outputs.py @@ -32,9 +32,35 @@ def scale(x: Float64) -> Float64: ... assert "bind_c_scale(&x, &result);" in c_source assert "PyObject * result_obj = Double_to_PyDouble(&result);" in c_source assert 'subroutine bind_c_scale(x, result) bind(c, name="bind_c_scale")' in fortran_source - native_interface = fortran_source.split("subroutine SCALE_OUT(x, result)", maxsplit=1)[1].split( + assert "external :: SCALE_OUT" in fortran_source + assert "subroutine SCALE_OUT(" not in fortran_source + assert "call SCALE_OUT(x, result)" in fortran_source + + +def test_required_explicit_interface_declares_hidden_result_in_native_order(): + module = parse_pyi_text( + """ +from x2py.contracts import Addr, Annotated, Arg, Float64, Immutable, Int32, Return, bind, external, native_call + +@bind("SCALE_OUT") +@external +@native_call([Addr(Arg(0)), Return("result", 0), Addr(Arg(1))]) +def scale( + x: Float64, + mode: Annotated[Int32, Immutable] | None = ..., +) -> Float64: ... +""", + module_name="hidden_result_explicit_interface", + ) + complete_semantic_policies(module) + artifacts = WrapperCodeGenerator().generate(WrapperPlanner().build(module)) + fortran_source = next(source.text for source in artifacts.sources if source.path.suffix == ".f90") + + native_interface = fortran_source.split("subroutine SCALE_OUT(x, result, mode)", maxsplit=1)[1].split( "end subroutine SCALE_OUT", maxsplit=1 )[0] assert "real(c_double) :: x" in native_interface assert "real(c_double) :: result" in native_interface - assert "call SCALE_OUT(x, result)" in fortran_source + assert "integer(c_int32_t), optional :: mode" in native_interface + assert "call SCALE_OUT(x=x, result=result, mode=mode)" in fortran_source + assert "call SCALE_OUT(x=x, result=result)" in fortran_source diff --git a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py index 36e1fce3c..13446ef85 100644 --- a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py +++ b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py @@ -80,10 +80,10 @@ def projected( def _late_extent_external_plan(): module = parse_pyi_text( """ -from x2py.contracts import Float64, Int32, external +from x2py.contracts import Annotated, Float64, Immutable, Int32, external @external -def late_extent(values: Float64[n], n: Int32) -> None: ... +def late_extent(values: Float64[n], n: Annotated[Int32, Immutable] | None = ...) -> None: ... """, module_name="late_extent_external", ) @@ -154,8 +154,9 @@ def test_dense_array_lowering_uses_planned_shape_checks_and_bridge_orientation() assert "subroutine bind_c_flat(n, bound_values, values_extent_0)" in bridge_source assert "subroutine bind_c_flat_rank2_runtime(" in bridge_source assert "subroutine bind_c_c_flat_rank2_runtime(" in bridge_source - assert "real(c_double) :: values(*)" in bridge_source - assert "real(c_double) :: values(3, *)" in bridge_source + assert "external :: flat_rank2_runtime" in bridge_source + assert "external :: c_flat_rank2_fixed" in bridge_source + assert "real(c_double), pointer, dimension(:, :) :: values" in bridge_source def test_external_interface_declares_late_extent_before_dependent_array(): @@ -164,7 +165,9 @@ def test_external_interface_declares_late_extent_before_dependent_array(): signature = "subroutine late_extent(values, n)" interface = bridge_source.split(signature, maxsplit=1)[1].split("end subroutine late_extent", maxsplit=1)[0] - assert interface.index("integer(c_int32_t) :: n") < interface.index("real(c_double), dimension(n) :: values") + assert interface.index("integer(c_int32_t), optional :: n") < interface.index( + "real(c_double), dimension(n) :: values" + ) def test_unavailable_dense_extent_role_fails_before_backend_lowering(): diff --git a/x2py/cli.py b/x2py/cli.py index d150f63ba..8cd476eca 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -856,7 +856,7 @@ def _format_semantic_blocker_item(code: str, item) -> str: return f"{item['owner']} needs literal value for Final constant {item['symbol']}" if code == "callback_signature_incomplete": needs = ", ".join(item.get("needs") or []) - return f"{item['owner']} needs Callable[[...], ...] metadata ({needs})" + return f"{item['owner']} needs a complete named @prototype ({needs})" if code.startswith("c_"): owner = item.get("owner", "") detail = item.get("type") or item.get("source") or item.get("function") or item.get("parameter") diff --git a/x2py/contracts/__init__.py b/x2py/contracts/__init__.py index 1b2f35a0a..449246d16 100644 --- a/x2py/contracts/__init__.py +++ b/x2py/contracts/__init__.py @@ -7,7 +7,6 @@ from __future__ import annotations -from collections.abc import Callable as Callable from typing import Annotated as Annotated, Any as Any, Final as Final @@ -103,15 +102,11 @@ def apply(target): Bounded = _expression Destruction = _expression Finite = _expression -In = _expression -InOut = _expression IsPresent = _expression Len = _expression Name = _expression -Out = _expression Ownership = _expression Pass = _expression -PassByRef = _expression PointerAssociation = _expression PointerPolicy = _expression Return = _expression @@ -126,6 +121,7 @@ def apply(target): native_call = _decorator native_type = _decorator overload = _decorator +prototype = _decorator raises = _decorator CAnonymous = _ContractType @@ -155,7 +151,6 @@ def apply(target): "CEnum", "CStruct", "CUnion", - "Callable", "Char", "Complex64", "Complex128", @@ -172,8 +167,6 @@ def apply(target): "Float128", "FortranAllocatable", "Immutable", - "In", - "InOut", "Int", "Int8", "Int16", @@ -188,10 +181,8 @@ def apply(target): "ORDER_ANY", "ORDER_C", "ORDER_F", - "Out", "Ownership", "Pass", - "PassByRef", "Pointer", "PointerAssociation", "PointerPolicy", @@ -219,6 +210,7 @@ def apply(target): "native_call", "native_type", "overload", + "prototype", "private", "raises", } @@ -237,7 +229,6 @@ def apply(target): "CEnum", "CStruct", "CUnion", - "Callable", "Char", "Complex64", "Complex128", diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 61f898890..fdb703e8e 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -32,6 +32,7 @@ SemanticFunction, SemanticImport, SemanticModule, + SemanticPrototype, SemanticVariable, ) from x2py.semantics.native_contract import NATIVE_CONTRACT_PREPARED_METADATA, validate_pyi_native_contract @@ -787,6 +788,14 @@ def _pyi_export_tree( origin=path, ) + for prototype in module.prototypes: + _merge_export_child( + tree, + prototype.name, + _PyiExportNode(declarations=[prototype], origins={path}), + origin=path, + ) + for semantic_import in module.imports: if not isinstance(semantic_import, SemanticImport) or not semantic_import.module.startswith("."): continue @@ -859,6 +868,8 @@ def _merge_export_child(tree: _PyiExportNode, name: str, child: _PyiExportNode, def _record_pyi_exports(tree: _PyiExportNode, namespace: tuple[str, ...] = ()) -> None: for name, child in tree.children.items(): for declaration in child.declarations: + if isinstance(declaration, SemanticPrototype): + continue exports = _declaration_exports(declaration) export = {"namespace": namespace, "name": name} if export not in exports: @@ -1432,6 +1443,7 @@ def _merge_wrapper_modules(modules: list[SemanticModule], *, name: str | None = return SemanticModule( name=name or modules[0].name, functions=[function for module in modules for function in module.functions], + prototypes=[prototype for module in modules for prototype in module.prototypes], overload_sets=[overload for module in modules for overload in module.overload_sets], classes=[semantic_class for module in modules for semantic_class in module.classes], variables=[variable for module in modules for variable in module.variables], diff --git a/x2py/semantics/README.md b/x2py/semantics/README.md index a0dd689b4..aaa23d3db 100644 --- a/x2py/semantics/README.md +++ b/x2py/semantics/README.md @@ -45,7 +45,7 @@ The native barrier says how the bridge presents the extracted value to native code: direct value, call-local address, caller/Python-backed storage address, raw address, packed array descriptor, or wrapper-owned native address. -Policy completion also validates the boundary spelling. Callable `Addr(T)` is +Policy completion also validates the boundary spelling. Procedure `Addr(T)` is an integer raw-address contract and is limited to primitive scalars, fixed-length strings, and primitive arrays with fully resolved extents. `Addr(Arg(i))` is limited to primitive scalar values that need call-local diff --git a/x2py/semantics/fortran2ir.py b/x2py/semantics/fortran2ir.py index 0a5c3ab48..984028945 100644 --- a/x2py/semantics/fortran2ir.py +++ b/x2py/semantics/fortran2ir.py @@ -27,7 +27,6 @@ from x2py.utilities.visitor import ClassVisitor from .models import ( - CALLBACK_DECLARATION_ACCESS_METADATA, EXTERNAL_TYPE_REF_METADATA, FORTRAN_GENERIC_NAME_METADATA, OVERLOAD_KIND_METADATA, @@ -36,6 +35,7 @@ PYTHON_BOUND_POSITION_METADATA, PYTHON_METHOD_NAME_METADATA, PYTHON_STATIC_METADATA, + PROTOTYPE_REF_METADATA, SemanticArgument, SemanticArrayContract, SemanticClass, @@ -47,6 +47,7 @@ SemanticMethod, SemanticModule, SemanticOrigin, + SemanticPrototype, SemanticStorageContract, SemanticType, SemanticVariable, @@ -386,7 +387,7 @@ def _visit_FortranArgument( else: semantic_type = self._convert_variable_type(arg, derived_type_context=derived_type_context) access = self._argument_access(arg, semantic_type) - if semantic_type.name == "Callable": + if semantic_type.storage is not None and semantic_type.storage.kind == "callback": pass elif semantic_type.rank > 0: self._apply_array_argument_contract(semantic_type, arg, writes_argument=access[1]) @@ -484,8 +485,8 @@ def _callback_semantic_type( ) -> SemanticType: if getattr(arg, "pointer", False): return self._convert_variable_type(arg, derived_type_context=derived_type_context) - interface_name = str(arg.kind or arg.name).casefold() - signature = callback_interfaces.get(interface_name) + interface_name = str(arg.kind or arg.name) + signature = callback_interfaces.get(interface_name.casefold()) if signature is None: return self._convert_variable_type(arg, derived_type_context=derived_type_context) @@ -493,26 +494,30 @@ def _callback_semantic_type( projected_arguments = list(signature.arguments) callback_arguments = [self.visit(item, derived_type_context=context) for item in projected_arguments] for source_argument, callback_argument in zip(projected_arguments, callback_arguments, strict=True): - access = self._callback_declaration_access(source_argument) - callback_argument.metadata[CALLBACK_DECLARATION_ACCESS_METADATA] = access - self._normalize_callback_character_storage(callback_argument, source_argument, access) + self._normalize_callback_reference_storage(callback_argument, source_argument) callback_return = ( self.visit(signature.result, derived_type_context=context, as_type=True) if signature.result else SemanticType("None", dtype="None") ) + prototype_module = str(signature.module or "") return SemanticType( - "Callable", - dtype="Callable", + interface_name, + dtype="Prototype", metadata={ "arguments": [item.semantic_type for item in callback_arguments], "callback_arguments": callback_arguments, "return": callback_return, - "fortran_callback_interface": signature.name, - "fortran_callback_kind": signature.kind, + PROTOTYPE_REF_METADATA: { + "name": interface_name, + "local_name": interface_name, + "origin_module": prototype_module, + }, + "native_callback_kind": signature.kind, "callback_lifetime": "call", "callback_thread": "entering_thread", "callback_exception": "print_traceback_and_abort", + "prototype_metadata": self._procedure_metadata(signature), }, storage=SemanticStorageContract( kind="callback", @@ -525,49 +530,85 @@ def _callback_semantic_type( native_scope=getattr(arg, "procedure", None), source_kind="dummy_procedure", source_type=self._fortran_source_type(arg), - metadata={"interface": signature.name}, + metadata={"prototype": interface_name}, ), ) @staticmethod - def _callback_declaration_access(arg: FortranArgument | FortranVariable) -> str: - """Return exact callback declaration access for Fortran adapter printing.""" - match getattr(arg, "intent", None): - case "in": - return "read" - case "out": - return "write" - case "inout": - return "readwrite" - case _: - return "unspecified" - - @staticmethod - def _normalize_callback_character_storage( + def _normalize_callback_reference_storage( callback_argument: SemanticArgument, source_argument: FortranArgument | FortranVariable, - access: str, ) -> None: - """Use mutable scalar bytes storage for writable callback character dummies.""" - semantic_type = callback_argument.semantic_type - if semantic_type.name != "String" or semantic_type.rank != 0: - return + """Make every non-value callback dummy a permissive reference contract.""" if getattr(source_argument, "pass_by_value", False): return - if access == "read": - return - semantic_type.storage = SemanticStorageContract( - kind="array", - read_only=False, - mutable=True, - array=SemanticArrayContract( - rank=0, - shape=[], - category=SCALAR_STORAGE_CATEGORY, - ), - ) + semantic_type = callback_argument.semantic_type + if semantic_type.name == "String" and semantic_type.rank == 0: + semantic_type.storage = SemanticStorageContract( + kind="array", + read_only=False, + mutable=True, + array=SemanticArrayContract( + rank=0, + shape=[], + category=SCALAR_STORAGE_CATEGORY, + ), + ) + elif semantic_type.storage is None: + semantic_type.storage = SemanticStorageContract( + kind="reference", + read_only=False, + mutable=True, + pointer_depth=1, + ) + else: + semantic_type.storage.read_only = False + semantic_type.storage.mutable = True semantic_type.ownership.mutable = True + def _module_prototypes( + self, + module: FortranModule, + context: _DerivedTypeContext, + referenced: set[str], + ) -> list[SemanticPrototype]: + """Convert abstract and callback-local interfaces into semantic prototypes.""" + prototypes: list[SemanticPrototype] = [] + seen: set[str] = set() + for interface in module.interfaces: + for signature in interface.procedures: + name = interface.name if interface.name and len(interface.procedures) == 1 else signature.name + if not interface.abstract and name.casefold() not in referenced: + continue + if name in seen: + continue + seen.add(name) + arguments = [self.visit(item, derived_type_context=context) for item in signature.arguments] + for source_argument, argument in zip(signature.arguments, arguments, strict=True): + self._normalize_callback_reference_storage(argument, source_argument) + return_type = ( + self.visit(signature.result, derived_type_context=context, as_type=True) + if signature.result is not None + else SemanticType("None", dtype="None") + ) + prototypes.append( + SemanticPrototype( + name=name, + native_name=name, + arguments=arguments, + return_type=return_type, + metadata=self._procedure_metadata(signature), + visibility=self._symbol_visibility(module, name), + origin=SemanticOrigin( + source_language="fortran", + native_name=name, + native_scope=module.name, + source_kind="prototype", + ), + ) + ) + return prototypes + @staticmethod def _visit_FortranEnumerator(enumerator: FortranEnumerator, *, enum: FortranEnum) -> SemanticVariable: semantic_type = SemanticType( @@ -732,6 +773,13 @@ def _visit_FortranModule( ) for proc in module.procedures ] + callback_prototypes = { + argument.semantic_type.name.casefold() + for function in semantic_functions + for argument in function.arguments + if argument.semantic_type.storage is not None and argument.semantic_type.storage.kind == "callback" + } + prototypes = self._module_prototypes(module, context, callback_prototypes) procedure_lookup = {func.name.casefold(): func for func in semantic_functions} semantic_classes = [ @@ -771,6 +819,7 @@ def _visit_FortranModule( return SemanticModule( name=module.name, functions=semantic_functions, + prototypes=prototypes, overload_sets=overload_sets, classes=semantic_classes, variables=module_variables + enum_constants, diff --git a/x2py/semantics/models.py b/x2py/semantics/models.py index 01a03e1e7..f0319af0e 100644 --- a/x2py/semantics/models.py +++ b/x2py/semantics/models.py @@ -6,7 +6,7 @@ EXTERNAL_TYPE_REF_METADATA = "external_type_ref" -CALLBACK_DECLARATION_ACCESS_METADATA = "callback_declaration_access" +PROTOTYPE_REF_METADATA = "prototype_ref" INTERNAL_MODULE_VARIABLE_ACCESS_METADATA = "internal_module_variable_access" INTERNAL_MODULE_VARIABLE_NAME_METADATA = "internal_module_variable_name" INTERNAL_NATIVE_ARRAY_HANDLE_OPERATION_METADATA = "internal_native_array_handle_operation" @@ -292,6 +292,16 @@ def __eq__(self, other: object) -> bool: ) +# ============================================================ +# Named Procedure Prototypes +# ============================================================ + + +@dataclass(eq=False) +class SemanticPrototype(SemanticFunction): + """Named native callback signature with no runtime Python export.""" + + # ============================================================ # Semantic Methods # ============================================================ @@ -572,6 +582,8 @@ class SemanticModule: functions: list[SemanticFunction] = field(default_factory=list) + prototypes: list[SemanticPrototype] = field(default_factory=list) + overload_sets: list[ProcedureOverloadSet] = field(default_factory=list) classes: list[SemanticClass] = field(default_factory=list) @@ -588,7 +600,7 @@ def _iter_semantic_type_tree(semantic_type: SemanticType | None): if semantic_type is None: return yield semantic_type - if semantic_type.name == "Callable": + if semantic_type.storage is not None and semantic_type.storage.kind == "callback": arguments = semantic_type.metadata.get("arguments") if isinstance(arguments, list): for argument in arguments: @@ -620,6 +632,10 @@ def iter_class(declaration: SemanticClass): for argument in function.arguments: yield from _iter_semantic_type_tree(argument.semantic_type) yield from _iter_semantic_type_tree(function.return_type) + for prototype in module.prototypes: + for argument in prototype.arguments: + yield from _iter_semantic_type_tree(argument.semantic_type) + yield from _iter_semantic_type_tree(prototype.return_type) for overload_set in module.overload_sets: for procedure in overload_set.procedures: for argument in procedure.arguments: diff --git a/x2py/semantics/native_contract.py b/x2py/semantics/native_contract.py index a621966f5..68775e478 100644 --- a/x2py/semantics/native_contract.py +++ b/x2py/semantics/native_contract.py @@ -13,6 +13,7 @@ SemanticFunction, SemanticMethod, SemanticModule, + SemanticPrototype, SemanticType, _iter_module_semantic_types, ) @@ -47,6 +48,8 @@ def _prepare_module(module: SemanticModule) -> None: _set_origin(variable, native_scope, "variable") for function in module.functions: _prepare_function(function, native_scope) + for prototype in module.prototypes: + _prepare_prototype(prototype, native_scope) for overload_set in module.overload_sets: for procedure in overload_set.procedures: _prepare_function(procedure, native_scope) @@ -73,6 +76,13 @@ def _prepare_function(function: SemanticFunction, native_scope: str) -> None: _set_origin(argument, function.origin.native_name, "argument") +def _prepare_prototype(prototype: SemanticPrototype, native_scope: str) -> None: + _set_origin(prototype, native_scope, "prototype") + prototype.origin.native_name = prototype.native_name or prototype.name + for argument in prototype.arguments: + _set_origin(argument, prototype.origin.native_name, "prototype_argument") + + def _prepare_class(semantic_class: SemanticClass, native_scope: str) -> None: _set_origin(semantic_class, native_scope, "derived_type") for field in semantic_class.fields: @@ -103,6 +113,11 @@ def native_contract_issues(module: SemanticModule) -> list[NativeContractIssue]: issues.extend(_type_issues(variable.semantic_type, f"{module.name}.{variable.name}")) for function in module.functions: issues.extend(_function_issues(function, module, owner_kind="module")) + for prototype in module.prototypes: + for argument in prototype.arguments: + issues.extend(_type_issues(argument.semantic_type, f"{module.name}.{prototype.name}.{argument.name}")) + if prototype.return_type is not None and prototype.return_type.name != "None": + issues.extend(_type_issues(prototype.return_type, f"{module.name}.{prototype.name}.return")) for overload_set in module.overload_sets: for procedure in overload_set.procedures: issues.extend(_function_issues(procedure, module, owner_kind="module")) @@ -235,15 +250,15 @@ def _type_issues(semantic_type: SemanticType, owner: str) -> list[NativeContract owner, ) ] - if semantic_type.name != "Callable": + if semantic_type.storage is None or semantic_type.storage.kind != "callback": return [] arguments = semantic_type.metadata.get("arguments") - kind = semantic_type.metadata.get("fortran_callback_kind") + kind = semantic_type.metadata.get("native_callback_kind") if not isinstance(arguments, list) or kind not in {"function", "subroutine"}: return [ NativeContractIssue( "pyi_native_callback_incomplete", - "Native callbacks require a complete Callable argument and return signature.", + "Native callbacks require a resolved named prototype.", owner, ) ] diff --git a/x2py/semantics/policy_completion.py b/x2py/semantics/policy_completion.py index 94db67287..78efd4fe5 100644 --- a/x2py/semantics/policy_completion.py +++ b/x2py/semantics/policy_completion.py @@ -1532,7 +1532,7 @@ def _complete_variable( ) -> None: decision = default_ownership_policy.decide_semantic_variable(variable, context) variable.metadata[models.RESOLVED_OWNERSHIP_POLICY_METADATA] = decision - _complete_callable_policy(variable.semantic_type, owner_path=owner_path or variable.name) + _complete_prototype_reference_policy(variable.semantic_type, owner_path=owner_path or variable.name) _complete_native_array_handle_variable_policy(variable, context) @@ -1609,12 +1609,12 @@ def _is_constant(variable: models.SemanticVariable) -> bool: return any(constraint.name == "Constant" for constraint in variable.semantic_type.constraints) -def _complete_callable_policy( +def _complete_prototype_reference_policy( semantic_type: models.SemanticType, *, owner_path: str, ) -> None: - if semantic_type.name != "Callable": + if semantic_type.storage is None or semantic_type.storage.kind != "callback": return visible_scalar_names: set[str] = set() @@ -1630,7 +1630,7 @@ def _complete_callable_policy( _validate_callback_argument_contract(argument) _validate_raw_address_type( argument.semantic_type, - owner="Callable", + owner="prototype", item=argument.name, visible_scalar_names=visible_scalar_names, ) @@ -1644,7 +1644,7 @@ def _complete_callable_policy( if isinstance(return_type, models.SemanticType) and return_type.name != "None": _validate_raw_address_type( return_type, - owner="Callable", + owner="prototype", item="return", visible_scalar_names=visible_scalar_names, ) @@ -1659,31 +1659,18 @@ def _complete_callable_policy( def _callback_argument_ownership_context(argument: models.SemanticArgument) -> OwnershipContext: if bool(getattr(argument.origin, "metadata", {}).get("value")): return OwnershipContext.argument(reads_argument=True, writes_argument=False) - access = argument.metadata.get(models.CALLBACK_DECLARATION_ACCESS_METADATA, "read") - match access: - case "write": - return OwnershipContext.argument(reads_argument=False, writes_argument=True) - case "readwrite" | "unspecified": - return OwnershipContext.argument(reads_argument=True, writes_argument=True) - case _: - return OwnershipContext.argument(reads_argument=True, writes_argument=False) + return OwnershipContext.argument(reads_argument=True, writes_argument=True) def _validate_callback_argument_contract(argument: models.SemanticArgument) -> None: semantic_type = argument.semantic_type if semantic_type.name != "String": return - access = argument.metadata.get(models.CALLBACK_DECLARATION_ACCESS_METADATA, "read") - if access == "read": - return if bool(getattr(argument.origin, "metadata", {}).get("value")): return if _is_scalar_string_storage(semantic_type): return - raise ValueError( - "Writable callback strings require mutable scalar character storage; " - "use String[n][()] inside Out(...), InOut(...), or missing-intent callback arguments" - ) + raise ValueError("Reference callback strings require mutable scalar character storage") def _is_scalar_string_storage(semantic_type: models.SemanticType) -> bool: diff --git a/x2py/semantics/pyi2ir.py b/x2py/semantics/pyi2ir.py index 4fee82ce9..7497ca725 100644 --- a/x2py/semantics/pyi2ir.py +++ b/x2py/semantics/pyi2ir.py @@ -24,7 +24,6 @@ from x2py.utilities.visitor import ClassVisitor from .models import ( - CALLBACK_DECLARATION_ACCESS_METADATA, EXTERNAL_TYPE_REF_METADATA, FORTRAN_GENERIC_NAME_METADATA, OVERLOAD_KIND_METADATA, @@ -35,6 +34,7 @@ PYTHON_STATIC_METADATA, PYTHON_VALUE_IMMUTABLE, PYTHON_VALUE_MUTABILITY_METADATA, + PROTOTYPE_REF_METADATA, RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA, ProjectionMapping, @@ -50,6 +50,7 @@ SemanticMethod, SemanticModule, SemanticOrigin, + SemanticPrototype, SemanticStorageContract, SemanticType, SemanticVariable, @@ -68,7 +69,6 @@ @dataclass(frozen=True) class _CallbackArgumentSpec: semantic_type: SemanticType - access: str passes_by_value: bool @@ -96,6 +96,7 @@ class _Decorators: is_static: bool = False hold_gil: bool = False error_status_policy: dict[str, object] | None = None + prototype: bool = False @dataclass @@ -118,8 +119,30 @@ def parse(self, tree: ast.Module) -> SemanticModule: _ModuleVisitor(self)._visit(tree) self._resolve_overloads() self._restore_type_bound_targets() + self._resolve_local_prototype_references() return self.module + def _resolve_local_prototype_references(self) -> None: + """Bind local prototype annotations after all declarations are known.""" + prototypes = {prototype.name: prototype for prototype in self.module.prototypes} + if len(prototypes) != len(self.module.prototypes): + raise ValueError("Prototype names must be unique within a semantic module") + runtime_names = {item.name for item in [*self.module.functions, *self.module.classes, *self.module.variables]} + collisions = sorted(runtime_names & prototypes.keys()) + if collisions: + raise ValueError(f"Prototype name collides with a runtime declaration: {collisions[0]!r}") + for semantic_type in _iter_module_semantic_types(self.module): + if semantic_type.storage is not None and semantic_type.storage.kind == "callback": + continue + prototype = prototypes.get(semantic_type.name) + if prototype is not None: + _bind_prototype_reference( + semantic_type, + prototype, + origin_module=self.module.name, + source_name=prototype.name, + ) + def import_from(self, node: ast.ImportFrom) -> SemanticImport: module_name = "." * node.level + (node.module or "") return SemanticImport( @@ -303,6 +326,41 @@ def function_def( origin=origin, ) + def prototype_def(self, node: ast.FunctionDef, *, visibility: str) -> SemanticPrototype: + """Convert one named callback prototype without creating a runtime function.""" + self._validate_callable_header(node) + arguments = [] + for argument, default in zip(node.args.args, self._argument_defaults(node), strict=False): + if argument.annotation is None: + raise ValueError(f"Expected typed prototype argument: {argument.arg!r}") + spec = self._prototype_argument_spec(argument.annotation) + arguments.append( + SemanticArgument( + argument.arg, + spec.semantic_type, + optional=self.default_marks_optional(default), + visibility=visibility, + origin=SemanticOrigin(metadata={"value": spec.passes_by_value}), + ) + ) + return_type = ( + SemanticType("None", dtype="None") + if isinstance(node.returns, ast.Constant) and node.returns.value is None + else self.semantic_type(node.returns) + ) + return SemanticPrototype( + name=node.name, + native_name=node.name, + arguments=arguments, + return_type=return_type, + visibility=visibility, + origin=SemanticOrigin( + native_name=node.name, + native_scope=self.module.name, + source_kind="prototype", + ), + ) + def method_def( self, node: ast.FunctionDef, @@ -419,6 +477,8 @@ def decorators(self, nodes: list[ast.expr], *, context: str) -> _Decorators: raise ValueError("bind cannot be combined with overload") if parsed.overload_target is not None and parsed.has_native_call: raise ValueError("overload cannot be combined with native_call; put native_call on the specific procedure") + if parsed.prototype and len(nodes) != 1: + raise ValueError("prototype cannot be combined with other decorators") return parsed def _apply_decorator(self, parsed: _Decorators, node: ast.expr, *, context: str) -> None: @@ -436,6 +496,7 @@ def _apply_decorator(self, parsed: _Decorators, node: ast.expr, *, context: str) "hold_gil": self._apply_hold_gil_decorator, "native_call": self._apply_native_call_decorator, "native_type": self._apply_native_type_decorator, + "prototype": self._apply_prototype_decorator, "raises": self._apply_raises_decorator, } handler = next((value for name, value in handlers.items() if self.matches_name(target, name)), None) @@ -443,6 +504,16 @@ def _apply_decorator(self, parsed: _Decorators, node: ast.expr, *, context: str) raise ValueError(f"Unsupported {context} decorator: {ast.unparse(node)!r}") handler(parsed, node, context) + @staticmethod + def _apply_prototype_decorator(parsed: _Decorators, node: ast.expr, context: str) -> None: + if isinstance(node, ast.Call): + raise ValueError("prototype does not accept arguments") + if context != ".pyi": + raise ValueError("prototype is only valid for module-level declarations") + if parsed.prototype: + raise ValueError("Duplicate prototype decorator") + parsed.prototype = True + def _apply_overload_decorator(self, parsed: _Decorators, node: ast.expr, context: str) -> None: if not isinstance(node, ast.Call): raise ValueError("overload expects one specific procedure name") @@ -1218,8 +1289,6 @@ def semantic_type(self, node: ast.expr) -> SemanticType: return semantic_type if self.is_subscript_of(node, "Final"): return self._final_type(node) - if self.matches_name(node, "Callable") or self.is_subscript_of(node, "Callable"): - return self.callable_type(node) if isinstance(node, ast.Call) and self._is_addr_call(node): return self._address_type(node) if isinstance(node, ast.Call): @@ -1827,130 +1896,46 @@ def slice_text(self, node: ast.Slice) -> str: return f"{lower}:{upper}:{step}" return f"{lower}:{upper}" - def callable_type(self, node: ast.expr) -> SemanticType: - if not isinstance(node, ast.Subscript): - return SemanticType(name="Callable", dtype="Callable") - - items = self.subscript_items(node) - if len(items) != 2: - raise ValueError(f"Callable expects argument types and a return type: {ast.unparse(node)!r}") - - raw_args, raw_return = items - if isinstance(raw_args, ast.Constant) and raw_args.value is Ellipsis: - return SemanticType( - name="Callable", - dtype="Callable", - metadata=self._callback_metadata(None, self.semantic_type(raw_return)), - storage=self._callback_storage(), - ) - if not isinstance(raw_args, ast.List): - raise ValueError(f"Callable arguments must be a list: {ast.unparse(node)!r}") - - argument_specs = [self._callback_argument_spec(item) for item in raw_args.elts] - argument_types = [item.semantic_type for item in argument_specs] - return_type = self.semantic_type(raw_return) - metadata = self._callback_metadata(argument_types, return_type) - metadata["callback_arguments"] = self._callback_arguments(argument_specs, return_type) - return SemanticType( - name="Callable", - dtype="Callable", - metadata=metadata, - storage=self._callback_storage(), - ) - - def _callback_argument_spec(self, node: ast.expr) -> _CallbackArgumentSpec: + def _prototype_argument_spec(self, node: ast.expr) -> _CallbackArgumentSpec: if isinstance(node, ast.Call): - wrapper = self._callback_reference_wrapper_access(node) - if wrapper is not None: + wrapper = self.contract_name(node.func) + if wrapper == "Value": if len(node.args) != 1 or node.keywords: - raise ValueError(f"{self.required_name(node.func)} expects one callback argument type") - if isinstance(node.args[0], ast.Call) and self._is_addr_call(node.args[0]): - raise ValueError( - "Addr(...) is not valid inside Callable callback signatures; " - f"{self.required_name(node.func)}(...) already describes callback reference passing" - ) + raise ValueError("Value expects one callback argument type") semantic_type = self.semantic_type(node.args[0]) - self._mark_callback_reference_type(semantic_type, wrapper) - return _CallbackArgumentSpec(semantic_type, wrapper, False) + if semantic_type.rank > 0: + raise ValueError("Value(...) callback arguments must be scalar") + return _CallbackArgumentSpec(semantic_type, True) if self._is_addr_call(node): raise ValueError( - "Addr(...) is not valid inside Callable callback signatures; " - "use PassByRef(...) for scalar reference callbacks without intent" + "Addr(...) is unnecessary inside prototype declarations; reference passing is the default" ) semantic_type = self.semantic_type(node) - passes_by_value = semantic_type.storage is None - if not passes_by_value: - self._mark_callback_reference_type(semantic_type, "unspecified") - return _CallbackArgumentSpec(semantic_type, "unspecified", passes_by_value) - - def _callback_reference_wrapper_access(self, node: ast.Call) -> str | None: - wrapper = self.contract_name(node.func) - if wrapper is None: - return None - return { - "PassByRef": "unspecified", - "In": "read", - "Out": "write", - "InOut": "readwrite", - }.get(wrapper) + self._mark_callback_reference_type(semantic_type) + return _CallbackArgumentSpec(semantic_type, False) @staticmethod - def _mark_callback_reference_type(semantic_type: SemanticType, access: str) -> None: - read_only = access == "read" - mutable = not read_only + def _mark_callback_reference_type(semantic_type: SemanticType) -> None: storage = semantic_type.storage - if storage is None: + if semantic_type.name == "String" and semantic_type.rank == 0: + semantic_type.storage = SemanticStorageContract( + kind="array", + read_only=False, + mutable=True, + array=SemanticArrayContract(rank=0, shape=[], category=SCALAR_STORAGE_CATEGORY), + ) + elif storage is None: semantic_type.storage = SemanticStorageContract( kind="reference", - read_only=read_only, - mutable=mutable, + read_only=False, + mutable=True, pointer_depth=1, ) else: - storage.read_only = read_only - storage.mutable = mutable - semantic_type.ownership.mutable = mutable - - @classmethod - def _callback_arguments( - cls, - argument_specs: list[_CallbackArgumentSpec], - return_type: SemanticType, - ) -> list[SemanticArgument]: - argument_types = [item.semantic_type for item in argument_specs] - shape_names = cls._callback_shape_names([*argument_types, return_type]) - used_names: set[str] = set() - arguments = [] - for index, spec in enumerate(argument_specs): - semantic_type = spec.semantic_type - name = f"arg_{index}" - if cls._is_dimension_scalar_callback_type(semantic_type): - inferred_name = next((item for item in shape_names if item not in used_names), None) - if inferred_name is not None: - name = inferred_name - used_names.add(inferred_name) - arguments.append( - SemanticArgument( - name, - semantic_type, - metadata={CALLBACK_DECLARATION_ACCESS_METADATA: spec.access}, - origin=SemanticOrigin(metadata={"value": spec.passes_by_value}), - ) - ) - return arguments - - @classmethod - def _callback_shape_names(cls, semantic_types: list[SemanticType]) -> list[str]: - names = [] - for semantic_type in semantic_types: - for dimension, strided in cls._semantic_shape_dimensions(semantic_type): - if strided: - dimension = re.sub(r"(?i)(?<=:)Strided\s*$", "", str(dimension)) - for name in re.findall(r"\b[A-Za-z_]\w*\b", str(dimension)): - if name not in names: - names.append(name) - return names + storage.read_only = False + storage.mutable = True + semantic_type.ownership.mutable = True @staticmethod def _semantic_shape_dimensions(semantic_type: SemanticType) -> list[tuple[str, bool]]: @@ -1964,10 +1949,6 @@ def _semantic_shape_dimensions(semantic_type: SemanticType) -> list[tuple[str, b axes = ["dense"] * len(dimensions) return [(str(dimension), axis == "strided") for dimension, axis in zip(dimensions, axes, strict=True)] - @staticmethod - def _is_dimension_scalar_callback_type(semantic_type: SemanticType) -> bool: - return semantic_type.rank == 0 and str(semantic_type.name).startswith("Int") - @staticmethod def _callback_metadata(arguments: list[SemanticType] | None, return_type: SemanticType) -> dict[str, object]: return { @@ -2233,7 +2214,7 @@ def _validate_callable_descriptor_arguments( continue if self._semantic_scalar_descriptor_kind(argument.semantic_type) is not None: raise ValueError( - "Callable scalar descriptors use nullable value annotations plus " + "Procedure scalar descriptors use nullable value annotations plus " "Allocatable(Arg(i)) or Pointer(Arg(i)) in native_call" ) @@ -2265,7 +2246,7 @@ def _validate_callable_descriptor_return( """Reject descriptor type wrappers on callable Python return annotations.""" if native_result is None and self._semantic_scalar_descriptor_kind(return_type) is not None: raise ValueError( - "Callable scalar descriptor results use a nullable value annotation plus " + "Procedure scalar descriptor results use a nullable value annotation plus " "native_call result=Allocatable(Return(0)) or result=Pointer(Return(0))" ) @@ -2767,6 +2748,9 @@ def _visit_FunctionDef(self, node: ast.FunctionDef) -> None: decorators = self.parser.decorators(node.decorator_list, context=".pyi") if decorators.native_type is not None: raise ValueError("native_type is only valid for classes") + if decorators.prototype: + self.parser.module.prototypes.append(self.parser.prototype_def(node, visibility=decorators.visibility)) + return function = self.parser.function_def( node, visibility=decorators.visibility, @@ -2855,13 +2839,77 @@ def _relative_imported_namespace(module_name: str, source_name: str) -> str: return f"{module_path}.{source_name}" +def _bind_prototype_reference( + semantic_type: SemanticType, + prototype: SemanticPrototype, + *, + origin_module: str, + source_name: str, +) -> None: + """Complete one type annotation as a named callback prototype reference.""" + local_name = semantic_type.name + arguments = deepcopy(prototype.arguments) + return_type = deepcopy(prototype.return_type) or SemanticType("None", dtype="None") + semantic_type.dtype = "Prototype" + semantic_type.metadata = { + "arguments": [argument.semantic_type for argument in arguments], + "callback_arguments": arguments, + "return": return_type, + "callback_lifetime": "call", + "callback_thread": "entering_thread", + "callback_exception": "print_traceback_and_abort", + "prototype_metadata": deepcopy(prototype.metadata), + "native_callback_kind": "subroutine" if return_type.name == "None" else "function", + PROTOTYPE_REF_METADATA: { + "name": source_name, + "local_name": local_name, + "origin_module": origin_module, + }, + } + semantic_type.storage = SemanticStorageContract( + kind="callback", + ownership="borrowed", + calling_convention="native_dummy_procedure", + ) + semantic_type.origin = SemanticOrigin( + native_name=source_name, + native_scope=origin_module, + source_kind="prototype_reference", + ) + + def reconcile_external_type_refs(modules: list[SemanticModule]) -> list[SemanticModule]: definitions = {(module.name, declaration.name): declaration for module in modules for declaration in module.classes} + prototypes = {(module.name, prototype.name): prototype for module in modules for prototype in module.prototypes} for module in modules: for semantic_type in _iter_module_semantic_types(module): ref = semantic_type.metadata.get(EXTERNAL_TYPE_REF_METADATA) if not isinstance(ref, dict): continue + origin_module = ref.get("origin_module") + source_name = ref.get("name") + if isinstance(origin_module, str) and isinstance(source_name, str): + module_candidates = ( + origin_module, + origin_module.lstrip("."), + origin_module.lstrip(".").rsplit(".", 1)[-1], + ) + prototype = next( + ( + candidate_prototype + for candidate in module_candidates + if candidate and (candidate_prototype := prototypes.get((candidate, source_name))) is not None + ), + None, + ) + if prototype is not None: + _bind_prototype_reference( + semantic_type, + prototype, + origin_module=str(prototype.origin.native_scope or origin_module.lstrip(".")), + source_name=source_name, + ) + continue declaration = definitions.get((ref.get("origin_module"), ref.get("name"))) wrapped = declaration is not None and ( not isinstance(declaration, SemanticClass) or "Opaque" not in declaration.base_classes diff --git a/x2py/semantics/readiness.py b/x2py/semantics/readiness.py index 06c1f9ae7..07ba98faa 100644 --- a/x2py/semantics/readiness.py +++ b/x2py/semantics/readiness.py @@ -36,7 +36,6 @@ { "Any", "Bool", - "Callable", "Complex64", "Complex128", "Float32", @@ -619,8 +618,8 @@ def _check_type( self._add_callback_blocker(type_name, owner, item, unit=unit, unit_kind=unit_kind) return - if type_name == "Callable": - self._check_callable_type( + if semantic_type.storage is not None and semantic_type.storage.kind == "callback": + self._check_prototype_reference( semantic_type, owner=owner, item=item, @@ -630,6 +629,7 @@ def _check_type( unit=unit, unit_kind=unit_kind, ) + return if not self.index.is_known_type(type_name, module) and not _is_external_type_ref(semantic_type): self._add_blocker( @@ -951,7 +951,7 @@ def _has_known_iso_c_kind(semantic_type: SemanticType) -> bool: source_type = (semantic_type.origin.source_type or "").casefold() return any(token in source_type for token in _ISO_C_KIND_TOKENS) - def _check_callable_type( + def _check_prototype_reference( self, semantic_type: SemanticType, *, @@ -966,7 +966,7 @@ def _check_callable_type( arguments = semantic_type.metadata.get("arguments") return_type = semantic_type.metadata.get("return") if not isinstance(arguments, list) or return_type is None: - self._add_callback_blocker("Callable", owner, item, unit=unit, unit_kind=unit_kind) + self._add_callback_blocker(semantic_type.name, owner, item, unit=unit, unit_kind=unit_kind) return for index, callback_arg in enumerate(arguments): @@ -1075,7 +1075,7 @@ def _add_callback_blocker( ) -> None: self._add_blocker( "callback_signature_incomplete", - "Some callback or procedure arguments need complete Callable[[...], ...] metadata in the .pyi file.", + "Some callback or procedure arguments need a complete named prototype in the .pyi file.", { "owner": owner, "item": item, diff --git a/x2py/semantics/wrapper_policy.py b/x2py/semantics/wrapper_policy.py index 09192e5a1..311cfc234 100644 --- a/x2py/semantics/wrapper_policy.py +++ b/x2py/semantics/wrapper_policy.py @@ -370,6 +370,14 @@ class NativeInvocationKind(str, Enum): DEFINED_ASSIGNMENT = "defined_assignment" +class ExternalDeclarationMode(str, Enum): + """Completed Fortran declaration form for one native procedure.""" + + NONE = "none" + IMPLICIT_EXTERNAL = "implicit_external" + EXPLICIT_INTERFACE = "explicit_interface" + + def overload_builtin_scalar_family(semantic_type_name: str) -> str: """Return the Python scalar family admitted by reflected dispatch.""" if semantic_type_name == "Bool": @@ -922,7 +930,7 @@ class CallbackTransferPolicy: semantic_type_name: str object_kind: ObjectKind rank: int - access: str + passed_by_value: bool abi: CallbackABIKind adapter_action: CallbackTransferAction python_action: PythonBarrierAction @@ -944,6 +952,9 @@ class CallbackHandoffPolicy: """Complete immediate-callback contract consumed by wrapper planning.""" owner_path: str + prototype_name: str + prototype_module: str | None + declaration_mode: ExternalDeclarationMode arguments: tuple[CallbackTransferPolicy, ...] result: CallbackResultPolicy lifecycle: tuple[CallbackLifecycleAction, ...] @@ -1081,6 +1092,7 @@ class FunctionWrapperPolicy: native_invocation: NativeInvocationKind native_operator: str | None external: bool + external_declaration: ExternalDeclarationMode native_module: str | None native_is_subroutine: bool hold_gil: bool @@ -1929,8 +1941,22 @@ def build_callback_handoff_policy( ) result = _callback_result_policy(return_type, owner_path=f"{owner_path}.callback_result") blockers.extend(_callback_result_blockers(return_type, result)) + prototype_ref = semantic_type.metadata.get(models.PROTOTYPE_REF_METADATA) + prototype_name = prototype_ref.get("name") if isinstance(prototype_ref, dict) else None + prototype_module = prototype_ref.get("origin_module") if isinstance(prototype_ref, dict) else None + if not isinstance(prototype_name, str) or not prototype_name: + blockers.append("callback argument requires a resolved named prototype") + prototype_name = semantic_type.name + declaration_mode = _callback_declaration_mode(raw_arguments, return_type, semantic_type) + if declaration_mode is ExternalDeclarationMode.EXPLICIT_INTERFACE and not ( + isinstance(prototype_module, str) and prototype_module + ): + blockers.append(f"prototype {prototype_name!r} requires an importable native module") return CallbackHandoffPolicy( owner_path=owner_path, + prototype_name=prototype_name, + prototype_module=(prototype_module if isinstance(prototype_module, str) and prototype_module else None), + declaration_mode=declaration_mode, arguments=arguments, result=result, lifecycle=( @@ -1949,6 +1975,60 @@ def build_callback_handoff_policy( ) +def _callback_declaration_mode( + arguments: object, + return_type: object, + semantic_type: models.SemanticType, +) -> ExternalDeclarationMode: + """Select the weakest correct adapter declaration from prototype facts.""" + if isinstance(arguments, list) and any( + isinstance(argument, models.SemanticArgument) and _prototype_argument_requires_explicit_interface(argument) + for argument in arguments + ): + return ExternalDeclarationMode.EXPLICIT_INTERFACE + if isinstance(return_type, models.SemanticType) and _prototype_result_requires_explicit_interface(return_type): + return ExternalDeclarationMode.EXPLICIT_INTERFACE + prototype_metadata = semantic_type.metadata.get("prototype_metadata") + attributes = prototype_metadata.get("fortran_attributes", ()) if isinstance(prototype_metadata, dict) else () + normalized = {str(attribute).casefold().replace(" ", "") for attribute in attributes} + if normalized & {"bind(c)", "elemental", "pure"}: + return ExternalDeclarationMode.EXPLICIT_INTERFACE + return ExternalDeclarationMode.IMPLICIT_EXTERNAL + + +def _prototype_argument_requires_explicit_interface(argument: models.SemanticArgument) -> bool: + semantic_type = argument.semantic_type + storage = semantic_type.storage + array = storage.array if storage is not None else None + if argument.optional: + return True + if any( + semantic_type.metadata.get(name) + for name in ( + "fortran_allocatable", + "fortran_pointer", + "fortran_polymorphic", + "fortran_assumed_type", + "fortran_target", + ) + ): + return True + return array is not None and array.category in {"assumed_shape", "deferred_shape", "assumed_rank"} + + +def _prototype_result_requires_explicit_interface(return_type: models.SemanticType) -> bool: + if return_type.name == "None": + return False + if return_type.rank > 0: + return True + if return_type.metadata.get("fortran_allocatable") or return_type.metadata.get("fortran_pointer"): + return True + if return_type.name != "String": + return False + length = return_type.metadata.get("fortran_character_length") + return length is None or str(length).strip() in {"", ":", "*"} + + def _callback_envelope_blockers(semantic_type: models.SemanticType) -> tuple[str, ...]: """Require the documented immediate, entering-thread fatal envelope.""" required = { @@ -1975,7 +2055,7 @@ def _callback_transfer_policy( decision = _ownership_decision(argument, models.RESOLVED_OWNERSHIP_POLICY_METADATA) if decision is None: raise ValueError(f"Callback transfer {owner_path!r} is missing completed ownership policy") - access = str(argument.metadata.get(models.CALLBACK_DECLARATION_ACCESS_METADATA, "read")) + passed_by_value = bool(argument.origin.metadata.get("value")) derived = _is_scalar_derived_type(semantic_type) array = _array_handoff_policy(semantic_type) if int(semantic_type.rank or 0) > 0 else None return CallbackTransferPolicy( @@ -1984,9 +2064,9 @@ def _callback_transfer_policy( semantic_type_name=semantic_type.name, object_kind=decision.kind, rank=int(semantic_type.rank or 0), - access=access, - abi=_callback_abi_kind(argument, access, derived=derived), - adapter_action=_callback_adapter_action(argument, access), + passed_by_value=passed_by_value, + abi=_callback_abi_kind(argument, derived=derived), + adapter_action=_callback_adapter_action(argument), python_action=decision.python_barrier_action, character_length=_character_length(semantic_type), array=array, @@ -1996,7 +2076,6 @@ def _callback_transfer_policy( def _callback_abi_kind( argument: models.SemanticArgument, - access: str, *, derived: bool, ) -> CallbackABIKind: @@ -2015,20 +2094,11 @@ def _callback_abi_kind( def _callback_adapter_action( argument: models.SemanticArgument, - access: str, ) -> CallbackTransferAction: - """Select adapter copy direction once from the callable declaration.""" - if bool(argument.origin.metadata.get("value")) or access == "read": - return CallbackTransferAction.COPY_IN - if access == "write": - return CallbackTransferAction.COPY_OUT - if access in {"readwrite", "unspecified"}: - return CallbackTransferAction.COPY_IN_OUT - if argument.semantic_type.name == "String" or int(argument.semantic_type.rank or 0) > 0: - return CallbackTransferAction.COPY_IN - if _is_scalar_derived_type(argument.semantic_type): + """Select permissive reference writeback or isolated value transport.""" + if bool(argument.origin.metadata.get("value")): return CallbackTransferAction.COPY_IN - return CallbackTransferAction.BORROW_READ_ONLY + return CallbackTransferAction.COPY_IN_OUT def _callback_transfer_blockers( @@ -2038,8 +2108,8 @@ def _callback_transfer_blockers( """Reject callback forms whose typed adapter ABI is incomplete.""" semantic_type = argument.semantic_type blockers = [] - if transfer.access not in {"read", "write", "readwrite", "unspecified"}: - blockers.append(f"callback argument {argument.name!r} has unsupported access {transfer.access!r}") + if transfer.passed_by_value and transfer.rank > 0: + blockers.append(f"callback argument {argument.name!r} cannot pass an array by value") if semantic_type.name == "String": if transfer.character_length is None or transfer.character_length <= 0: blockers.append(f"callback argument {argument.name!r} requires a fixed positive character length") @@ -2077,7 +2147,7 @@ def _callback_result_policy( semantic_type_name=return_type.name, object_kind=decision.kind, rank=int(return_type.rank or 0), - access="result", + passed_by_value=False, abi=( CallbackABIKind.DERIVED_ADDRESS if derived @@ -2158,6 +2228,7 @@ def build_function_wrapper_policy( writeback_actions, lifecycle_blockers = _lifecycle_policies(arguments) cleanup_actions, release_actions = _derived_result_lifecycle_policies(results) status_error = _completed_native_status_error_policy(function) + native_module = _native_module(function, owner_path) blockers = ( _function_shape_blockers(function, class_call) + argument_blockers @@ -2172,14 +2243,22 @@ def build_function_wrapper_policy( ) native_name = native_dispatch_name or _native_name(function) native_invocation, native_operator = _native_invocation_policy(native_name) + external = _is_external(function) return FunctionWrapperPolicy( owner_path=owner_path, python_exports=completed_python_exports(function, function.name), native_name=native_name, native_invocation=native_invocation, native_operator=native_operator, - external=_is_external(function), - native_module=_native_module(function, owner_path), + external=external, + external_declaration=_external_declaration_mode( + external=external, + native_invocation=native_invocation, + arguments=tuple(arguments), + results=results, + native_call_slots=tuple(native_call_slots), + ), + native_module=native_module, native_is_subroutine=_native_is_subroutine(function), hold_gil=bool(function.metadata.get(models.RUNTIME_HOLD_GIL_METADATA)) or any(argument.callback is not None for argument in arguments), @@ -2209,6 +2288,64 @@ def _native_invocation_policy(native_name: str) -> tuple[NativeInvocationKind, s return NativeInvocationKind.PROCEDURE, None +def _external_declaration_mode( + *, + external: bool, + native_invocation: NativeInvocationKind, + arguments: tuple[ArgumentPolicy, ...], + results: tuple[ResultPolicy, ...], + native_call_slots: tuple[NativeCallSlotPolicy, ...], +) -> ExternalDeclarationMode: + """Choose the weakest correct native declaration from completed ABI facts.""" + if not external: + return ExternalDeclarationMode.NONE + if native_invocation is not NativeInvocationKind.PROCEDURE: + return ExternalDeclarationMode.EXPLICIT_INTERFACE + if any(_argument_requires_explicit_interface(argument) for argument in arguments): + return ExternalDeclarationMode.EXPLICIT_INTERFACE + if any(_result_requires_explicit_interface(result) for result in results): + return ExternalDeclarationMode.EXPLICIT_INTERFACE + if any(_slot_requires_explicit_interface(slot) for slot in native_call_slots): + return ExternalDeclarationMode.EXPLICIT_INTERFACE + return ExternalDeclarationMode.IMPLICIT_EXTERNAL + + +def _argument_requires_explicit_interface(argument: ArgumentPolicy) -> bool: + """Return whether one completed native dummy cannot use an implicit interface.""" + if argument.optional_mode is not OptionalMode.REQUIRED: + return True + if argument.native_array_handle is not None or argument.derived is not None or argument.polymorphic is not None: + return True + if argument.handoff_mode is ArgumentHandoffMode.NATIVE_DESCRIPTOR: + return True + return _array_requires_explicit_interface(argument.array) + + +def _result_requires_explicit_interface(result: ResultPolicy) -> bool: + """Return whether one completed native result requires an explicit interface.""" + if result.source_kind != "direct_return": + return False + if result.scalar_descriptor is not None or result.native_array_handle is not None or result.derived is not None: + return True + if result.rank > 0 or result.semantic_type_name == "String": + return True + return _array_requires_explicit_interface(result.array) + + +def _slot_requires_explicit_interface(slot: NativeCallSlotPolicy) -> bool: + """Return whether one ordered native slot carries descriptor-only ABI semantics.""" + if slot.value_kind == "value": + return True + if slot.native_array_handle is not None or slot.scalar_descriptor is not None or slot.derived is not None: + return True + return _array_requires_explicit_interface(slot.array) + + +def _array_requires_explicit_interface(array: ArrayHandoffPolicy | None) -> bool: + """Return whether an array dummy uses a descriptor-based Fortran category.""" + return array is not None and array.category in {"assumed_shape", "deferred_shape", "assumed_rank"} + + def _argument_policies( function: models.SemanticFunction, owner_path: str, @@ -5491,7 +5628,7 @@ def _is_scalar_derived_type(semantic_type: models.SemanticType) -> bool: return bool( int(semantic_type.rank or 0) == 0 and semantic_type.name not in _PLAN_PRIMITIVE_SCALAR_TYPES | {"String", "Void"} - and semantic_type.name not in {"Callable", "Procedure", "Callback", "FunctionPointer", "CFunctionPointer"} + and semantic_type.name not in {"Procedure", "Callback", "FunctionPointer", "CFunctionPointer"} ) diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index ca956edc1..dc1805671 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -913,7 +913,6 @@ def _callback_trampoline_function(self, callback: CallbackHandoffPlan) -> CFunct callback.trampoline_symbol, self._callback_c_return_type(callback), parameters=self._callback_c_parameters(callback.arguments), - storage="static", body=tuple(nodes), ) @@ -974,18 +973,8 @@ def _callback_scalar_reference_nodes( transfer: CallbackTransferPlan, target: str, ) -> tuple[CDeclaration, ...]: - """Convert read-only reference input by value; preserve writable storage.""" - if transfer.access != "read": - return self._callback_scalar_storage_nodes(transfer, target) - scalar = PrimitiveScalarTypeRegistry.type_for(transfer.semantic_type_name) - parameter = self._callback_parameter_base_name(transfer) - return ( - CDeclaration( - target, - "PyObject *", - CodeExpression(f"{scalar.python_result_converter}(({scalar.c_spelling} *){parameter}_data)"), - ), - ) + """Expose every scalar reference as permissive mutable rank-zero storage.""" + return self._callback_scalar_storage_nodes(transfer, target) def _callback_scalar_storage_nodes( self, @@ -9404,7 +9393,7 @@ def _bridge_direct_result_values( def _bridge_call_arguments(self, plan: ArgumentTransferPlan, names: _CArgumentNames) -> tuple[str, ...]: """Return one binding-to-bridge C handoff, including helper ABI fields.""" if plan.callback is not None: - return (plan.callback.trampoline_symbol,) + return () if plan.bridge.handoff_mode is ArgumentHandoffMode.CHARACTER_BUFFER: return self._string_bridge_call_arguments(names) if plan.bridge.handoff_mode is ArgumentHandoffMode.ARRAY_BUFFER: @@ -9529,30 +9518,11 @@ def _bridge_argument_parameters(self, argument: ArgumentTransferPlan) -> tuple[C """Return the bridge ABI parameters for one Python argument.""" name = argument.bridge.native_name.lower() if argument.callback is not None: - return self._callback_bridge_argument_parameters(argument, name) + return () if argument.derived_call is not None: return self._derived_bridge_argument_parameters(argument, name) return self._ordinary_bridge_argument_parameters(argument, name) - def _callback_bridge_argument_parameters( - self, - argument: ArgumentTransferPlan, - name: str, - ) -> tuple[CParameter, ...]: - """Declare one typed callback function parameter from its callback plan.""" - callback = argument.callback - if callback is None: - raise ValueError(f"Callback argument {argument.owner_path!r} has no handoff plan") - return ( - CParameter( - name, - self._callback_c_return_type(callback), - function_parameters=tuple( - parameter.type_name for parameter in self._callback_c_parameters(callback.arguments) - ), - ), - ) - @staticmethod def _derived_bridge_argument_parameters( argument: ArgumentTransferPlan, diff --git a/x2py/wrapper_codegen/docstrings.py b/x2py/wrapper_codegen/docstrings.py index d3cd8d937..19091491d 100644 --- a/x2py/wrapper_codegen/docstrings.py +++ b/x2py/wrapper_codegen/docstrings.py @@ -94,7 +94,7 @@ def class_surface( ) return "\n".join(lines) - # Callable documentation. + # Callback documentation. def function( self, python_name: str, @@ -557,16 +557,14 @@ def _base_type(self, transfer) -> str: def _callback_type(self, callback: CallbackHandoffPlan | None) -> str: if callback is None: raise ValueError("Callback documentation requires a completed handoff plan") - arguments = ", ".join(self._callback_transfer_type(item) for item in callback.arguments) - result = "None" if callback.result.transfer is None else self._callback_transfer_type(callback.result.transfer) - return f"Callable[[{arguments}], {result}]" + return callback.prototype_name @staticmethod def _callback_transfer_type(transfer: CallbackTransferPlan) -> str: if transfer.derived_type_identity is not None: return transfer.semantic_type_name scalar = _SCALAR_TYPES.get(transfer.semantic_type_name, transfer.semantic_type_name) - if transfer.array is not None or (transfer.abi.value == "reference" and transfer.access != "read"): + if transfer.array is not None or transfer.abi.value == "reference": return f"ndarray[{scalar}]" return scalar diff --git a/x2py/wrapper_codegen/fortran/bridge.py b/x2py/wrapper_codegen/fortran/bridge.py index 4ed1c40f3..fd7116b04 100644 --- a/x2py/wrapper_codegen/fortran/bridge.py +++ b/x2py/wrapper_codegen/fortran/bridge.py @@ -27,6 +27,7 @@ DerivedDummyCategory, DerivedObjectStorage, DerivedRelease, + ExternalDeclarationMode, ModuleGetterAction, ModuleObjectAccessMechanism, NativeArrayDescriptorKind, @@ -587,7 +588,6 @@ def _visit_ModulePlan(self, plan: ModulePlan) -> FortranModule: ), type_definitions=self._derived_holder_definitions(plan), interfaces=( - *self._callback_interfaces(plan), *self._derived_call_interfaces(plan), *self._external_interfaces(plan), *self._module_descriptor_callback_interfaces(plan), @@ -617,6 +617,13 @@ def _visit_ModulePlan(self, plan: ModulePlan) -> FortranModule: for procedure in self._derived_origin_procedures(variable) ), ), + external_procedures=self._callback_external_adapter_procedures(plan), + ) + + def _callback_external_adapter_procedures(self, plan: ModulePlan) -> tuple[FortranFunction, ...]: + """Return separately linked callback adapters in stable site order.""" + return tuple( + self._callback_external_adapter_procedure(callback, plan) for callback in self._callback_sites(plan) ) def _derived_holder_definitions(self, plan: ModulePlan) -> tuple[FortranTypeDefinition, ...]: @@ -683,7 +690,6 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: bridge_name = self._bridge_function_name(plan) is_subroutine = plan.bridge.native_is_subroutine or owned_direct_result is not None function_body, optional_procedures = self._function_body(plan, result_name) - callback_procedures = self._callback_adapter_procedures(plan) native_body = ( *self._derived_pointer_call_initializers(plan), *function_body, @@ -703,6 +709,8 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: result_type=result_type, bind_name=bridge_name, declarations=( + *self._callback_external_declarations(plan), + *self._native_external_declarations(plan), *self._optional_declarations(plan), *self._opaque_address_declarations(plan), *self._array_declarations(plan), @@ -726,34 +734,39 @@ def _visit_FunctionPlan(self, plan: FunctionPlan) -> FortranFunction: ), is_subroutine=is_subroutine, internal_procedures=( - *callback_procedures, *optional_procedures, *internal_procedures, ), ) # Immediate callback adapters. - def _callback_adapter_procedures(self, plan: FunctionPlan) -> tuple[FortranFunction, ...]: - """Return one typed native adapter for each Python callback argument.""" - return tuple( - self._callback_adapter_procedure(argument.callback, argument.bridge.native_name.lower()) - for argument in sorted(plan.arguments, key=lambda item: item.native_position) - if argument.callback is not None - ) - - def _callback_adapter_procedure( + def _callback_external_adapter_procedure( self, callback: CallbackHandoffPlan, - callback_name: str, + plan: ModulePlan, ) -> FortranFunction: - """Adapt one native callback signature to its completed C trampoline ABI.""" + """Adapt one native callback through a separately declared external procedure.""" result = callback.result.transfer is_subroutine = callback.result.action is CallbackResultAction.RETURN_VOID + trampoline_name = f"{callback.trampoline_symbol}_call" return FortranFunction( name=callback.adapter_symbol, parameters=tuple(self._callback_native_parameter(transfer) for transfer in callback.arguments), result_name=None if is_subroutine else "callback_result", result_type=None if is_subroutine else self._callback_native_result_type(result), + uses=self._callback_external_adapter_uses(callback, plan), + implicit_none=True, + interfaces=( + FortranInterface( + ( + self._callback_c_interface( + callback, + name=trampoline_name, + bind_name=callback.trampoline_symbol, + ), + ) + ), + ), declarations=( *( declaration @@ -768,7 +781,7 @@ def _callback_adapter_procedure( for transfer in callback.arguments for statement in self._callback_transfer_preparation(transfer) ), - *self._callback_invocation(callback, callback_name), + *self._callback_invocation(callback, trampoline_name), *( statement for transfer in callback.arguments @@ -781,10 +794,8 @@ def _callback_adapter_procedure( def _callback_native_parameter(self, transfer: CallbackTransferPlan) -> FortranParameter: """Declare the exact native callback dummy represented by one transfer.""" - attributes = list(self._callback_intent_attributes(transfer)) - if transfer.abi is CallbackABIKind.VALUE: - if transfer.access == "unspecified": - attributes.append("intent(in)") + attributes = [] + if transfer.passed_by_value: attributes.append("value") if transfer.abi is not CallbackABIKind.VALUE and transfer.adapter_action in { CallbackTransferAction.BORROW_READ_ONLY, @@ -799,15 +810,74 @@ def _callback_native_parameter(self, transfer: CallbackTransferPlan) -> FortranP tuple(attributes), ) - @staticmethod - def _callback_intent_attributes(transfer: CallbackTransferPlan) -> tuple[str, ...]: - """Map the completed callback access mode to one native INTENT.""" - intent = { - "read": "intent(in)", - "write": "intent(out)", - "readwrite": "intent(inout)", - }.get(transfer.access) - return (intent,) if intent is not None else () + def _callback_external_adapter_uses( + self, + callback: CallbackHandoffPlan, + plan: ModulePlan, + ) -> tuple[FortranUse, ...]: + """Import the native types and C ABI kinds used by one external adapter.""" + native_imports = self._callback_native_imports(callback) + adapter_imports = ( + *(("c_loc",) if any(transfer.abi is not CallbackABIKind.VALUE for transfer in callback.arguments) else ()), + *( + ("c_f_pointer",) + if callback.result.action + in {CallbackResultAction.RETURN_ARRAY_ADDRESS, CallbackResultAction.RETURN_DERIVED_ADDRESS} + else () + ), + ) + iso_imports = tuple( + symbol for symbol in (*native_imports, *adapter_imports) if not symbol.startswith("x2py_type_") + ) + derived_imports = tuple(symbol for symbol in native_imports if symbol.startswith("x2py_type_")) + return ( + FortranUse( + "iso_c_binding", + tuple(dict.fromkeys((*iso_imports, *self._callback_c_imports(callback)))), + ), + *( + ( + FortranUse( + f"bind_c_{plan.bridge.owner_path}_wrapper", + derived_imports, + ), + ) + if derived_imports + else () + ), + ) + + def _callback_external_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Declare external callback adapters from completed callback policy.""" + declarations = [] + for callback in ( + argument.callback + for argument in sorted(plan.arguments, key=lambda item: item.native_position) + if argument.callback is not None + ): + if callback.declaration_mode is ExternalDeclarationMode.EXPLICIT_INTERFACE: + declarations.append( + FortranDeclaration( + callback.adapter_symbol, + f"procedure({self._callback_imported_prototype_symbol(callback)})", + ) + ) + continue + if callback.declaration_mode is not ExternalDeclarationMode.IMPLICIT_EXTERNAL: + raise ValueError( + f"Callback {callback.owner_path!r} has unsupported declaration mode {callback.declaration_mode!r}" + ) + if callback.result.action is CallbackResultAction.RETURN_VOID: + declarations.append(FortranDeclaration(callback.adapter_symbol, "external")) + continue + declarations.append( + FortranDeclaration( + callback.adapter_symbol, + self._callback_native_result_type(callback.result.transfer), + ("external",), + ) + ) + return tuple(declarations) def _callback_transfer_declarations( self, @@ -2561,12 +2631,7 @@ def _visit_ArgumentTransferPlan(self, plan: ArgumentTransferPlan) -> tuple[Fortr def _lower_argument(self, plan: ArgumentTransferPlan) -> tuple[FortranParameter, ...]: """Dispatch one completed bridge optional mode explicitly.""" if plan.callback is not None: - return ( - FortranParameter( - plan.bridge.native_name.lower(), - f"procedure({self._callback_c_interface_symbol(plan.callback)})", - ), - ) + return () mode = plan.bridge.optional_mode if plan.object_kind is ObjectKind.DERIVED_TYPE: return self._lower_derived_argument(plan, mode) @@ -4939,6 +5004,7 @@ def _add_derived_module_uses(self, plan: ModulePlan, modules: dict[str, list[str def _add_function_module_uses(self, plan: ModulePlan, modules: dict[str, list[str]]) -> None: """Import module procedures, excluding direct type-bound invocation.""" for function in self._functions(plan): + self._add_callback_prototype_uses(function, modules) if ( function.bridge.native_module is not None and function.bridge.native_invocation is not NativeInvocationKind.PROCEDURE @@ -4952,6 +5018,23 @@ def _add_function_module_uses(self, plan: ModulePlan, modules: dict[str, list[st f"{self._native_function_name(function)} => {function.bridge.native_name}" ) + def _add_callback_prototype_uses( + self, + function: FunctionPlan, + modules: dict[str, list[str]], + ) -> None: + """Import named prototypes selected by completed callback policy.""" + for argument in function.arguments: + callback = argument.callback + if ( + callback is not None + and callback.declaration_mode is ExternalDeclarationMode.EXPLICIT_INTERFACE + and callback.prototype_module is not None + ): + modules.setdefault(callback.prototype_module, []).append( + f"{self._callback_imported_prototype_symbol(callback)} => {callback.prototype_name}" + ) + def _add_variable_module_uses(self, plan: ModulePlan, modules: dict[str, list[str]]) -> None: """Import only module variables with a planned getter, setter, or proxy.""" for variable in self._variables(plan): @@ -6232,21 +6315,27 @@ def _external_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...] procedures = tuple( self._external_interface_procedure(function) for function in self._functions(plan) - if function.bridge.external + if function.bridge.external_declaration is ExternalDeclarationMode.EXPLICIT_INTERFACE ) return (FortranInterface(procedures),) if procedures else () - def _callback_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...]: - """Declare the C trampoline and native adapter signatures for each site.""" - procedures = tuple( - procedure - for callback in self._callback_sites(plan) - for procedure in ( - self._callback_c_interface(callback), - self._callback_native_interface(callback), - ) + def _native_external_declarations(self, plan: FunctionPlan) -> tuple[FortranDeclaration, ...]: + """Lower the completed implicit-external declaration mode.""" + mode = plan.bridge.external_declaration + if mode in {ExternalDeclarationMode.NONE, ExternalDeclarationMode.EXPLICIT_INTERFACE}: + return () + if mode is not ExternalDeclarationMode.IMPLICIT_EXTERNAL: + raise ValueError(f"Unsupported external declaration mode for {plan.owner_path!r}: {mode!r}") + name = self._native_function_name(plan) + if plan.bridge.native_is_subroutine: + return (FortranDeclaration(name, "external"),) + return ( + FortranDeclaration( + name, + self._native_result_type(plan, self._direct_result(plan)), + ("external",), + ), ) - return (FortranInterface(procedures, abstract=True),) if procedures else () def _callback_sites(self, plan: ModulePlan) -> tuple[CallbackHandoffPlan, ...]: """Return callback sites in stable native-call order.""" @@ -6257,12 +6346,18 @@ def _callback_sites(self, plan: ModulePlan) -> tuple[CallbackHandoffPlan, ...]: if argument.callback is not None ) - def _callback_c_interface(self, callback: CallbackHandoffPlan) -> FortranInterfaceProcedure: + def _callback_c_interface( + self, + callback: CallbackHandoffPlan, + *, + name: str, + bind_name: str, + ) -> FortranInterfaceProcedure: """Declare the flattened C ABI implemented by one Python trampoline.""" result = callback.result.transfer is_subroutine = callback.result.action is CallbackResultAction.RETURN_VOID return FortranInterfaceProcedure( - name=self._callback_c_interface_symbol(callback), + name=name, imports=self._callback_c_imports(callback), parameters=tuple( parameter for transfer in callback.arguments for parameter in self._callback_c_parameters(transfer) @@ -6270,22 +6365,10 @@ def _callback_c_interface(self, callback: CallbackHandoffPlan) -> FortranInterfa result_name=None if is_subroutine else "callback_result", result_type=None if is_subroutine else self._callback_c_result_type(result), is_subroutine=is_subroutine, + bind_name=bind_name, bind_c=True, ) - def _callback_native_interface(self, callback: CallbackHandoffPlan) -> FortranInterfaceProcedure: - """Declare the native signature that the internal adapter must satisfy.""" - result = callback.result.transfer - is_subroutine = callback.result.action is CallbackResultAction.RETURN_VOID - return FortranInterfaceProcedure( - name=self._callback_native_interface_symbol(callback), - imports=self._callback_native_imports(callback), - parameters=tuple(self._callback_native_parameter(transfer) for transfer in callback.arguments), - result_name=None if is_subroutine else "callback_result", - result_type=None if is_subroutine else self._callback_native_result_type(result), - is_subroutine=is_subroutine, - ) - def _callback_c_parameters(self, transfer: CallbackTransferPlan) -> tuple[FortranParameter, ...]: """Flatten one callback transfer into interoperable C parameters.""" base = self._callback_parameter_base_name(transfer) @@ -6350,12 +6433,8 @@ def _callback_native_imports(self, callback: CallbackHandoffPlan) -> tuple[str, return tuple(dict.fromkeys(imports)) @staticmethod - def _callback_c_interface_symbol(callback: CallbackHandoffPlan) -> str: - return f"{callback.trampoline_symbol}_interface" - - @staticmethod - def _callback_native_interface_symbol(callback: CallbackHandoffPlan) -> str: - return f"{callback.adapter_symbol}_interface" + def _callback_imported_prototype_symbol(callback: CallbackHandoffPlan) -> str: + return f"{callback.adapter_symbol}_prototype" def _derived_call_interfaces(self, plan: ModulePlan) -> tuple[FortranInterface, ...]: """Declare the typed callback ABI shared by every scalar-derived call.""" @@ -6800,10 +6879,7 @@ def _external_interface_parameter( """Return the native external dummy declaration for one planned argument.""" parameter_name = name or argument.bridge.native_name.lower() if argument.callback is not None: - return FortranParameter( - parameter_name, - f"procedure({self._callback_native_interface_symbol(argument.callback)})", - ) + return FortranParameter(parameter_name, "external") attributes = ( ("optional",) if argument.bridge.optional_mode in {OptionalMode.NULLABLE_VALUE, OptionalMode.DESCRIPTOR} diff --git a/x2py/wrapper_codegen/nodes.py b/x2py/wrapper_codegen/nodes.py index 1e055f0fd..5c7911323 100644 --- a/x2py/wrapper_codegen/nodes.py +++ b/x2py/wrapper_codegen/nodes.py @@ -399,6 +399,9 @@ class FortranFunction(StageRecord): result_type: str | None = None bind_name: str | None = None bind_c: bool = False + uses: tuple[FortranUse, ...] = () + implicit_none: bool = False + interfaces: tuple[FortranInterface, ...] = () declarations: tuple[FortranDeclaration, ...] = () body: tuple[ FortranAllocate @@ -424,3 +427,4 @@ class FortranModule(StageRecord): type_definitions: tuple[FortranTypeDefinition, ...] = () interfaces: tuple[FortranInterface, ...] = () procedures: tuple[FortranFunction, ...] = () + external_procedures: tuple[FortranFunction, ...] = () diff --git a/x2py/wrapper_codegen/plan.py b/x2py/wrapper_codegen/plan.py index cc5d7e4ff..ecf18ba39 100644 --- a/x2py/wrapper_codegen/plan.py +++ b/x2py/wrapper_codegen/plan.py @@ -46,6 +46,7 @@ DerivedRelease, DerivedTargetLifetime, DerivedWriteback, + ExternalDeclarationMode, ModuleGetterAction, ModuleObjectAccessMechanism, NativeArrayDescriptorInterop, @@ -463,6 +464,7 @@ class BridgeFunctionPlan(StageRecord): native_invocation: NativeInvocationKind native_operator: str | None external: bool + external_declaration: ExternalDeclarationMode native_module: str | None native_is_subroutine: bool @@ -609,7 +611,7 @@ class CallbackTransferPlan(StageRecord): semantic_type_name: str object_kind: ObjectKind rank: int - access: str + passed_by_value: bool abi: CallbackABIKind adapter_action: CallbackTransferAction python_action: PythonBarrierAction @@ -635,6 +637,9 @@ class CallbackHandoffPlan(StageRecord): """Call-scoped callback context, symbols, transfers, and fatal contract.""" owner_path: str + prototype_name: str + prototype_module: str | None + declaration_mode: ExternalDeclarationMode context_type_symbol: str context_current_symbol: str adapter_symbol: str diff --git a/x2py/wrapper_codegen/planner.py b/x2py/wrapper_codegen/planner.py index b217eecab..ea465d866 100644 --- a/x2py/wrapper_codegen/planner.py +++ b/x2py/wrapper_codegen/planner.py @@ -908,6 +908,7 @@ def _function_plan( policy.native_invocation, policy.native_operator, policy.external, + policy.external_declaration, policy.native_module, policy.native_is_subroutine, ), @@ -1024,6 +1025,9 @@ def _callback_handoff_plan( stem = NativeSymbolNames.compact(policy.owner_path, "callback", limit=24) return CallbackHandoffPlan( owner_path=policy.owner_path, + prototype_name=policy.prototype_name, + prototype_module=policy.prototype_module, + declaration_mode=policy.declaration_mode, context_type_symbol=f"x2py_callback_context_{stem}", context_current_symbol=f"x2py_callback_current_{stem}", adapter_symbol=f"x2py_callback_adapter_{stem}", @@ -1053,7 +1057,7 @@ def _callback_transfer_plan(self, policy: CallbackTransferPolicy) -> CallbackTra semantic_type_name=policy.semantic_type_name, object_kind=policy.object_kind, rank=policy.rank, - access=policy.access, + passed_by_value=policy.passed_by_value, abi=policy.abi, adapter_action=policy.adapter_action, python_action=policy.python_action, diff --git a/x2py/wrapper_codegen/printers/pyi_printer.py b/x2py/wrapper_codegen/printers/pyi_printer.py index 505ee23f6..372d774f7 100644 --- a/x2py/wrapper_codegen/printers/pyi_printer.py +++ b/x2py/wrapper_codegen/printers/pyi_printer.py @@ -22,7 +22,6 @@ USER_PRIVATE_METADATA, ) from x2py.semantics.models import ( - CALLBACK_DECLARATION_ACCESS_METADATA, EXTERNAL_TYPE_REF_METADATA, FORTRAN_GENERIC_NAME_METADATA, OVERLOAD_KIND_METADATA, @@ -33,6 +32,7 @@ PYTHON_STATIC_METADATA, PYTHON_VALUE_IMMUTABLE, PYTHON_VALUE_MUTABILITY_METADATA, + PROTOTYPE_REF_METADATA, RUNTIME_HOLD_GIL_METADATA, RUNTIME_STATUS_ERROR_METADATA, ProjectionMapping, @@ -46,6 +46,7 @@ SemanticImportItem, SemanticMethod, SemanticModule, + SemanticPrototype, SemanticStorageContract, SemanticType, SemanticVariable, @@ -58,11 +59,6 @@ _CONTRACT_MODULE = "x2py.contracts" _CONTRACT_ALIAS_PREFIX = "_x2py_" _FLAT_DIMENSION_PRINT_SENTINEL = "@x2py.Flat" -_CALLABLE_ACCESS_WRAPPER = { - "read": "In", - "write": "Out", - "readwrite": "InOut", -} class PyiPrinter(ClassVisitor): @@ -130,15 +126,15 @@ def _visit_SemanticType(self, semantic_type: SemanticType) -> str: if semantic_type.name == "Unknown" or semantic_type.dtype == "Unknown": raise ValueError("Cannot emit .pyi with unresolved semantic type 'Unknown'") array_descriptor = native_array_descriptor_kind(semantic_type) - if array_descriptor is not None: + if PROTOTYPE_REF_METADATA in semantic_type.metadata: + text = semantic_type.name + elif array_descriptor is not None: wrapper = "Allocatable" if array_descriptor == "allocatable" else "Pointer" text = f"{self._contract(wrapper)}[{self._visit(native_array_data_type(semantic_type))}]" elif self._is_scalar_allocatable_descriptor(semantic_type): text = f"{self._contract('Allocatable')}[{self._scalar_descriptor_inner_text(semantic_type)}]" elif self._is_scalar_pointer_descriptor(semantic_type): text = f"{self._contract('Pointer')}[{self._scalar_descriptor_inner_text(semantic_type)}]" - elif semantic_type.name == "Callable": - text = self._emit_callable_type(semantic_type) elif semantic_type.storage is not None: text = self._emit_storage_type(semantic_type) else: @@ -169,6 +165,24 @@ def _visit_SemanticFunction(self, func: SemanticFunction) -> str: """Emit function syntax.""" return self._emit_function(func) + def _visit_SemanticPrototype(self, prototype: SemanticPrototype) -> str: + """Emit one semantic-only named callback prototype.""" + return_type = prototype.return_type or SemanticType("None", dtype="None") + arguments = [] + for argument in prototype.arguments: + text = f"{self._parameter_target(argument.name)}: {self._emit_prototype_argument(argument)}" + if argument.optional: + text += " = ..." + arguments.append(text) + return self._emit_callable( + name=prototype.name, + arguments=arguments, + return_type=self._visit(return_type), + decorator=f"@{self._contract('prototype')}\n", + def_indent="", + parameter_indent=" ", + ) + def _emit_function(self, func: SemanticFunction, *, name_owner: object | None = None) -> str: """Emit function syntax with an optional shared overload-set public name.""" return_type = self._projected_return_annotation(func) @@ -283,6 +297,7 @@ def _visit_SemanticModule(self, module: SemanticModule) -> str: body_sections: list[str] = [] try: self._append_items(body_sections, self._contract_items(module.classes), self.emit) + self._append_items(body_sections, module.prototypes, self._visit) self._append_items(body_sections, self._contract_items(module.variables), self._emit_module_variable) overload_targets = self._module_overload_target_names(module) self._append_items( @@ -479,46 +494,40 @@ def _semantic_annotation_metadata(self, semantic_type: SemanticType) -> list[str pointer_association = semantic_type.metadata.get("fortran_pointer_association") if pointer_association is not None and not self._is_scalar_pointer_descriptor(semantic_type): metadata.append(f"{self._contract('PointerAssociation')}({json.dumps(str(pointer_association))})") - pointer_policy = semantic_type.metadata.get(POINTER_POLICY_METADATA) - if isinstance(pointer_policy, dict): - arguments = [] - for name in POINTER_POLICY_FIELDS: - value = pointer_policy.get(name) - if value is not None: - rendered = repr(value) if isinstance(value, bool) else json.dumps(str(value)) - arguments.append(f"{name}={rendered}") - metadata.append(f"{self._contract('PointerPolicy')}({', '.join(arguments)})") - ownership_policy = semantic_type.metadata.get(OWNERSHIP_POLICY_METADATA) - if isinstance(ownership_policy, dict): - owner = ownership_policy.get("owner") - transfer = ownership_policy.get("transfer") - destruction = ownership_policy.get("destruction") - if owner is not None: - metadata.append(f"{self._contract('Ownership')}({json.dumps(str(owner))})") - if transfer is not None: - metadata.append(f"{self._contract('Transfer')}({json.dumps(str(transfer))})") - if destruction is not None: - metadata.append(f"{self._contract('Destruction')}({json.dumps(str(destruction))})") + pointer_policy = self._pointer_policy_annotation(semantic_type) + if pointer_policy is not None: + metadata.append(pointer_policy) + metadata.extend(self._ownership_policy_annotations(semantic_type)) return metadata - def _emit_callable_type(self, semantic_type: SemanticType) -> str: - """Emit callable type syntax.""" - arguments = semantic_type.metadata.get("arguments") - return_type = semantic_type.metadata.get("return") - if isinstance(arguments, list) and return_type is not None: - callback_arguments = semantic_type.metadata.get("callback_arguments") - if ( - isinstance(callback_arguments, list) - and len(callback_arguments) == len(arguments) - and all(isinstance(arg, SemanticArgument) for arg in callback_arguments) - ): - args = ", ".join(self._emit_callable_argument(arg) for arg in callback_arguments) - else: - args = ", ".join(self._visit(arg) for arg in arguments) - return f"{self._contract('Callable')}[[{args}], {self._visit(return_type)}]" - if return_type is not None: - return f"{self._contract('Callable')}[..., {self._visit(return_type)}]" - return self._contract("Callable") + def _pointer_policy_annotation(self, semantic_type: SemanticType) -> str | None: + """Render one structured pointer policy annotation when present.""" + pointer_policy = semantic_type.metadata.get(POINTER_POLICY_METADATA) + if not isinstance(pointer_policy, dict): + return None + arguments = [] + for name in POINTER_POLICY_FIELDS: + value = pointer_policy.get(name) + if value is not None: + rendered = repr(value) if isinstance(value, bool) else json.dumps(str(value)) + arguments.append(f"{name}={rendered}") + return f"{self._contract('PointerPolicy')}({', '.join(arguments)})" + + def _ownership_policy_annotations(self, semantic_type: SemanticType) -> tuple[str, ...]: + """Render explicit owner, transfer, and destruction policy metadata.""" + ownership_policy = semantic_type.metadata.get(OWNERSHIP_POLICY_METADATA) + if not isinstance(ownership_policy, dict): + return () + fields = ( + ("owner", "Ownership"), + ("transfer", "Transfer"), + ("destruction", "Destruction"), + ) + return tuple( + f"{self._contract(contract)}({json.dumps(str(ownership_policy[key]))})" + for key, contract in fields + if ownership_policy.get(key) is not None + ) @staticmethod def _is_scalar_allocatable_descriptor(semantic_type: SemanticType) -> bool: @@ -565,32 +574,27 @@ def _visible_scalar_descriptor_type(semantic_type: SemanticType) -> SemanticType visible.ownership.mutable = False return visible - def _emit_callable_argument(self, argument: SemanticArgument) -> str: - """Emit one callback dummy argument with its callback ABI wrapper.""" - inner = self._callable_argument_inner_type(argument.semantic_type) + def _emit_prototype_argument(self, argument: SemanticArgument) -> str: + """Emit one prototype dummy with reference default and one value override.""" + inner = self._prototype_argument_inner_type(argument.semantic_type) if bool(getattr(argument.origin, "metadata", {}).get("value")): - return inner - access = argument.metadata.get(CALLBACK_DECLARATION_ACCESS_METADATA, "unspecified") - wrapper = _CALLABLE_ACCESS_WRAPPER.get(access) - if wrapper is None: - if self._callable_argument_requires_pass_by_ref_wrapper(argument.semantic_type): - return f"{self._contract('PassByRef')}({inner})" - return inner - return f"{self._contract(wrapper)}({inner})" - - def _callable_argument_inner_type(self, semantic_type: SemanticType) -> str: - """Return the native callback dummy type without callback ABI wrappers.""" + return f"{self._contract('Value')}({inner})" + return inner + + def _prototype_argument_inner_type(self, semantic_type: SemanticType) -> str: + """Return the native prototype dummy type without transport wrappers.""" storage = semantic_type.storage + if ( + semantic_type.name == "String" + and storage is not None + and storage.array is not None + and storage.array.category == SCALAR_STORAGE_CATEGORY + ): + return self._semantic_base_type(semantic_type, include_deferred_length=True) if storage is not None and storage.kind in {"reference", "address", "pointer"}: return self._address_target_type(semantic_type) return self._visit(semantic_type) - @staticmethod - def _callable_argument_requires_pass_by_ref_wrapper(semantic_type: SemanticType) -> bool: - """Return whether missing callback access needs an explicit scalar reference wrapper.""" - storage = semantic_type.storage - return bool(storage is not None and storage.kind in {"reference", "pointer"} and semantic_type.rank == 0) - def _emit_data_member(self, variable: SemanticVariable) -> str: """Emit a variable in class-field context rather than argument context.""" name = self._data_member_name(variable) @@ -997,7 +1001,7 @@ def _module_reserved_names(cls, module: SemanticModule) -> set[str]: names.update(cls._required_procedure_namespace_import_names(module)) for imp in module.imports: names.update(cls._import_local_names(imp)) - for item in [*module.classes, *module.variables, *module.functions, *module.overload_sets]: + for item in [*module.classes, *module.prototypes, *module.variables, *module.functions, *module.overload_sets]: cls._collect_reserved_item_names(item, names) for semantic_type in _iter_module_semantic_types(module): names.update(cls._contract_like_dimension_names(semantic_type)) @@ -1200,7 +1204,13 @@ def _top_level_declaration_names(module: SemanticModule) -> set[str]: """Return names emitted in a module-level stub namespace.""" return { str(item.name) - for item in [*module.classes, *module.variables, *module.functions, *module.overload_sets] + for item in [ + *module.classes, + *module.prototypes, + *module.variables, + *module.functions, + *module.overload_sets, + ] if getattr(item, "name", None) } diff --git a/x2py/wrapper_codegen/printers/source_printers.py b/x2py/wrapper_codegen/printers/source_printers.py index 8ef57898a..df2c22fbc 100644 --- a/x2py/wrapper_codegen/printers/source_printers.py +++ b/x2py/wrapper_codegen/printers/source_printers.py @@ -321,6 +321,7 @@ def _visit_FortranModule(self, node: FortranModule) -> str: lines.append("contains") lines.extend(self._indented(self.visit(procedure)) for procedure in node.procedures) lines.append(f"end module {node.name}") + lines.extend(self.visit(procedure) for procedure in node.external_procedures) return "\n".join(lines) def _visit_FortranUse(self, node: FortranUse) -> str: @@ -339,11 +340,7 @@ def _visit_FortranUse(self, node: FortranUse) -> str: def _visit_FortranFunction(self, node: FortranFunction) -> str: """Render one Fortran function.""" signature = self._function_signature(node) - lines = [signature] - lines.extend(self._indented(self.visit(parameter)) for parameter in node.parameters) - if node.result_name is not None and node.result_type is not None: - lines.append(self._indented(f"{node.result_type} :: {node.result_name}")) - lines.extend(self._indented(self.visit(declaration)) for declaration in node.declarations) + lines = [signature, *self._fortran_function_specification(node)] lines.extend(self._indented(self.visit(statement)) for statement in node.body) if node.internal_procedures: lines.append("contains") @@ -351,6 +348,19 @@ def _visit_FortranFunction(self, node: FortranFunction) -> str: lines.append(f"end {'subroutine' if node.is_subroutine else 'function'} {node.name}") return "\n".join(lines) + def _fortran_function_specification(self, node: FortranFunction) -> list[str]: + """Render use, declaration, and local-interface specification lines.""" + lines = [] + lines.extend(self._indented(self.visit(use)) for use in node.uses) + if node.implicit_none: + lines.append(" implicit none") + lines.extend(self._indented(self.visit(parameter)) for parameter in node.parameters) + if node.result_name is not None and node.result_type is not None: + lines.append(self._indented(f"{node.result_type} :: {node.result_name}")) + lines.extend(self._indented(self.visit(declaration)) for declaration in node.declarations) + lines.extend(self._indented(self.visit(interface)) for interface in node.interfaces) + return lines + def _visit_FortranParameter(self, node: FortranParameter) -> str: """Render one Fortran parameter declaration.""" assumed_size = next( From 1b6b1c2eab9cd1892a764b50563738291ce13562 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 02:01:55 +0100 Subject: [PATCH 20/30] update docs --- README.md | 4 ++ .../getting-started/first-wrapped-function.md | 12 +++-- docs/user/getting-started/index.md | 2 +- docs/user/getting-started/verification.md | 2 +- docs/user/guide/allocatables.md | 17 +++---- docs/user/guide/arrays.md | 12 ++--- docs/user/guide/callbacks.md | 2 +- docs/user/guide/distribution.md | 2 +- .../guide/editing-semantic-pyi-contracts.md | 34 ++++++++++---- docs/user/guide/error-handling.md | 4 +- docs/user/guide/fortran-wrapper.md | 26 ++++++----- docs/user/guide/generic-interfaces.md | 2 + docs/user/guide/pointers.md | 21 ++++++++- docs/user/guide/wrapping-functions.md | 12 ++--- tests/docs/test_examples.py | 45 +++++++++++++++++++ 15 files changed, 141 insertions(+), 56 deletions(-) diff --git a/README.md b/README.md index a98b0f630..f9bcf17b8 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,8 @@ contracts/ Expected contract (`contracts/__init__.pyi`): ```python +from x2py.contracts import Addr, Arg, Float64, external, native_call + @external @native_call([Addr(Arg(0)), Addr(Arg(1))]) def scale( @@ -219,6 +221,8 @@ python3 -m x2py points.f90 --pyi --out contracts Expected contract (`contracts/points.pyi`): ```python +from x2py.contracts import Addr, Arg, Float64, native_call + class point: def __init__( self, diff --git a/docs/user/getting-started/first-wrapped-function.md b/docs/user/getting-started/first-wrapped-function.md index d52f1cbc2..7094deabc 100644 --- a/docs/user/getting-started/first-wrapped-function.md +++ b/docs/user/getting-started/first-wrapped-function.md @@ -17,7 +17,7 @@ Reuse the same `scale.f90` input from the [README Quick Start](../../../README.md#quick-start). The generated Python call accepts two `numpy.float64` values and returns a -`numpy.float64` result. +Python `float` result. ## Build @@ -43,11 +43,11 @@ import scale result = scale.scale(np.float64(3.0), np.float64(2.5)) -assert isinstance(result, np.float64) -assert result == np.float64(7.5) +assert isinstance(result, float) +assert result == 7.5 ``` -The checked call returns `numpy.float64(7.5)`. +The checked call returns the Python value `7.5`. ## Inspect The Generated Signature @@ -88,8 +88,6 @@ Native scalar arguments use exact NumPy dtypes. A plain Python `float` is not a replacement for `numpy.float64` at this boundary: ```python -from x2py.contracts import raises - scale.scale(3.0, 2.5) # raises TypeError ``` @@ -112,6 +110,6 @@ generated `.pyi` contract. ## Evidence The linked `scale.f90` input is checked against the repository fixture by -[`test_documentation_examples.py`](../../../tests/docs/test_examples.py). +[`test_examples.py`](../../../tests/docs/test_examples.py). The default extension name and `7.5` runtime result are checked by [`test_build_modes.py`](../../../tests/wrapper/fortran/build_from_source/test_build_modes.py). diff --git a/docs/user/getting-started/index.md b/docs/user/getting-started/index.md index 88f1bcbd2..93853ad65 100644 --- a/docs/user/getting-started/index.md +++ b/docs/user/getting-started/index.md @@ -59,6 +59,6 @@ failures are routed through [Troubleshooting](../troubleshooting/index.md). The standalone example used throughout this section is checked against its fixture by -[`test_documentation_examples.py`](../../../tests/docs/test_examples.py), +[`test_examples.py`](../../../tests/docs/test_examples.py), and its `7.5` runtime result is checked by [`test_build_modes.py`](../../../tests/wrapper/fortran/build_from_source/test_build_modes.py). diff --git a/docs/user/getting-started/verification.md b/docs/user/getting-started/verification.md index c5bcc12e1..f78cdae47 100644 --- a/docs/user/getting-started/verification.md +++ b/docs/user/getting-started/verification.md @@ -139,7 +139,7 @@ the full GitHub Actions matrix is the final cross-version evidence. ## Evidence The linked `scale.f90` input is checked against the repository fixture by -[`test_documentation_examples.py`](../../../tests/docs/test_examples.py). +[`test_examples.py`](../../../tests/docs/test_examples.py). Native artifact placement and runtime calls are checked by [`test_build_modes.py`](../../../tests/wrapper/fortran/build_from_source/test_build_modes.py) and diff --git a/docs/user/guide/allocatables.md b/docs/user/guide/allocatables.md index 4360355bd..0557adaba 100644 --- a/docs/user/guide/allocatables.md +++ b/docs/user/guide/allocatables.md @@ -57,10 +57,10 @@ def maybe_resize(values: Allocatable[Float64[:]] | None = ...) -> None: ... That spelling is valid only for optional callable arguments. Do not use `Allocatable[T[...]] | None` for module variables, derived-type fields, or function results; those surfaces return a present handle. Module variables, -fields, allocatable output dummies, and handles changed by later operations may -be unallocated. A native allocatable function result must be allocated when the -function returns; returning it unallocated is nonconforming Fortran and x2py -does not define a fallback for it. +fields, allocatable output dummies, function results, and handles changed by +later operations may be unallocated. The handle then reports +`allocated is False` and `to_numpy() is None`; this is descriptor state, not an +absent optional argument. Passing a handle to `Allocatable[T[...]]` passes the native descriptor. Passing the same allocated handle to a normal `T[...]` argument uses ordinary Fortran @@ -145,9 +145,8 @@ Passing `None` creates a present but unallocated call-local descriptor. Omitting a defaulted scalar descriptor argument creates native optional absence, so `present(scale)` is false. Passing a value creates a present allocated call-local descriptor. A projected output becomes `None` when its descriptor is -unallocated. An allocatable function result must instead be allocated and -defined when returned; an unallocated result is nonconforming native Fortran, -not a nullable x2py value. Ordinary scalar projection rules still apply: +unallocated, including an allocatable scalar function result. Ordinary scalar +projection rules still apply: `intent(out)` uses `Allocatable(Return("name", j))`, while `intent(inout)` uses `Allocatable(Arg(i))` plus a matching `Returns["name", T] | None` readback. The singular `result=Allocatable(Return(j))` mapping describes the native function @@ -261,6 +260,7 @@ def allocate_plain( def release_shared() -> None: ... +@native_call([Addr(Arg(0))]) def scale_plain( scale: Float64 ) -> None: ... @@ -388,7 +388,8 @@ end module character_names ``` The generated `.pyi` represents a fixed-length rank-one character array as -`String[4][::]`. A deferred-length allocatable rank-one array uses the two-axis +`String[n][::]`, where `n` is its fixed element length. A deferred-length +allocatable rank-one array uses the two-axis handle spelling `Allocatable[String[:][:]]`, so the element width can come from the native allocation at runtime: diff --git a/docs/user/guide/arrays.md b/docs/user/guide/arrays.md index 2fc707b4b..23f3c4b6b 100644 --- a/docs/user/guide/arrays.md +++ b/docs/user/guide/arrays.md @@ -35,12 +35,12 @@ contains values = values + 1.0_8 end subroutine shift - function automatic_vector(size) result(values) - integer(4), intent(in) :: size - real(8) :: values(size) + function automatic_vector(count) result(values) + integer(4), intent(in) :: count + real(8) :: values(count) integer(4) :: index - values = [(2.0_8 * index, index = 1, size)] + values = [(2.0_8 * index, index = 1, count)] end function automatic_vector end module array_ops ``` @@ -65,8 +65,8 @@ def shift( @native_call([Addr(Arg(0))]) def automatic_vector( - size: Int32 -) -> Float64[size]: ... + count: Int32 +) -> Float64[count]: ... ``` Build it: diff --git a/docs/user/guide/callbacks.md b/docs/user/guide/callbacks.md index ff690a132..62557a1b3 100644 --- a/docs/user/guide/callbacks.md +++ b/docs/user/guide/callbacks.md @@ -87,7 +87,7 @@ def update_values( values: Float64[count], ) -> None: ... -def apply_update(callback: update_values, ...) -> ...: ... +def apply_update(callback: update_values, count: Int32) -> None: ... ``` The spellings mean: diff --git a/docs/user/guide/distribution.md b/docs/user/guide/distribution.md index 2a08227b4..4061430e7 100644 --- a/docs/user/guide/distribution.md +++ b/docs/user/guide/distribution.md @@ -36,7 +36,7 @@ python3 -m x2py src/scale.f90 --out-dir build/scale python3 python/check_scale.py ``` -The asserted result remains `numpy.float64(7.5)`, as shown with the original +The asserted result remains the Python value `7.5`, as shown with the original source in the packaging example. Record the required Python and NumPy versions, compiler family, compiler flags, diff --git a/docs/user/guide/editing-semantic-pyi-contracts.md b/docs/user/guide/editing-semantic-pyi-contracts.md index c46082537..9d171926c 100644 --- a/docs/user/guide/editing-semantic-pyi-contracts.md +++ b/docs/user/guide/editing-semantic-pyi-contracts.md @@ -169,7 +169,10 @@ Each candidate is an independent declaration. Removing one candidate narrows runtime dispatch without removing the generic name: ```python -from x2py.contracts import Float64, Int32, overload +from x2py.contracts import Float64, Int32, overload, private + +@private +def convert_integer(value: Int32) -> Int32: ... @overload("convert_integer") def convert(value: Int32) -> Int32: ... @@ -251,7 +254,13 @@ def norm2(values: Float64[:]) -> Float64: ... Link every Python overload to one concrete native specific: ```python -from x2py.contracts import Float64, Int32, overload +from x2py.contracts import Float64, Int32, overload, private + +@private +def scale_integer(value: Int32) -> Int32: ... + +@private +def scale_real(value: Float64) -> Float64: ... @overload("scale_integer") def scale(value: Int32) -> Int32: ... @@ -264,7 +273,10 @@ To rename the Python overload group while calling an existing native generic, preserve the native generic explicitly: ```python -from x2py.contracts import Int32, overload +from x2py.contracts import Int32, overload, private + +@private +def convert_integer(value: Int32) -> Int32: ... @overload("convert_integer", generic="convert") def convert_number(value: Int32) -> Int32: ... @@ -279,9 +291,13 @@ silently choose a native procedure. An edited class may bind `__init__` to one concrete native initializer: ```python -from x2py.contracts import Addr, Arg, Int32, Pass, bind, native_call +from x2py.contracts import Addr, Arg, Int32, Pass, bind, native_call, private class state: + @private + @native_call([Pass(), Addr(Arg(0))]) + def init_state(self, size: Int32) -> None: ... + @bind("init_state") @native_call([Pass(), Addr(Arg(0))]) def __init__(self, size: Int32) -> None: ... @@ -437,10 +453,10 @@ Use `@raises(...)` to turn a projected native status into a Python exception: from x2py.contracts import Float64, Int32, Returns, String, raises @raises(status="status", message="message", success=0) -def solve(values: Float64[:]) -> Returns[ - "result", Float64[:], - "status", Int32, - "message", String, +def solve(values: Float64[:]) -> tuple[ + Returns["result", Float64[:]], + Returns["status", Int32], + Returns["message", String], ]: ... ``` @@ -473,7 +489,7 @@ Ownership edits use a complete policy triple: ```python from x2py.contracts import Annotated, Destruction, Float64, Ownership, Transfer -Annotated[ +values: Annotated[ Float64[:], Ownership("native"), Transfer("borrowed_view"), diff --git a/docs/user/guide/error-handling.md b/docs/user/guide/error-handling.md index 0e87c6433..18c7df1e8 100644 --- a/docs/user/guide/error-handling.md +++ b/docs/user/guide/error-handling.md @@ -44,7 +44,7 @@ Generate an editable contract package: python3 -m x2py solver.f90 --pyi --out contracts/solver ``` -In `contracts/solver/solver_api.pyi`, keep the generated native types and add +In `contracts/solver/solver.pyi`, keep the generated native types and add the explicit status policy: ```python @@ -63,7 +63,7 @@ Build that contract against the same simple native source: python3 -m x2py contracts/solver/__init__.pyi \ --wrap \ --native-fortran-sources solver.f90 \ - --out-dir build/solver \ + --out-dir build/solver ``` The success outputs are consumed, while a nonzero status becomes diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index e23d39e07..9b43e871b 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -358,9 +358,10 @@ The plan records: `link_items` is the order-sensitive representation. It can record objects, static archives, direct shared-library paths, named `-l` libraries, and explicit linker arguments without flattening them into one ambiguous string list. -Current CLI options expose objects, archives, shared libraries, named -libraries, library directories, and include/module directories. A general CLI -for arbitrary ordered linker arguments is planned in the later manifest stage. +The typed convenience options expose objects, archives, shared libraries, +named libraries, library directories, and include/module directories. +`--native-link-item KIND:VALUE` exposes the ordered form directly, including +explicit linker arguments, when caller order must be preserved. Example 1: a source-driven build records native source compilation separately from generated wrapper files. @@ -432,7 +433,7 @@ assert result.native_build_plan.library_dirs == (Path("vendor"),) ``` Example 5: the ordered representation can express linker control arguments for -future manifest and Makefile replay without pretending they are objects or +CLI, manifest, and Makefile replay without pretending they are objects or libraries. ```python @@ -1959,9 +1960,11 @@ Runtime tests: [`test_visibility_naming.py`](../../../tests/wrapper/fortran/nami ## Immediate Python Callbacks x2py supports dummy procedures invoked during the wrapped call. It resolves -local explicit interfaces and named abstract interfaces into a complete -callable contract containing argument order, types, intents, array ranks and -shapes, derived-type references, and optional result type. +local explicit interfaces and named abstract interfaces into named +`@prototype` declarations containing argument order, types, value/reference +transport, array ranks and shapes, derived-type references, and result type. +Source `intent` remains in the native interface when that interface must be +imported, but it is not repeated in the semantic prototype. ```fortran abstract interface @@ -1987,12 +1990,13 @@ thread are supported. ### Callback Values -- scalars use the matching Python numeric conversion; +- value scalars use the matching Python numeric conversion, while reference + scalars use writable rank-zero NumPy storage; - arrays require exact dtype, rank, declared shape, alignment, and Fortran contiguity; - derived values require the generated wrapper type; -- array and derived `intent(out)` or `intent(inout)` values are copied back - before the adapter returns; and +- reference scalar, array, character, and derived storage is handled + permissively and written back before the adapter returns; and - temporary NumPy views and borrowed derived wrappers passed to the callback are valid only during that callback invocation. @@ -2064,8 +2068,6 @@ def solve( ``` ```python -from x2py.contracts import raises - solve(values) # returns None when status == 0 solve(bad_values) # raises RuntimeError(message) otherwise ``` diff --git a/docs/user/guide/generic-interfaces.md b/docs/user/guide/generic-interfaces.md index 9d1987168..066309971 100644 --- a/docs/user/guide/generic-interfaces.md +++ b/docs/user/guide/generic-interfaces.md @@ -20,6 +20,8 @@ Create `generic.f90`: ```fortran module conversions implicit none + private + public :: convert interface convert module procedure convert_integer module procedure convert_real diff --git a/docs/user/guide/pointers.md b/docs/user/guide/pointers.md index d904afce5..51f758343 100644 --- a/docs/user/guide/pointers.md +++ b/docs/user/guide/pointers.md @@ -176,14 +176,31 @@ The generated semantic contract distinguishes the module descriptor, an ordinary array parameter, and a pointer-descriptor parameter: ```python -from x2py.contracts import Aliased, Annotated, Float64, Pointer, PointerAssociation +from x2py.contracts import ( + Aliased, + Annotated, + Destruction, + Float64, + Ownership, + Pointer, + PointerAssociation, + Transfer, +) storage: Annotated[Float64[3], Aliased] values: Annotated[Pointer[Float64[:]], PointerAssociation("runtime")] def associate_values() -> None: ... def sum_array(actual: Float64[::]) -> Float64: ... -def sum_pointer(actual: Pointer[Float64[:]]) -> Float64: ... +def sum_pointer( + actual: Annotated[ + Pointer[Float64[:]], + PointerAssociation("runtime"), + Ownership("caller"), + Transfer("call_local"), + Destruction("none"), + ] +) -> Float64: ... ``` Build it: diff --git a/docs/user/guide/wrapping-functions.md b/docs/user/guide/wrapping-functions.md index 0c5adfa58..eecac4345 100644 --- a/docs/user/guide/wrapping-functions.md +++ b/docs/user/guide/wrapping-functions.md @@ -62,12 +62,12 @@ Example (`function_results.f90`): module results implicit none contains - function squares(size) result(values) - integer(4), intent(in) :: size - real(8) :: values(size) + function squares(count) result(values) + integer(4), intent(in) :: count + real(8) :: values(count) integer(4) :: index - values = [(real(index, 8) * real(index, 8), index = 1, size)] + values = [(real(index, 8) * real(index, 8), index = 1, count)] end function squares end module results ``` @@ -79,8 +79,8 @@ from x2py.contracts import Addr, Arg, Float64, Int32, native_call @native_call([Addr(Arg(0))]) def squares( - size: Int32 -) -> Float64[size]: ... + count: Int32 +) -> Float64[count]: ... ``` Build it: diff --git a/tests/docs/test_examples.py b/tests/docs/test_examples.py index 49e0b14b3..6cd20a596 100644 --- a/tests/docs/test_examples.py +++ b/tests/docs/test_examples.py @@ -2,6 +2,7 @@ from __future__ import annotations +import ast from dataclasses import dataclass import os import platform @@ -13,12 +14,19 @@ import pytest +from x2py import pyi_text_to_semantic_module + ROOT = Path(__file__).parents[2] DOC_PATHS = [ ROOT / "README.md", *sorted(path for path in (ROOT / "docs").rglob("*.md") if "old_docs" not in path.parts), ] +AUDITED_PYTHON_DOC_PATHS = [ + ROOT / "README.md", + *sorted((ROOT / "docs/user/getting-started").glob("*.md")), + *sorted((ROOT / "docs/user/guide").glob("*.md")), +] TEST_MARKER = re.compile(r"^\s*\s*$") OUTPUT_MARKER = re.compile(r"^\s*\s*$") SOURCE_MARKER = re.compile(r"^\s*\s*$") @@ -63,6 +71,17 @@ def test_id(self) -> str: return f"{self.path.relative_to(ROOT)}:{self.line}" +@dataclass(frozen=True) +class DocumentedPythonBlock: + path: Path + line: int + source: str + + @property + def test_id(self) -> str: + return f"{self.path.relative_to(ROOT)}:{self.line}" + + def _platform_id() -> str: machine = platform.machine().lower() machine = {"amd64": "x86_64", "arm64": "aarch64"}.get(machine, machine) @@ -198,6 +217,24 @@ def _documented_content_from_path(path: Path) -> tuple[list[DocumentationExample DOCUMENTED_SOURCES = [source for _examples, sources in DOCUMENTATION_CONTENT for source in sources] +def _documented_python_blocks(path: Path) -> list[DocumentedPythonBlock]: + """Collect every visible Python fence for syntax and contract validation.""" + lines = _visible_documentation_lines(path) + blocks = [] + index = 0 + while index < len(lines): + if lines[index].strip() != "```python": + index += 1 + continue + source, after_block, _language = _fenced_block(lines, index, language="python") + blocks.append(DocumentedPythonBlock(path=path, line=index + 1, source=source)) + index = after_block + return blocks + + +DOCUMENTED_PYTHON_BLOCKS = [block for path in AUDITED_PYTHON_DOC_PATHS for block in _documented_python_blocks(path)] + + def _command_argv(example: DocumentationExample) -> list[str]: if example.language == "python": return [sys.executable, "-c", example.command] @@ -229,6 +266,14 @@ def test_documented_source_input(source: DocumentedSource): assert source.source_text.rstrip("\n") == source.source_path.read_text(encoding="utf-8").rstrip("\n") +@pytest.mark.parametrize("block", DOCUMENTED_PYTHON_BLOCKS, ids=lambda block: block.test_id) +def test_documented_python_block_is_valid(block: DocumentedPythonBlock): + """Keep Python examples parseable and semantic contract examples loadable.""" + ast.parse(block.source, filename=block.test_id) + if "from x2py.contracts import" in block.source: + pyi_text_to_semantic_module(block.source, module_name="documentation_example") + + @pytest.mark.parametrize("path", DOC_PATHS, ids=lambda path: str(path.relative_to(ROOT))) def test_documented_expected_output_labels_are_automatically_verified(path: Path): lines = _visible_documentation_lines(path) From 24fbc3638a93c6ff9ba658b070cb0e6e306f1674 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 06:52:38 +0100 Subject: [PATCH 21/30] update docs, put parsers in a new folder called parsers --- .github/workflows/parser-reference-guard.yml | 31 +++++-- CONTRIBUTING.md | 4 +- README.md | 1 - docs/developer/c-parser-reference.md | 16 ++-- docs/developer/development-workflow.md | 44 +++++----- docs/developer/feature-to-code-map.md | 8 +- docs/developer/fortran-parser-reference.md | 16 ++-- docs/developer/repository-structure.md | 13 ++- docs/developer/source-map.md | 35 ++++---- .../internal-architecture/pipeline-map.md | 10 +-- .../roadmap/semantic-pyi-wrapper-checklist.md | 2 +- .../test-suite-organization-checklist.md | 2 +- .../examples/recipes/build-and-import-cli.md | 5 +- .../recipes/build-multiple-fortran-sources.md | 1 - .../recipes/generate-editable-makefile.md | 8 +- .../recipes/semantic-pyi-contracts.md | 1 - .../guide/editing-semantic-pyi-contracts.md | 5 +- docs/user/guide/error-handling.md | 1 - docs/user/guide/fortran-wrapper.md | 26 +++--- docs/user/guide/packaging.md | 5 +- docs/user/reference/cli-commands.md | 21 ++--- docs/user/reference/configuration-files.md | 7 +- docs/user/reference/semantic-ir.md | 2 +- docs/user/reference/semantic-pyi-format.md | 23 ++--- docs/user/tutorials/basic-wrapper.md | 5 +- tests/README.md | 6 +- tests/_shared/fixture_outputs.py | 4 +- tests/_shared/parser_property_support.py | 4 +- tests/_shared/pyi_conversion_support.py | 2 +- tests/architecture/test_test_suite_layout.py | 6 +- tests/architecture/test_visitor_protocol.py | 6 +- tests/benchmarks/test_parser_benchmarks.py | 2 +- tests/cli/_cli_support.py | 2 +- tests/cli/test_argument_contract.py | 45 +++++----- tests/cli/test_output_contract.py | 4 +- tests/cli/test_stage_dispatch.py | 22 ++--- tests/cli/test_wrap_readiness.py | 2 +- tests/docs/test_structure.py | 38 ++++----- tests/parser/c/README.md | 2 +- .../errors/generate_c_parser_error_goldens.py | 2 +- tests/parser/c/generate_c_parser_goldens.py | 2 +- tests/parsing/c/test_c_cli_skeleton.py | 16 ++-- tests/parsing/c/test_c_compiler_extensions.py | 24 +++--- tests/parsing/c/test_c_corpus.py | 10 +-- .../c/test_c_declarations_and_declarators.py | 84 +++++++++---------- tests/parsing/c/test_c_error_fixture_suite.py | 2 +- tests/parsing/c/test_c_fixture_suite.py | 6 +- tests/parsing/c/test_c_functions.py | 58 ++++++------- tests/parsing/c/test_c_lexer_preprocessor.py | 50 +++++------ tests/parsing/c/test_c_model_serialization.py | 2 +- .../c/test_c_parser_developer_tutorial.py | 6 +- tests/parsing/c/test_c_project_resolution.py | 54 ++++++------ tests/parsing/c/test_c_public_api_skeleton.py | 40 ++++----- .../c/test_c_structs_unions_enums_typedefs.py | 52 ++++++------ tests/parsing/fortran/_procedure_support.py | 2 +- tests/parsing/fortran/_regression_support.py | 4 +- .../test_declaration_and_interface_edges.py | 4 +- .../fortran/test_declarations_and_shapes.py | 4 +- .../fortran/test_developer_tutorial.py | 4 +- .../fortran/test_public_entrypoints.py | 6 +- tests/parsing/fortran/test_scope_handling.py | 2 +- .../preprocessing/test_parser_boundaries.py | 4 +- .../semantics/conversion/_property_support.py | 2 +- tests/semantics/conversion/c/_support.py | 4 +- .../semantics/conversion/fortran/_support.py | 2 +- tests/semantics/policy/test_wrapper_policy.py | 2 +- tests/semantics/readiness/test_c_readiness.py | 14 ++-- tests/tools/test_check_radon_policy.py | 2 +- tests/wrapper/fortran/_support.py | 2 +- .../test_contract_package_runtime.py | 2 - .../build_from_pyi/test_pyi_wrapper_builds.py | 10 +-- .../build_from_source/test_runtime_abi.py | 1 - .../test_multi_source_builds.py | 2 - tools/wrapper_plan_staged_walkthrough.py | 2 +- x2py/README.md | 5 +- x2py/__init__.py | 10 +-- x2py/cli.py | 44 +++++----- x2py/parsers/README.md | 16 ++++ x2py/parsers/__init__.py | 3 + x2py/{c_parser => parsers/c}/README.md | 4 + x2py/{c_parser => parsers/c}/__init__.py | 0 x2py/{c_parser => parsers/c}/__main__.py | 0 x2py/{c_parser => parsers/c}/cli.py | 0 x2py/{c_parser => parsers/c}/lexer.py | 0 x2py/{c_parser => parsers/c}/models.py | 0 x2py/{c_parser => parsers/c}/parser.py | 0 x2py/{c_parser => parsers/c}/preprocessor.py | 0 x2py/{c_parser => parsers/c}/type_resolver.py | 0 .../fortran}/README.md | 4 + .../fortran}/__init__.py | 0 .../fortran}/__main__.py | 0 .../fortran}/cli.py | 2 +- .../fortran}/lexer.py | 0 .../fortran}/models.py | 0 .../fortran}/parser.py | 0 .../fortran}/type_resolver.py | 0 .../fortran}/utils.py | 0 x2py/{pyi_parser => parsers/pyi}/README.md | 3 + x2py/{pyi_parser => parsers/pyi}/__init__.py | 0 x2py/{pyi_parser => parsers/pyi}/parser.py | 0 x2py/pipeline/build.py | 2 +- x2py/pipeline/pyi.py | 2 +- x2py/probes/report.py | 4 +- x2py/semantics/c2ir.py | 2 +- x2py/semantics/fortran2ir.py | 2 +- x2py/semantics/pyi2ir.py | 2 +- 106 files changed, 525 insertions(+), 500 deletions(-) create mode 100644 x2py/parsers/README.md create mode 100644 x2py/parsers/__init__.py rename x2py/{c_parser => parsers/c}/README.md (88%) rename x2py/{c_parser => parsers/c}/__init__.py (100%) rename x2py/{c_parser => parsers/c}/__main__.py (100%) rename x2py/{c_parser => parsers/c}/cli.py (100%) rename x2py/{c_parser => parsers/c}/lexer.py (100%) rename x2py/{c_parser => parsers/c}/models.py (100%) rename x2py/{c_parser => parsers/c}/parser.py (100%) rename x2py/{c_parser => parsers/c}/preprocessor.py (100%) rename x2py/{c_parser => parsers/c}/type_resolver.py (100%) rename x2py/{fortran_parser => parsers/fortran}/README.md (88%) rename x2py/{fortran_parser => parsers/fortran}/__init__.py (100%) rename x2py/{fortran_parser => parsers/fortran}/__main__.py (100%) rename x2py/{fortran_parser => parsers/fortran}/cli.py (99%) rename x2py/{fortran_parser => parsers/fortran}/lexer.py (100%) rename x2py/{fortran_parser => parsers/fortran}/models.py (100%) rename x2py/{fortran_parser => parsers/fortran}/parser.py (100%) rename x2py/{fortran_parser => parsers/fortran}/type_resolver.py (100%) rename x2py/{fortran_parser => parsers/fortran}/utils.py (100%) rename x2py/{pyi_parser => parsers/pyi}/README.md (68%) rename x2py/{pyi_parser => parsers/pyi}/__init__.py (100%) rename x2py/{pyi_parser => parsers/pyi}/parser.py (100%) diff --git a/.github/workflows/parser-reference-guard.yml b/.github/workflows/parser-reference-guard.yml index 4584f7f28..d0ea4f810 100644 --- a/.github/workflows/parser-reference-guard.yml +++ b/.github/workflows/parser-reference-guard.yml @@ -48,11 +48,14 @@ jobs: C_DOC="docs/developer/c-parser-reference.md" FORTRAN_DOC="docs/developer/fortran-parser-reference.md" + PYI_DOC="docs/user/reference/semantic-pyi-format.md" C_DOC_CHANGED=false FORTRAN_DOC_CHANGED=false + PYI_DOC_CHANGED=false C_PARSER_CHANGED=false FORTRAN_PARSER_CHANGED=false + PYI_PARSER_CHANGED=false SHARED_PARSER_CHANGED=false if grep -Fxq "$C_DOC" <<< "$CHANGED_FILES"; then @@ -61,18 +64,25 @@ jobs: if grep -Fxq "$FORTRAN_DOC" <<< "$CHANGED_FILES"; then FORTRAN_DOC_CHANGED=true fi + if grep -Fxq "$PYI_DOC" <<< "$CHANGED_FILES"; then + PYI_DOC_CHANGED=true + fi while IFS= read -r changed_file; do case "$changed_file" in - c_parser/*|x2py/c_parser/*|tests/parser/c/*|tests/parsing/c/*|tests/data/c/*|tests/probes/test_c_types.py) + x2py/parsers/c/*|tests/parser/c/*|tests/parsing/c/*|tests/data/c/*|tests/probes/test_c_types.py) C_PARSER_CHANGED=true ;; - fortran_parser/*|x2py/fortran_parser/*|tests/parser/fortran/*|tests/parsing/fortran/*|tests/data/fortran/*|tests/probes/test_fortran_types.py) + x2py/parsers/fortran/*|tests/parser/fortran/*|tests/parsing/fortran/*|tests/data/fortran/*|tests/probes/test_fortran_types.py) FORTRAN_PARSER_CHANGED=true ;; + x2py/parsers/pyi/*|tests/parsing/pyi/*|tests/pipeline/pyi_builds/*) + PYI_PARSER_CHANGED=true + ;; tests/parser/conftest.py|\ tests/cli/*|\ tests/pipeline/preprocessing/*|\ + x2py/parsers/__init__.py|\ x2py/preprocessing.py) SHARED_PARSER_CHANGED=true ;; @@ -80,11 +90,11 @@ jobs: done <<< "$CHANGED_FILES" if [[ ",${PR_LABELS}," == *",${FORCE_LABEL},"* ]]; then - if [ "$C_DOC_CHANGED" = true ] || [ "$FORTRAN_DOC_CHANGED" = true ]; then + if [ "$C_DOC_CHANGED" = true ] || [ "$FORTRAN_DOC_CHANGED" = true ] || [ "$PYI_DOC_CHANGED" = true ]; then echo "${FORCE_LABEL} label present and at least one parser reference changed." else echo "${FORCE_LABEL} label present, but no parser reference changed." - echo "Update ${C_DOC} or ${FORTRAN_DOC}, or remove ${FORCE_LABEL}." + echo "Update ${C_DOC}, ${FORTRAN_DOC}, or ${PYI_DOC}, or remove ${FORCE_LABEL}." exit 1 fi fi @@ -101,11 +111,17 @@ jobs: FAILED=true fi + if [ "$PYI_PARSER_CHANGED" = true ] && [ "$PYI_DOC_CHANGED" != true ]; then + echo "Semantic .pyi parser-related files changed without updating ${PYI_DOC}." + FAILED=true + fi + if [ "$SHARED_PARSER_CHANGED" = true ] && \ [ "$C_DOC_CHANGED" != true ] && \ - [ "$FORTRAN_DOC_CHANGED" != true ]; then + [ "$FORTRAN_DOC_CHANGED" != true ] && \ + [ "$PYI_DOC_CHANGED" != true ]; then echo "Shared parser workflow files changed without updating a parser reference." - echo "Update ${C_DOC} or ${FORTRAN_DOC}, whichever behavior changed." + echo "Update ${C_DOC}, ${FORTRAN_DOC}, or ${PYI_DOC}, whichever behavior changed." FAILED=true fi @@ -120,5 +136,8 @@ jobs: if [ "$FORTRAN_DOC_CHANGED" = true ]; then echo "${FORTRAN_DOC} changed." fi + if [ "$PYI_DOC_CHANGED" = true ]; then + echo "${PYI_DOC} changed." + fi echo "Parser reference guard passed." diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 793f28ab6..425560d54 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,12 +11,12 @@ This repo includes a CI guard that may require updating parser reference docs when parser-related files change. -- **C parser changes**: if you change `x2py/c_parser/`, `tests/parser/c/`, or +- **C parser changes**: if you change `x2py/parsers/c/`, `tests/parser/c/`, or `tests/data/c/`, update `docs/c_parser.md` when the change affects the documented feature inventory, public API, diagnostics, fixtures, semantic handoff, or maintenance workflow. The guard also treats `tests/probes/test_c_types.py` as C parser related. -- **Fortran parser changes**: if you change `x2py/fortran_parser/`, +- **Fortran parser changes**: if you change `x2py/parsers/fortran/`, `tests/parser/fortran/`, or `tests/data/fortran/`, update `docs/fortran_parser.md` when the change affects the documented feature inventory, public API, diagnostics, fixtures, semantic handoff, or diff --git a/README.md b/README.md index f9bcf17b8..c9e1e1b7a 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,6 @@ same native implementation source: ```bash python3 -m x2py contracts/__init__.pyi \ - --wrap \ --native-fortran-sources scale.f90 \ --out SCALE \ --out-dir build/SCALE_from_pyi diff --git a/docs/developer/c-parser-reference.md b/docs/developer/c-parser-reference.md index 33cad4876..bd859a891 100644 --- a/docs/developer/c-parser-reference.md +++ b/docs/developer/c-parser-reference.md @@ -14,7 +14,7 @@ status: maintained X2PY_C_DOCS_END --> ## Parser Organization Notes @@ -471,7 +471,7 @@ X2PY_C_DOCS_END --> @@ -484,7 +484,7 @@ Implemented top-level and package entrypoints: ```python from x2py import parse_c_file, parse_c_project # Equivalent parser-package imports remain available: -# from x2py.c_parser import parse_c_file, parse_c_project +# from x2py.parsers.c import parse_c_file, parse_c_project ``` X2PY_C_DOCS_END --> diff --git a/docs/developer/development-workflow.md b/docs/developer/development-workflow.md index e807e6b2e..05b94f012 100644 --- a/docs/developer/development-workflow.md +++ b/docs/developer/development-workflow.md @@ -203,13 +203,13 @@ implementation files. | User-visible area | Main implementation files | Main tests | | --- | --- | --- | -| Fortran parse output | `x2py/fortran_parser/parser.py`, `x2py/fortran_parser/models.py`, `x2py/fortran_parser/lexer.py` | `tests/parsing/fortran/`, `tests/parsing/fortran/test_fortran_fixture_suite.py`, `tests/parsing/fortran/test_error_handling.py` | -| CLI stage selection and output | `x2py/cli.py`, `x2py/fortran_parser/cli.py` | `tests/cli/` | +| Fortran parse output | `x2py/parsers/fortran/parser.py`, `x2py/parsers/fortran/models.py`, `x2py/parsers/fortran/lexer.py` | `tests/parsing/fortran/`, `tests/parsing/fortran/test_fortran_fixture_suite.py`, `tests/parsing/fortran/test_error_handling.py` | +| CLI stage selection and output | `x2py/cli.py`, `x2py/parsers/fortran/cli.py` | `tests/cli/` | | Fortran target type probing and cache | `x2py/probes/fortran_types.py` | `tests/probes/test_fortran_types.py` | | Generated target datatype mapping examples | `x2py/probes/report.py` | `tests/types/test_mapping_report.py`, `tests/docs/test_examples.py` | | Fortran to semantic IR | `x2py/semantics/fortran2ir.py`, `x2py/semantics/models.py` | `tests/semantics/conversion/fortran/` | | `.pyi` printing | `x2py/wrapper_codegen/printers/pyi_printer.py` | `tests/wrapper_codegen/printers/`, `tests/wrapper_codegen/printers/test_modern_example.py` | -| `.pyi` parsing/loading/editing | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py` | `tests/parsing/pyi/`, `tests/pipeline/pyi_builds/test_contract_fixtures.py` | +| `.pyi` parsing/loading/editing | `x2py/parsers/pyi/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py` | `tests/parsing/pyi/`, `tests/pipeline/pyi_builds/test_contract_fixtures.py` | | Semantic policy completion | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py` | `tests/semantics/policy/` | | Readiness reports | `x2py/semantics/readiness.py` | `tests/semantics/readiness/`, `tests/semantics/readiness/test_wrap_readiness_fixture_suite.py` | | Fortran wrapper orchestration | `x2py/pipeline/build.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py` | @@ -218,7 +218,7 @@ implementation files. | Executable Markdown examples | `README.md`, `docs/*.md` | `tests/docs/test_examples.py` | Organize generators and printers using `FortranParser` in -`x2py/fortran_parser/parser.py` as the structural reference. A developer +`x2py/parsers/fortran/parser.py` as the structural reference. A developer should be able to read each class from top to bottom in the same order that data moves through it: @@ -278,7 +278,7 @@ module-level function only to preserve an old internal call path. ### `.pyi` Contract Internals User-visible `.pyi` syntax is first parsed to Python AST by -`x2py/pyi_parser/parser.py`, loaded from text/files by +`x2py/parsers/pyi/parser.py`, loaded from text/files by `x2py/pipeline/pyi.py`, converted to semantic IR by `x2py/semantics/pyi2ir.py`, and printed by `x2py/wrapper_codegen/printers/pyi_printer.py`. The converter and printer operate on @@ -447,7 +447,7 @@ C files and directories require explicit language selection. Keep this behavior tested in `tests/cli/` whenever stage selection changes. X2PY_C_DOCS_END --> -The package-specific `x2py/fortran_parser/cli.py` remains for the Fortran parser +The package-specific `x2py/parsers/fortran/cli.py` remains for the Fortran parser package entrypoint. New cross-language user behavior normally belongs in `x2py/cli.py`. @@ -677,7 +677,7 @@ CLI `.pyi` readiness: ```text .pyi path(s) or directory - -> x2py/pyi_parser/parser.py + -> x2py/parsers/pyi/parser.py -> x2py/pipeline/pyi.py pyi_paths_to_semantic_modules(...) -> x2py/semantics/pyi2ir.py -> SemanticModule list @@ -714,7 +714,7 @@ report = assess_semantic_wrap_readiness(modules, source="interfaces") Use the `.pyi` helpers by input shape: -- `parse_pyi_text(source, filename=...)` from `x2py.pyi_parser` for parser-only +- `parse_pyi_text(source, filename=...)` from `x2py.parsers.pyi` for parser-only AST parsing. - `convert_pyi_to_ir(tree, module_name=..., source=...)` from `pyi2ir.py` for AST-to-IR conversion. @@ -892,9 +892,9 @@ rather than "what Python wrapper should be generated?" Fortran: -- `x2py/fortran_parser/parser.py` slices the file into grammar units, then parses +- `x2py/parsers/fortran/parser.py` slices the file into grammar units, then parses each unit's specification region. -- `x2py/fortran_parser/models.py` stores `FortranFile`, modules, procedures, +- `x2py/parsers/fortran/models.py` stores `FortranFile`, modules, procedures, variables, derived types, interfaces, programs, submodules, and diagnostics. - Execution bodies are intentionally skipped after the parser has enough signature/source facts. @@ -904,11 +904,11 @@ C: X2PY_C_DOCS_END --> @@ -930,7 +930,7 @@ X2PY_C_DOCS_END --> variables, kinds, shapes, storage contracts, visibility, imported references, and compile-time values. - `x2py/wrapper_codegen/printers/pyi_printer.py` emits editable user contracts. -- `x2py/pyi_parser/parser.py` parses edited contracts to Python AST. +- `x2py/parsers/pyi/parser.py` parses edited contracts to Python AST. - `x2py/pipeline/pyi.py` converts edited contract text, files, and path sets. - `x2py/semantics/pyi2ir.py` converts parsed `.pyi` AST back into semantic IR. - `x2py/semantics/native_contract.py` validates immutable native scope, ABI, @@ -1093,11 +1093,11 @@ X2PY_C_DOCS_END --> `tests/parsing/c/test_c_declarations_and_declarators.py`, `tests/parsing/c/test_c_compiler_extensions.py`, or `tests/parsing/c/test_c_structs_unions_enums_typedefs.py`. -2. Implement the parser change in `x2py/c_parser/parser.py`. Add or update model - fields in `x2py/c_parser/models.py` only if the serialized parser contract needs +2. Implement the parser change in `x2py/parsers/c/parser.py`. Add or update model + fields in `x2py/parsers/c/models.py` only if the serialized parser contract needs new facts. 3. If source splitting or raw directive handling changes, update - `x2py/c_parser/lexer.py` and `tests/parsing/c/test_c_lexer_preprocessor.py`. + `x2py/parsers/c/lexer.py` and `tests/parsing/c/test_c_lexer_preprocessor.py`. 4. If project-level resolution changes, update `tests/parsing/c/test_c_project_resolution.py`. 5. If parser JSON changes intentionally, regenerate the relevant project @@ -1141,8 +1141,8 @@ metadata item. `tests/parsing/fortran/`, `tests/parsing/fortran/test_scope_handling.py`, or `tests/pipeline/preprocessing/test_parser_boundaries.py`. -2. Implement parsing in `x2py/fortran_parser/parser.py`. Add model fields in - `x2py/fortran_parser/models.py` only if the parser output needs to expose the +2. Implement parsing in `x2py/parsers/fortran/parser.py`. Add model fields in + `x2py/parsers/fortran/models.py` only if the parser output needs to expose the new fact. 3. Add parser diagnostic coverage in `tests/parsing/fortran/test_error_handling.py` if malformed source should now fail differently. @@ -1219,7 +1219,7 @@ Example target: add a new `Annotated[...]` metadata item or projection helper. 1. Add loader tests in `tests/parsing/pyi/`. 2. Update `x2py/semantics/pyi2ir.py`. Update `x2py/pipeline/pyi.py` when loading or cross-file reconciliation changes. Update - `x2py/pyi_parser/parser.py` only when the raw Python AST parsing boundary + `x2py/parsers/pyi/parser.py` only when the raw Python AST parsing boundary changes. 3. Add printer tests in `tests/wrapper_codegen/printers/`. 4. Update `x2py/wrapper_codegen/printers/pyi_printer.py`. @@ -1290,7 +1290,7 @@ diagnostic formatting. 1. Add CLI tests in `tests/cli/` first. 2. Implement shared dispatch and output behavior in `x2py/cli.py`. -3. Keep Fortran package-specific CLI behavior in `x2py/fortran_parser/cli.py`. +3. Keep Fortran package-specific CLI behavior in `x2py/parsers/fortran/cli.py`. 4. If compiler preprocessing behavior changes, update `x2py/pipeline/preprocessing.py` and preprocessing tests. 5. Update [Basic wrapper tutorial](../user/tutorials/basic-wrapper.md) or [Verified examples cookbook](../user/examples/verified-cookbook.md) for diff --git a/docs/developer/feature-to-code-map.md b/docs/developer/feature-to-code-map.md index 26678aacf..477292926 100644 --- a/docs/developer/feature-to-code-map.md +++ b/docs/developer/feature-to-code-map.md @@ -16,9 +16,9 @@ before documentation may call the behavior supported. | Feature or behavior | Public docs | Main implementation files | Focused tests | Support evidence | | --- | --- | --- | --- | --- | -| Fortran parse output | `docs/developer/fortran-parser-reference.md` | `x2py/fortran_parser/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/parsing/fortran/`, `tests/parsing/fortran/test_fortran_fixture_suite.py` | Parser facts and diagnostics match fixtures | +| Fortran parse output | `docs/developer/fortran-parser-reference.md` | `x2py/parsers/fortran/parser.py`, `models.py`, `lexer.py`, `type_resolver.py` | `tests/parsing/fortran/`, `tests/parsing/fortran/test_fortran_fixture_suite.py` | Parser facts and diagnostics match fixtures | | Semantic `.pyi` generation | `docs/user/reference/semantic-pyi-format.md` | `x2py/wrapper_codegen/printers/pyi_printer.py` | `tests/wrapper_codegen/printers/`, `tests/wrapper_codegen/printers/test_modern_example.py` | Printed `.pyi` round-trips or matches fixtures | -| Semantic `.pyi` conversion and editing | `docs/user/guide/editing-semantic-pyi-contracts.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `models.py` | `tests/parsing/pyi/`, `tests/pipeline/pyi_builds/test_contract_fixtures.py` | Edited contracts parse to Python AST, then become semantic IR with preserved native facts | +| Semantic `.pyi` conversion and editing | `docs/user/guide/editing-semantic-pyi-contracts.md`, `docs/user/reference/semantic-pyi-format.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `x2py/parsers/pyi/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `models.py` | `tests/parsing/pyi/`, `tests/pipeline/pyi_builds/test_contract_fixtures.py` | Edited contracts parse to Python AST, then become semantic IR with preserved native facts | | Readiness blockers | `docs/user/reference/diagnostic-codes.md`, `docs/user/reference/semantic-pyi-format.md` | `x2py/semantics/readiness.py` | `tests/semantics/readiness/`, readiness fixture tests | Unsupported or incomplete contracts fail before codegen | | Fortran wrapper orchestration | `docs/user/guide/fortran-wrapper.md`, `docs/user/examples/recipes/build-and-import-cli.md`, `docs/user/examples/recipes/build-multiple-fortran-sources.md` | `x2py/pipeline/build.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, multi-source wrapper tests | Builds report artifacts and compile/link as documented | | Completed semantic policy to wrapper artifacts | `docs/user/guide/fortran-wrapper.md` | `x2py/semantics/policy_completion.py`, `x2py/wrapper_codegen/plan.py`, `planner.py`, `generator.py` | `tests/semantics/policy/`, `tests/wrapper_codegen/`, `tests/wrapper/fortran/` | Runtime policy is explicit, the typed plan is complete, and generated artifacts compile and run | @@ -26,9 +26,9 @@ before documentation may call the behavior supported. | Source documentation structure | `docs/developer/source-map.md` | `docs/`, package README files, `tests/docs/test_structure.py` | documentation structure and example tests | Pages have metadata, audience separation, and source coverage checks | - The major source packages have local README files under `x2py/` for developers reading directly in the source tree. Those README files should link back to the maintained source-navigation docs instead of old top-level docs. @@ -40,9 +36,10 @@ back to the maintained source-navigation docs instead of old top-level docs. Only `x2py/__init__.py`, `x2py/__main__.py`, and `x2py/cli.py` live directly at the package root. Public library symbols are deliberately flattened through `x2py/__init__.py`; internal modules are imported through their owning package. -The one public submodule namespace is `x2py.contracts`, because semantic `.pyi` -files use direct `from x2py.contracts import ...` declarations as part of their -contract syntax. +The deliberate public submodule namespaces are `x2py.contracts`, whose import +path is part of semantic `.pyi` syntax, and `x2py.parsers`, which groups the +language-specific frontends. Stable convenience functions remain flattened +through `x2py/__init__.py`. ## Tests diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index 3e4ecd13f..978f135da 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -38,8 +38,8 @@ change crosses ownership boundaries. | --- | --- | --- | --- | | CLI flags, stage selection, output formatting, diagnostics | `x2py/cli.py` | `docs/user/reference/cli-commands.md`, `docs/user/tutorials/basic-wrapper.md`, `docs/user/examples/verified-cookbook.md` | `tests/cli/`, `tests/docs/test_examples.py` | | Compiler preprocessing, include paths, macros, and target flags | `x2py/pipeline/preprocessing.py` | `docs/user/examples/recipes/compiler-preprocessing.md`, `docs/developer/fortran-parser-reference.md` | `tests/pipeline/preprocessing/`, `tests/pipeline/preprocessing/test_parser_boundaries.py` | -| Fortran parser facts and diagnostics | `x2py/fortran_parser/parser.py` | `docs/developer/fortran-parser-reference.md`, `docs/user/examples/recipes/inspect-fortran-api.md` | `tests/parser/`, `tests/parsing/fortran/test_fortran_fixture_suite.py` | -| Semantic `.pyi` parsing, conversion, printing, package generation, and round-trip behavior | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md`, `docs/user/guide/editing-semantic-pyi-contracts.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `tests/pyi/`, `tests/pipeline/pyi_builds/test_contract_package_generation.py`, `tests/wrapper_codegen/printers/` | +| Fortran parser facts and diagnostics | `x2py/parsers/fortran/parser.py` | `docs/developer/fortran-parser-reference.md`, `docs/user/examples/recipes/inspect-fortran-api.md` | `tests/parser/`, `tests/parsing/fortran/test_fortran_fixture_suite.py` | +| Semantic `.pyi` parsing, conversion, printing, package generation, and round-trip behavior | `x2py/parsers/pyi/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md`, `docs/user/guide/editing-semantic-pyi-contracts.md`, `docs/user/examples/recipes/semantic-pyi-contracts.md` | `tests/pyi/`, `tests/pipeline/pyi_builds/test_contract_package_generation.py`, `tests/wrapper_codegen/printers/` | | Readiness blockers and support claims | `x2py/semantics/readiness.py` | `docs/user/reference/diagnostic-codes.md`, `docs/user/language-support/feature-matrix.md` | `tests/semantics/readiness/`, readiness fixture tests | | Source-driven Fortran wrapper orchestration | `x2py/pipeline/build.py` | `docs/user/guide/fortran-wrapper.md`, `docs/user/examples/recipes/build-and-import-cli.md` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/multiple_files/test_multi_source_builds.py` | | Semantic `.pyi` wrapper orchestration from native artifacts | `x2py/pipeline/build.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py` | `docs/user/guide/fortran-wrapper.md`, `docs/user/reference/semantic-pyi-format.md` | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py`, `tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py` | @@ -54,7 +54,7 @@ change crosses ownership boundaries. X2PY_C_DOCS_END --> | `x2py/probes/` | Compiler-derived target facts plus mapping reports | `fortran_types.py`, `report.py` | target probe and type mapping report tests | | `x2py/runtime/` | Python runtime objects consumed by generated extensions | `handles.py` | runtime handle and wrapper runtime tests | | `x2py/types/` | Semantic-to-Python ecosystem type mappings | `numpy.py` | `tests/types/test_numpy.py` | -| `x2py/fortran_parser/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | +| `x2py/parsers/` | Public namespace for language and semantic `.pyi` frontends | child parser packages | `tests/parsing/`, parser references, semantic `.pyi` reference | +| `x2py/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | | `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `basic.py`, `compilers.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | | `x2py/stdlib/` | Native runtime support files copied into generated wrapper builds | `x2py_runtime/` | wrapper runtime tests | | `x2py/utilities/` | Small shared Python utilities | `metaclasses.py`, `strings.py` | tests that exercise callers | @@ -136,7 +137,7 @@ For source-driven Fortran wrappers, read in this order: x2py/cli.py -> x2py/pipeline/build.py -> x2py/pipeline/preprocessing.py - -> x2py/fortran_parser/parser.py + -> x2py/parsers/fortran/parser.py -> x2py/probes/fortran_types.py -> x2py/semantics/fortran2ir.py -> x2py/semantics/policy_completion.py @@ -153,7 +154,7 @@ X2PY_C_DOCS_END --> For semantic `.pyi` builds, the parser branch is replaced by: ```text -x2py/pyi_parser/parser.py +x2py/parsers/pyi/parser.py -> x2py/pipeline/pyi.py -> x2py/semantics/pyi2ir.py -> x2py/semantics/policy_completion.py @@ -170,7 +171,7 @@ X2PY_C_DOCS_END --> The hardest source packages also have local README files: - `x2py/README.md` -- `x2py/fortran_parser/README.md` +- `x2py/parsers/README.md` +- `x2py/parsers/fortran/README.md` +- `x2py/parsers/pyi/README.md` - `x2py/semantics/README.md` - `x2py/compiling/README.md` Keep these files short. They should tell developers where to enter the code, diff --git a/docs/maintainer/internal-architecture/pipeline-map.md b/docs/maintainer/internal-architecture/pipeline-map.md index db879c404..36518ebc2 100644 --- a/docs/maintainer/internal-architecture/pipeline-map.md +++ b/docs/maintainer/internal-architecture/pipeline-map.md @@ -40,7 +40,7 @@ X2PY_C_DOCS_END --> | CLI request | `x2py/cli.py` | source paths and stage flags | selected stage or wrapper build options | `tests/cli/` | | Build orchestration | `x2py/pipeline/build.py` | ordered Fortran sources or `.pyi` contracts plus explicit native artifacts | `WrapperBuildResult`, `NativeBuildPlan`, and generated artifact plan | wrapper build-mode tests | | Preprocessing | `x2py/pipeline/preprocessing.py` | source path, compiler config | preprocessed source and dependency facts | preprocessing tests | -| Parser project model | `x2py/fortran_parser/parser.py` | preprocessed Fortran source | parser project with modules, procedures, types, visibility | Fortran parser fixture tests | +| Parser project model | `x2py/parsers/fortran/parser.py` | preprocessed Fortran source | parser project with modules, procedures, types, visibility | Fortran parser fixture tests | | Target probes | `x2py/probes/fortran_types.py` | semantic type requirements and compiler flags | resolved kind/storage facts | Fortran type probe tests | | Semantic IR | `x2py/semantics/fortran2ir.py` | parser project and target facts | `SemanticModule` objects | semantic Fortran tests | | Semantic policy completion | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py` | full semantic modules with signatures and `.pyi` overrides | semantic modules annotated with every ownership, transfer, destruction, mutability, storage, accessor, and projection decision needed by wrapper generation | ownership-policy and readiness tests | @@ -157,13 +157,13 @@ X2PY_C_DOCS_END --> | --- | --- | --- | | CLI and output routing | `x2py/cli.py`, parser CLI helpers | `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md` | | Source loading and preprocessing | `x2py/pipeline/preprocessing.py` | `docs/developer/source-map.md`, parser references | -| Editable semantic contracts | `x2py/pyi_parser/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | +| Editable semantic contracts | `x2py/parsers/pyi/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | | Readiness | `x2py/semantics/readiness.py` | `docs/user/reference/diagnostic-codes.md` | | Wrapper policy and lowering | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py`, `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/generator.py` | `docs/user/guide/fortran-wrapper.md`, ownership docs | | Native build | `x2py/pipeline/build.py`, `x2py/compiling/compilers.py`, `x2py/compiling/runtime_support.py` | compiling package README and build-system docs | @@ -176,7 +176,7 @@ the Python API. ```text .pyi contract - -> x2py/pyi_parser/parser.py + -> x2py/parsers/pyi/parser.py -> x2py/pipeline/pyi.py -> x2py/semantics/pyi2ir.py -> x2py/semantics/native_contract.py @@ -210,7 +210,7 @@ X2PY_C_DOCS_END --> ```text C parser -> x2py/semantics/c2ir.py Fortran parser -> x2py/semantics/fortran2ir.py -.pyi parser -> x2py/pyi_parser/parser.py -> x2py/pipeline/pyi.py -> x2py/semantics/pyi2ir.py +.pyi parser -> x2py/parsers/pyi/parser.py -> x2py/pipeline/pyi.py -> x2py/semantics/pyi2ir.py -> SemanticModule objects -> x2py/semantics/policy_completion.py -> readiness and lowering diff --git a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md b/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md index 9a0f3a17f..e03318bb5 100644 --- a/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md +++ b/docs/maintainer/roadmap/semantic-pyi-wrapper-checklist.md @@ -415,7 +415,7 @@ X2PY_C_DOCS_END --> `tests/semantics/readiness/`, and `x2py/semantics/README.md`. - [x] `.pyi` parsing and `.pyi` semantic conversion are separate stages: - `x2py/pyi_parser/parser.py` parses text/files to Python AST, and + `x2py/parsers/pyi/parser.py` parses text/files to Python AST, and `x2py/semantics/pyi2ir.py` converts that AST into `SemanticModule` objects before semantic policy completion runs. Evidence: `tests/parsing/pyi/test_python_ast_contracts.py::test_pyi_parser_returns_python_ast_only`, diff --git a/docs/maintainer/roadmap/test-suite-organization-checklist.md b/docs/maintainer/roadmap/test-suite-organization-checklist.md index 28ab807e8..5cda84406 100644 --- a/docs/maintainer/roadmap/test-suite-organization-checklist.md +++ b/docs/maintainer/roadmap/test-suite-organization-checklist.md @@ -25,7 +25,7 @@ has been recorded here. - [x] Exclude LAPACK runtime execution from local verification. - [x] Preserve unrelated dirty-worktree changes. Initial audit: worktree clean. - [x] No executable product behavior changed. The only product-module edit is a - path-only documentation-string update in `x2py/c_parser/parser.py`. + path-only documentation-string update in `x2py/parsers/c/parser.py`. ## Baseline collection evidence diff --git a/docs/user/examples/recipes/build-and-import-cli.md b/docs/user/examples/recipes/build-and-import-cli.md index 215b1e315..c1f5cd927 100644 --- a/docs/user/examples/recipes/build-and-import-cli.md +++ b/docs/user/examples/recipes/build-and-import-cli.md @@ -29,13 +29,12 @@ end module fruntime_abi_f90 ```bash python3 -m x2py tests/data/fortran/wrapper/fruntime_abi_f90.f90 \ - --wrap \ --out-dir build/fruntime_abi \ --json ``` -Recognizable Fortran sources default to `--wrap` when no inspection stage is -selected, so this is equivalent: +Recognizable Fortran sources select the wrapper build when no inspection stage +is selected, so this is equivalent: ```bash python3 -m x2py tests/data/fortran/wrapper/fruntime_abi_f90.f90 \ diff --git a/docs/user/examples/recipes/build-multiple-fortran-sources.md b/docs/user/examples/recipes/build-multiple-fortran-sources.md index 32ddca8dd..dbd26c6f2 100644 --- a/docs/user/examples/recipes/build-multiple-fortran-sources.md +++ b/docs/user/examples/recipes/build-multiple-fortran-sources.md @@ -20,7 +20,6 @@ merged extension: python3 -m x2py \ tests/data/fortran/wrapper/first_api.f90 \ tests/data/fortran/wrapper/second_api.f90 \ - --wrap \ --out-dir build/multi_api \ --json ``` diff --git a/docs/user/examples/recipes/generate-editable-makefile.md b/docs/user/examples/recipes/generate-editable-makefile.md index ac0fca4e4..b26436828 100644 --- a/docs/user/examples/recipes/generate-editable-makefile.md +++ b/docs/user/examples/recipes/generate-editable-makefile.md @@ -17,7 +17,6 @@ manifest is the source of truth used to generate the Makefile. ```bash python3 -m x2py tests/data/fortran/wrapper/fruntime_abi_f90.f90 \ - --wrap \ --makefile \ --out-dir build/fruntime_abi \ --json @@ -31,7 +30,6 @@ mode with explicit native inputs: ```bash python3 -m x2py contracts/fruntime_abi_f90.pyi \ - --wrap \ --native-fortran-sources native/fruntime_abi_f90.f90 \ --native-fortran-flags="-O3 -fopenmp" \ --out-dir build/fruntime_abi \ @@ -68,11 +66,11 @@ X2PY_C_DOCS_END --> ## Notes - `--makefile` generates the build plan without compiling immediately. -- `--makefile` is a wrapper-build option and must be used with `--wrap`. +- `--makefile` selects the editable wrapper-build mode directly. - `--makefile` and `--verbose` are mutually exclusive. - `.pyi` Makefile generation is replayable through - `python3 -m x2py --build-manifest build/fruntime_abi/x2py-build.json --wrap --makefile` + `python3 -m x2py --build-manifest build/fruntime_abi/x2py-build.json --makefile` or buildable through - `python3 -m x2py --build-manifest build/fruntime_abi/x2py-build.json --wrap`. + `python3 -m x2py --build-manifest build/fruntime_abi/x2py-build.json`. - User Fortran sources remain in caller-provided order. Generated independent objects may be built in parallel by Make. diff --git a/docs/user/examples/recipes/semantic-pyi-contracts.md b/docs/user/examples/recipes/semantic-pyi-contracts.md index 8dd3318a0..dba18a36c 100644 --- a/docs/user/examples/recipes/semantic-pyi-contracts.md +++ b/docs/user/examples/recipes/semantic-pyi-contracts.md @@ -34,7 +34,6 @@ you provide the native artifacts explicitly: ```bash python3 -m x2py path/to/module.pyi \ - --wrap \ --native-objects path/to/module.o path/to/support.a \ --native-include-dir path/to/mod-files path/to/vendor-mod-files \ --out-dir build/module diff --git a/docs/user/guide/editing-semantic-pyi-contracts.md b/docs/user/guide/editing-semantic-pyi-contracts.md index 9d171926c..ce4a8a186 100644 --- a/docs/user/guide/editing-semantic-pyi-contracts.md +++ b/docs/user/guide/editing-semantic-pyi-contracts.md @@ -40,14 +40,13 @@ Build the edited entry contract with the same native implementation artifacts: ```bash python3 -m x2py contracts/edited_solver/__init__.pyi \ - --wrap \ --native-objects build/solver.o \ --native-include-dir build/mod \ --out-dir build/edited-solver ``` -The explicit `--wrap` is required here because the entry input is a semantic -`.pyi` contract, not a Fortran source file. +The semantic `.pyi` entry contract selects the wrapper build automatically; +the native artifact options provide the implementation to compile or link. The entry `.pyi` is the sole semantic input to wrapper generation. x2py does not reparse the native source to restore a removed declaration, projection, or diff --git a/docs/user/guide/error-handling.md b/docs/user/guide/error-handling.md index 18c7df1e8..22d40223e 100644 --- a/docs/user/guide/error-handling.md +++ b/docs/user/guide/error-handling.md @@ -61,7 +61,6 @@ Build that contract against the same simple native source: ```bash python3 -m x2py contracts/solver/__init__.pyi \ - --wrap \ --native-fortran-sources solver.f90 \ --out-dir build/solver ``` diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 9b43e871b..91f5f16e2 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -207,11 +207,11 @@ ownership, and destruction, is explained later in Editing Semantic `.pyi` Contracts. The complete grammar appears later in the Semantic `.pyi` Format reference. The normal CLI build is source-driven: recognizable Fortran sources build -wrappers without a stage flag and cannot be combined with `--pyi`. For the -implemented `.pyi` subset, pass `--wrap` with a semantic `.pyi` file and native -build artifacts such as `.o`, `.a`, or `.so` inputs. In that mode the `.pyi` is -the Python API source of truth; native source is not reparsed during wrapper -generation. +wrappers without a stage flag and cannot be combined with `--pyi`. A semantic +`.pyi` entry contract also selects the wrapper stage automatically when its +native build artifacts, such as `.o`, `.a`, or `.so` inputs, are supplied. In +that mode the `.pyi` is the Python API source of truth; native source is not +reparsed during wrapper generation. The current `.pyi` build subset requires the contract filename stem to match the native Fortran module name. Supply the native module file directory as an @@ -219,7 +219,6 @@ include directory when the generated bridge contains `use `: ```bash python3 -m x2py path/to/module.pyi \ - --wrap \ --native-objects path/to/module.o \ --native-include-dir path/to/mod-files \ --out-dir build/module @@ -245,14 +244,13 @@ replayed directly: ```bash python3 -m x2py contracts/module.pyi \ - --wrap \ --native-fortran-sources native/module.f90 \ --native-fortran-flags="-O3 -fopenmp" \ --out-dir build/module \ --makefile -python3 -m x2py --build-manifest build/module/x2py-build.json --wrap -python3 -m x2py --build-manifest build/module/x2py-build.json --wrap --makefile +python3 -m x2py --build-manifest build/module/x2py-build.json +python3 -m x2py --build-manifest build/module/x2py-build.json --makefile ``` Edited `.pyi` contracts may expose the native call shape directly. If every @@ -285,7 +283,7 @@ Runtime tests: [`test_pyi_wrapper_builds.py`](../../../tests/wrapper/fortran/bui Use `--verbose` to execute a build while printing every exact, shell-escaped compiler and linker command. Verbose builds also print elapsed time for each compiler/linker command and for the wrapper creation, printing, and compilation -stages. Use `--wrap --makefile` to generate an editable `Makefile.x2py` without +stages. Use `--makefile` to generate an editable `Makefile.x2py` without compiling. These modes are mutually exclusive. -The normal `--wrap` workflow remains source-driven and accepts Fortran source -files. A `.pyi`-driven wrapper workflow is also available for the implemented -subset: pass the semantic `.pyi` file as the wrapper input and provide native -object, archive, shared-library, module, include, and link inputs with the -native artifact flags. This path treats the `.pyi` as the source of truth for -the Python API and does not reparse native source to reconstruct the contract. +The normal wrapper workflow accepts recognizable Fortran source without a +stage flag. A `.pyi`-driven wrapper workflow is also available for the +implemented subset: pass the semantic `.pyi` file as the wrapper input and +provide native object, archive, shared-library, module, include, and link +inputs with the native artifact flags. The `.pyi` input selects the wrapper +stage automatically and remains the source of truth for the Python API; native +source is not reparsed to reconstruct the contract. The implemented subset and remaining parity limits are stated in this reference and summarized later in Language Support. @@ -35,13 +36,17 @@ Status terms used below: - **Generated**: emitted today by `--pyi` or `wrapper_codegen.printers.pyi_printer`. -- **Loaded**: accepted today by `x2py.pyi_parser` and converted back to +- **Loaded**: accepted today by `x2py.parsers.pyi` and converted back to semantic IR. - **Readiness**: understood by the semantic readiness checker. - **Build input**: accepted by the `.pyi` wrapper build for the implemented subset when the required native artifacts are supplied. - **Roadmap**: design direction, not implemented wrapper behavior. +Parser-related pull requests that change `x2py/parsers/pyi/` or its focused +loading tests must update this reference. The parser-reference guard checks +that contract independently from the language parser references. + ## Contract Imports Every semantic `.pyi` control name is imported from `x2py.contracts`. This @@ -400,7 +405,6 @@ leaves: ```bash python3 -m x2py contracts/basic_subroutine/__init__.pyi \ - --wrap \ --native-objects basic_subroutine.o ``` @@ -696,14 +700,12 @@ Target CLI shapes are: ```bash python3 -m x2py contracts/library/__init__.pyi \ - --wrap \ --out library \ --native-objects native.a ``` ```bash python3 -m x2py api.pyi \ - --wrap \ --out library \ --native-library native \ --native-library-dir /path/to/libs @@ -713,7 +715,6 @@ For a single standalone fragment, no `__init__.pyi` is required: ```bash python3 -m x2py dgesv.pyi \ - --wrap \ --out lapack_dgesv \ --native-objects dgesv.o ``` diff --git a/docs/user/tutorials/basic-wrapper.md b/docs/user/tutorials/basic-wrapper.md index 6ad97cce1..45837ed8c 100644 --- a/docs/user/tutorials/basic-wrapper.md +++ b/docs/user/tutorials/basic-wrapper.md @@ -206,15 +206,14 @@ From the command line, a build looks like this: ```bash python3 -m x2py tests/data/fortran/wrapper/fruntime_abi_f90.f90 \ - --wrap \ --out-dir build/fruntime_abi \ --json ``` The command writes generated bridge, binding, runtime, object, and shared library artifacts under the output directory. The JSON output reports the -module name and generated files. The `--wrap` flag is optional when all inputs -are recognizable Fortran sources and no inspection stage is selected. +module name and generated files. Recognizable wrapper inputs select the wrapper +build stage automatically when no inspection stage is selected. ## Step 5: Import And Call The Extension diff --git a/tests/README.md b/tests/README.md index c2a517ef9..3ca545386 100644 --- a/tests/README.md +++ b/tests/README.md @@ -37,9 +37,9 @@ empty directory merely to mirror this table. | Source package or surface | Primary test owner | | --- | --- | -| `x2py.c_parser` | `tests/parsing/c/` | -| `x2py.fortran_parser` | `tests/parsing/fortran/` | -| `x2py.pyi_parser` | `tests/parsing/pyi/` | +| `x2py.parsers.c` | `tests/parsing/c/` | +| `x2py.parsers.fortran` | `tests/parsing/fortran/` | +| `x2py.parsers.pyi` | `tests/parsing/pyi/` | | `x2py.probes` | `tests/probes/` | | `x2py.pipeline` | matching subject under `tests/pipeline/` | | `x2py.semantics.c2ir`, `fortran2ir`, `pyi2ir` | matching language under `tests/semantics/conversion/` | diff --git a/tests/_shared/fixture_outputs.py b/tests/_shared/fixture_outputs.py index 4ee3ce936..90743ccf8 100644 --- a/tests/_shared/fixture_outputs.py +++ b/tests/_shared/fixture_outputs.py @@ -5,8 +5,8 @@ from pathlib import Path from tempfile import TemporaryDirectory -from x2py.c_parser import CParser -from x2py.c_parser.cli import attach_preprocessing_recipe +from x2py.parsers.c import CParser +from x2py.parsers.c.cli import attach_preprocessing_recipe from x2py import parse_fortran_file from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source from x2py.semantics.c2ir import c_project_to_semantic_module diff --git a/tests/_shared/parser_property_support.py b/tests/_shared/parser_property_support.py index 5d7dd1e28..f6129db81 100644 --- a/tests/_shared/parser_property_support.py +++ b/tests/_shared/parser_property_support.py @@ -22,9 +22,9 @@ import x2py.pipeline.preprocessing as preprocessing -from x2py.c_parser import CParseError, parse_c_file +from x2py.parsers.c import CParseError, parse_c_file -from x2py.c_parser.lexer import split_top_level_c_source, top_level_split +from x2py.parsers.c.lexer import split_top_level_c_source, top_level_split from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules diff --git a/tests/_shared/pyi_conversion_support.py b/tests/_shared/pyi_conversion_support.py index 2b4e0d06a..edb7e795b 100644 --- a/tests/_shared/pyi_conversion_support.py +++ b/tests/_shared/pyi_conversion_support.py @@ -56,7 +56,7 @@ from x2py.pipeline.pyi import pyi_file_to_semantic_module, pyi_paths_to_semantic_modules, pyi_text_to_semantic_module -from x2py.pyi_parser import parse_pyi_text as parse_pyi_ast_text +from x2py.parsers.pyi import parse_pyi_text as parse_pyi_ast_text from x2py.semantics.native_contract import native_contract_issues diff --git a/tests/architecture/test_test_suite_layout.py b/tests/architecture/test_test_suite_layout.py index 637d1b755..bdb2fbd7a 100644 --- a/tests/architecture/test_test_suite_layout.py +++ b/tests/architecture/test_test_suite_layout.py @@ -30,9 +30,9 @@ } NON_STAGE_DIRECTORIES = {"wrapper"} DOCUMENTED_SOURCE_OWNERS = { - "x2py.c_parser": "tests/parsing/c/", - "x2py.fortran_parser": "tests/parsing/fortran/", - "x2py.pyi_parser": "tests/parsing/pyi/", + "x2py.parsers.c": "tests/parsing/c/", + "x2py.parsers.fortran": "tests/parsing/fortran/", + "x2py.parsers.pyi": "tests/parsing/pyi/", "x2py.probes": "tests/probes/", "x2py.pipeline": "tests/pipeline/", "x2py.wrapper_codegen": "tests/wrapper_codegen/", diff --git a/tests/architecture/test_visitor_protocol.py b/tests/architecture/test_visitor_protocol.py index d8d1e3cdb..8a37fcee2 100644 --- a/tests/architecture/test_visitor_protocol.py +++ b/tests/architecture/test_visitor_protocol.py @@ -6,8 +6,8 @@ import inspect from tests.wrapper.fortran._support import REPO_ROOT -from x2py.c_parser.parser import CParser -from x2py.fortran_parser.parser import FortranParser, SourceUnit, _SOURCE_UNIT_TYPES +from x2py.parsers.c.parser import CParser +from x2py.parsers.fortran.parser import FortranParser, SourceUnit, _SOURCE_UNIT_TYPES from x2py.semantics.c2ir import CToIRConverter from x2py.semantics.fortran2ir import FortranToIRConverter, _FortranVariableContextVisitor from x2py.semantics.pyi2ir import _ClassBodyVisitor, _ModuleVisitor @@ -36,7 +36,7 @@ FortranBridgeGenerator, ) VISITOR_IMPLEMENTATION_PATHS = ( - REPO_ROOT / "x2py" / "fortran_parser" / "parser.py", + REPO_ROOT / "x2py" / "parsers" / "fortran" / "parser.py", REPO_ROOT / "x2py" / "semantics" / "c2ir.py", REPO_ROOT / "x2py" / "semantics" / "fortran2ir.py", REPO_ROOT / "x2py" / "semantics" / "pyi2ir.py", diff --git a/tests/benchmarks/test_parser_benchmarks.py b/tests/benchmarks/test_parser_benchmarks.py index 27f0ef5a3..986ff9b79 100644 --- a/tests/benchmarks/test_parser_benchmarks.py +++ b/tests/benchmarks/test_parser_benchmarks.py @@ -6,7 +6,7 @@ import pytest -from x2py.c_parser import parse_c_file +from x2py.parsers.c import parse_c_file from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules from x2py.wrapper_codegen.printers import emit_module_stubs from x2py import parse_fortran_file diff --git a/tests/cli/_cli_support.py b/tests/cli/_cli_support.py index b15a835ba..143cb0a11 100644 --- a/tests/cli/_cli_support.py +++ b/tests/cli/_cli_support.py @@ -18,7 +18,7 @@ import pytest -from x2py.fortran_parser import cli as fortran_parser_cli +from x2py.parsers.fortran import cli as fortran_parser_cli from x2py import FortranParseError diff --git a/tests/cli/test_argument_contract.py b/tests/cli/test_argument_contract.py index a4ab979db..2ff144332 100644 --- a/tests/cli/test_argument_contract.py +++ b/tests/cli/test_argument_contract.py @@ -132,15 +132,11 @@ class extra(Opaque): "--out for wrapper builds expects a valid Python module name", ), ( - {"wrap": True, "out": "module", "makefile": True}, + {"out": "module", "makefile": True}, "--out names a compiled wrapper extension and cannot be combined with --makefile", ), ( - {"makefile": True}, - "--makefile requires --wrap", - ), - ( - {"wrap": True, "makefile": True, "verbose": True}, + {"makefile": True, "verbose": True}, "--makefile cannot be combined with --verbose", ), ( @@ -210,11 +206,8 @@ class extra(Opaque): ), ( {"paths": ["input.pyi"]}, - "Select at least one stage flag: --parse, --semantics, --pyi, --wrap-readiness, or --wrap", - ), - ( - {"paths": [], "build_manifest": "build/x2py-build.json"}, - "--build-manifest requires --wrap", + "A .pyi wrapper build requires --native-fortran-sources, --native-objects, " + "--native-library, or --native-link-item", ), ], ) @@ -252,7 +245,6 @@ def test_x2py_main_collects_many_native_inputs_from_one_option_group( [ "x2py", str(contract), - "--wrap", "--native-fortran-sources", "source_one.f90", "source_two.f90", @@ -311,6 +303,23 @@ def test_x2py_main_collects_many_native_inputs_from_one_option_group( assert payload["module_name"] == "module" +@pytest.mark.parametrize( + "overrides", + [ + {"paths": ["input.f90"]}, + {"paths": ["contract.pyi"], "native_objects": ["native.o"]}, + {"paths": ["input.f90"], "makefile": True}, + {"paths": [], "build_manifest": "build/x2py-build.json"}, + ], +) +def test_wrapper_inputs_select_the_default_build_stage(overrides): + assert x2py_cli._stage_defaults_to_wrap(_main_args(**overrides)) + + +def test_explicit_inspection_stage_prevents_default_wrapper_selection(): + assert not x2py_cli._stage_defaults_to_wrap(_main_args(parse=True)) + + def test_cli_native_fortran_flags_split_grouped_shell_words(): assert x2py_cli._cli_native_fortran_flags(["-O2 -g0", "-DNAME='value with spaces'"]) == ( "-O2", @@ -791,19 +800,13 @@ def test_cli_fortran_rejects_embedded_c_declaration_outside_execution_body(tmp_p assert "Unknown or unsupported datatype declaration" in result.stderr -@pytest.mark.parametrize( - ("extra_args", "message"), - [ - ([], "Select at least one stage flag"), - ], -) -def test_x2py_cli_rejects_pyi_without_stage(extra_args, message, tmp_path: Path): +def test_x2py_cli_defaults_pyi_to_wrapper_and_requires_native_implementation(tmp_path: Path): pyi = tmp_path / "module.pyi" pyi.write_text("def f() -> None: ...\n", encoding="utf-8") - cmd = [sys.executable, "-m", "x2py", str(pyi), *extra_args] + cmd = [sys.executable, "-m", "x2py", str(pyi)] res = subprocess.run(cmd, capture_output=True, text=True) assert res.returncode == 2 - assert message in res.stderr + assert "A .pyi wrapper build requires --native-fortran-sources" in res.stderr @pytest.mark.parametrize("macro_flag", ["-D", "-U"]) diff --git a/tests/cli/test_output_contract.py b/tests/cli/test_output_contract.py index 92ef03d1b..119a511a0 100644 --- a/tests/cli/test_output_contract.py +++ b/tests/cli/test_output_contract.py @@ -816,13 +816,13 @@ def test_fortran_parser_cli_json_and_parse_errors(tmp_path: Path): good = tmp_path / "good.f90" good.write_text("subroutine work(n)\n integer, intent(in) :: n\nend subroutine work\n", encoding="utf-8") - json_cmd = [sys.executable, "-m", "x2py.fortran_parser", str(good), "--json"] + json_cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(good), "--json"] json_res = subprocess.run(json_cmd, capture_output=True, text=True, check=True) assert str(good) in json.loads(json_res.stdout) bad = tmp_path / "bad.f90" bad.write_text("subroutine bad(x)\n weirdtype :: x\nend subroutine bad\n", encoding="utf-8") - bad_cmd = [sys.executable, "-m", "x2py.fortran_parser", str(bad), "--no-color"] + bad_cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(bad), "--no-color"] bad_res = subprocess.run(bad_cmd, capture_output=True, text=True) assert bad_res.returncode == 1 assert bad_res.stdout == "" diff --git a/tests/cli/test_stage_dispatch.py b/tests/cli/test_stage_dispatch.py index b36f31b74..4ccdba5df 100644 --- a/tests/cli/test_stage_dispatch.py +++ b/tests/cli/test_stage_dispatch.py @@ -149,7 +149,7 @@ def test_fortran_parser_cli_reports_full_source_tree_from_inline_code(tmp_path: encoding="utf-8", ) - cmd = [sys.executable, "-m", "x2py.fortran_parser", str(f90)] + cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(f90)] res = subprocess.run(cmd, capture_output=True, text=True, check=True) assert f"File: {f90}" in res.stdout @@ -196,7 +196,7 @@ def test_fortran_parser_cli_semantics_pyi_and_empty_module_report_from_inline_co semantics_cmd = [ sys.executable, "-m", - "x2py.fortran_parser", + "x2py.parsers.fortran", str(module_source), "--semantics", "--json-out", @@ -209,13 +209,13 @@ def test_fortran_parser_cli_semantics_pyi_and_empty_module_report_from_inline_co assert str(module_source) in payload assert payload[str(module_source)]["semantic_modules"][0]["functions"][0]["name"] == "solve" - pyi_cmd = [sys.executable, "-m", "x2py.fortran_parser", str(module_source), "--pyi"] + pyi_cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(module_source), "--pyi"] pyi_res = subprocess.run(pyi_cmd, capture_output=True, text=True, check=True) assert "@native_call([Addr(Arg(0)), Return('x', 0), Addr(Arg(1))])" in pyi_res.stdout assert "x: Addr(Float64)" not in pyi_res.stdout assert "def solve(" in pyi_res.stdout - empty_pyi_cmd = [sys.executable, "-m", "x2py.fortran_parser", str(program_source), "--pyi"] + empty_pyi_cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(program_source), "--pyi"] empty_pyi_res = subprocess.run(empty_pyi_cmd, capture_output=True, text=True, check=True) assert "" in empty_pyi_res.stdout @@ -729,7 +729,7 @@ def test_x2py_and_fortran_module_entrypoints_and_debug_errors(monkeypatch, capsy monkeypatch.setattr(fortran_parser_cli, "main", lambda: 0) with pytest.raises(SystemExit) as fortran_exit: - runpy.run_module("x2py.fortran_parser.__main__", run_name="__main__") + runpy.run_module("x2py.parsers.fortran.__main__", run_name="__main__") assert fortran_exit.value.code == 0 monkeypatch.setattr(fortran_parser_cli, "main", original_fortran_main) @@ -737,7 +737,7 @@ def fail_parse(_paths): raise FortranParseError("bad", filename="bad.f90", line_number=1, source_line="bad") monkeypatch.setattr(fortran_parser_cli, "_parse_paths", fail_parse) - monkeypatch.setattr(sys, "argv", ["x2py.fortran_parser", "bad.f90", "--no-color"]) + monkeypatch.setattr(sys, "argv", ["x2py.parsers.fortran", "bad.f90", "--no-color"]) assert fortran_parser_cli.main() == 1 assert "bad.f90:1:1: error[PARSE_ERROR]: bad" in capsys.readouterr().err monkeypatch.setenv("FORTRAN_PARSER_DEBUG", "1") @@ -788,7 +788,7 @@ def test_fortran_parser_cli_debug_flag_reraises_parse_errors(tmp_path: Path): encoding="utf-8", ) - cmd = [sys.executable, "-m", "x2py.fortran_parser", str(f90), "--debug"] + cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(f90), "--debug"] res = subprocess.run(cmd, capture_output=True, text=True) assert res.returncode == 1 @@ -806,7 +806,7 @@ def test_fortran_parser_cli_debug_traceback_env_reraises_parse_errors(tmp_path: encoding="utf-8", ) - cmd = [sys.executable, "-m", "x2py.fortran_parser", str(f90)] + cmd = [sys.executable, "-m", "x2py.parsers.fortran", str(f90)] res = subprocess.run( cmd, capture_output=True, @@ -833,19 +833,19 @@ def test_fortran_parser_main_public_api_modes_from_inline_source(tmp_path: Path, ) json_out = tmp_path / "report.json" - monkeypatch.setattr(sys, "argv", ["x2py.fortran_parser", str(f90), "--json-out", str(json_out), "--json"]) + monkeypatch.setattr(sys, "argv", ["x2py.parsers.fortran", str(f90), "--json-out", str(json_out), "--json"]) assert fortran_parser_cli.main() == 0 stdout_payload = json.loads(capsys.readouterr().out) assert str(f90) in stdout_payload assert json_out.exists() - monkeypatch.setattr(sys, "argv", ["x2py.fortran_parser", str(f90), "--pyi"]) + monkeypatch.setattr(sys, "argv", ["x2py.parsers.fortran", str(f90), "--pyi"]) assert fortran_parser_cli.main() == 0 pyi_out = capsys.readouterr().out assert "File:" in pyi_out assert "def work(" in pyi_out - monkeypatch.setattr(sys, "argv", ["x2py.fortran_parser", str(f90)]) + monkeypatch.setattr(sys, "argv", ["x2py.parsers.fortran", str(f90)]) assert fortran_parser_cli.main() == 0 readable = capsys.readouterr().out assert "module m" in readable diff --git a/tests/cli/test_wrap_readiness.py b/tests/cli/test_wrap_readiness.py index 5d5022eef..a5a835946 100644 --- a/tests/cli/test_wrap_readiness.py +++ b/tests/cli/test_wrap_readiness.py @@ -171,4 +171,4 @@ def test_x2py_main_argument_validation_errors(tmp_path: Path, monkeypatch, capsy with pytest.raises(SystemExit) as stage_error: x2py_cli.main() assert stage_error.value.code == 2 - assert "Select at least one stage flag" in capsys.readouterr().err + assert "A .pyi wrapper build requires --native-fortran-sources" in capsys.readouterr().err diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index a08d3ff44..24b206f3b 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -64,7 +64,7 @@ r"|\b(?:ORDER_C|REQUIRE_C_CONTIGUOUS|NPY_C_CONTIGUOUS)\b" r"|\b(?:CToIR|CFile|CProject|CParse|CDiagnostic)[A-Za-z0-9_]*\b" r"|\b(?:parse_c|c_file|c_project|c_function|c_parameter|c_struct|c_type)_[A-Za-z0-9_]+\b" - r"|(?:tests/data/c|tests/parser/c|x2py/c_parser|/c/general/)" + r"|(?:tests/data/c|tests/parser/c|x2py/parsers/c|/c/general/)" r"|(?:c-parser|inspect-c-api|c-api)" r"|\b(?:structs?|unions?|typedefs?|declarators?|bitfields?|K&R)\b" r"|--language\s+c\b" @@ -212,9 +212,10 @@ "docs/developer/repository-structure.md", "docs/maintainer/internal-architecture/pipeline-map.md", "x2py/README.md", - "x2py/c_parser/README.md", - "x2py/fortran_parser/README.md", - "x2py/pyi_parser/README.md", + "x2py/parsers/README.md", + "x2py/parsers/c/README.md", + "x2py/parsers/fortran/README.md", + "x2py/parsers/pyi/README.md", "x2py/semantics/README.md", "x2py/compiling/README.md", ] @@ -226,11 +227,11 @@ "x2py/probes/c_types.py", "x2py/probes/fortran_types.py", "x2py/semantics/ownership.py", - "x2py/c_parser/parser.py", - "x2py/c_parser/cli.py", - "x2py/fortran_parser/parser.py", - "x2py/fortran_parser/cli.py", - "x2py/pyi_parser/parser.py", + "x2py/parsers/c/parser.py", + "x2py/parsers/c/cli.py", + "x2py/parsers/fortran/parser.py", + "x2py/parsers/fortran/cli.py", + "x2py/parsers/pyi/parser.py", "x2py/semantics/models.py", "x2py/semantics/fortran2ir.py", "x2py/semantics/c2ir.py", @@ -373,16 +374,14 @@ "user/examples/recipes/compiler-preprocessing.md", ] MAJOR_SOURCE_PACKAGES = [ - "x2py/c_parser/", - "x2py/fortran_parser/", + "x2py/parsers/", "x2py/semantics/", "x2py/wrapper_codegen/", "x2py/compiling/", ] PACKAGE_READMES = [ "x2py/README.md", - "x2py/c_parser/README.md", - "x2py/fortran_parser/README.md", + "x2py/parsers/README.md", "x2py/semantics/README.md", "x2py/compiling/README.md", ] @@ -874,19 +873,14 @@ def test_getting_started_pages_keep_advanced_stage_flags_out_of_beginner_path() assert "--json" not in content -def test_user_guide_keeps_default_source_builds_free_of_redundant_stage_flags() -> None: +def test_user_guide_uses_automatic_wrapper_stage_selection() -> None: content = "\n".join( _visible_documentation_source(DOCS_ROOT / relative_path) for relative_path in REQUIRED_USER_GUIDE_PAGES ) - assert "--json" not in content - assert "--wrap-readiness" not in content - assert "points.f90 --wrap" not in content - assert "src/scale.f90 --wrap --out-dir" not in content - assert "fruntime_abi_f90.f90 \\\n --wrap" not in content - assert "solver.f90 \\\n diagnostics.f90 \\\n --wrap" not in content - assert "python3 -m x2py contracts/solver/__init__.pyi \\\n --wrap" in content - assert "Makefile mode is an explicit wrapper submode" in content + assert "python3 -m x2py src/scale.f90 \\\n --makefile" in content + assert "python3 -m x2py contracts/solver/__init__.pyi \\\n --native-fortran-sources solver.f90" in content + assert "python3 -m x2py mesh.f90 solver.f90 --makefile --out-dir build" in content def test_array_handle_docs_keep_views_copies_and_handles_distinct() -> None: diff --git a/tests/parser/c/README.md b/tests/parser/c/README.md index 27ba1717f..416bf4b21 100644 --- a/tests/parser/c/README.md +++ b/tests/parser/c/README.md @@ -31,7 +31,7 @@ curated fixture workflow. ## Developer Walkthrough `test_c_parser_developer_tutorial.py` is an executable reading guide for -`x2py/c_parser/parser.py`. It shows the shared declaration/declarator gateway, the +`x2py/parsers/c/parser.py`. It shows the shared declaration/declarator gateway, the `parse_file` routing of declaration roles, and the preprocessed linemarker path without replacing the feature-focused test modules. diff --git a/tests/parser/c/errors/generate_c_parser_error_goldens.py b/tests/parser/c/errors/generate_c_parser_error_goldens.py index 349ba2d51..4bd760a60 100644 --- a/tests/parser/c/errors/generate_c_parser_error_goldens.py +++ b/tests/parser/c/errors/generate_c_parser_error_goldens.py @@ -6,7 +6,7 @@ import sys from pathlib import Path -from x2py.c_parser import CParseError, parse_c_file +from x2py.parsers.c import CParseError, parse_c_file _TESTS_DIR = Path(__file__).resolve().parents[3] diff --git a/tests/parser/c/generate_c_parser_goldens.py b/tests/parser/c/generate_c_parser_goldens.py index be4e59fb7..73a4aa105 100644 --- a/tests/parser/c/generate_c_parser_goldens.py +++ b/tests/parser/c/generate_c_parser_goldens.py @@ -266,7 +266,7 @@ def _stable_project_payload(payload: dict) -> dict: def _serialize_project(fixtures: list[Path]) -> dict: - from x2py.c_parser import CParser + from x2py.parsers.c import CParser parser = CParser() include_dirs = sorted({fixture.parent for fixture in fixtures}) diff --git a/tests/parsing/c/test_c_cli_skeleton.py b/tests/parsing/c/test_c_cli_skeleton.py index 2c7a47922..2dcc05621 100644 --- a/tests/parsing/c/test_c_cli_skeleton.py +++ b/tests/parsing/c/test_c_cli_skeleton.py @@ -10,8 +10,8 @@ import pytest -from x2py.c_parser import CParseError -from x2py.c_parser import cli as c_parser_cli +from x2py.parsers.c import CParseError +from x2py.parsers.c import cli as c_parser_cli from x2py import cli as x2py_cli from x2py.pipeline.preprocessing import PreprocessingConfig @@ -180,7 +180,7 @@ def fail_parse(_paths): monkeypatch.setattr(c_parser_cli, "main", lambda _argv=None: 0) with pytest.raises(SystemExit) as exc_info: - runpy.run_module("x2py.c_parser.__main__", run_name="__main__") + runpy.run_module("x2py.parsers.c.__main__", run_name="__main__") assert exc_info.value.code == 0 @@ -559,13 +559,13 @@ def test_c_parser_cli_module_handles_directory_loader_and_output_modes(tmp_path: def test_c_parser_module_entrypoint_and_exports(tmp_path: Path): - import x2py.c_parser.__main__ as c_module_entrypoint - from x2py.c_parser.parser import parse_c_project + import x2py.parsers.c.__main__ as c_module_entrypoint + from x2py.parsers.c.parser import parse_c_project header = tmp_path / "api.h" header.write_text("int run(void);\n", encoding="utf-8") result = subprocess.run( - [sys.executable, "-m", "x2py.c_parser", str(header), "--json"], + [sys.executable, "-m", "x2py.parsers.c", str(header), "--json"], capture_output=True, text=True, check=True, @@ -581,7 +581,7 @@ def test_c_parser_module_formats_parse_errors_without_traceback(tmp_path: Path): header.write_text("@@@;\n", encoding="utf-8") result = subprocess.run( - [sys.executable, "-m", "x2py.c_parser", str(header), "--no-color"], + [sys.executable, "-m", "x2py.parsers.c", str(header), "--no-color"], capture_output=True, text=True, ) @@ -596,7 +596,7 @@ def test_c_parser_module_debug_reraises_parse_errors(tmp_path: Path): header.write_text("@@@;\n", encoding="utf-8") result = subprocess.run( - [sys.executable, "-m", "x2py.c_parser", str(header), "--debug"], + [sys.executable, "-m", "x2py.parsers.c", str(header), "--debug"], capture_output=True, text=True, ) diff --git a/tests/parsing/c/test_c_compiler_extensions.py b/tests/parsing/c/test_c_compiler_extensions.py index 7c469e73a..8fc8e9c8e 100644 --- a/tests/parsing/c/test_c_compiler_extensions.py +++ b/tests/parsing/c/test_c_compiler_extensions.py @@ -7,7 +7,7 @@ def test_raw_mode_keeps_compiler_extension_declarations_conservative(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( 'int exported(void) __attribute__((visibility("default")));\n', @@ -22,7 +22,7 @@ def test_raw_mode_keeps_compiler_extension_declarations_conservative(): def test_gnu_header_spelling_aliases_and_harmless_attributes_are_tolerated(): - from x2py.c_parser import CComposedType, CConst, CRestrict, parse_c_file + from x2py.parsers.c import CComposedType, CConst, CRestrict, parse_c_file parsed = parse_c_file( """ @@ -52,7 +52,7 @@ def test_gnu_header_spelling_aliases_and_harmless_attributes_are_tolerated(): def test_layout_and_abi_attributes_are_parsed_with_explicit_warnings(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -79,7 +79,7 @@ def test_layout_and_abi_attributes_are_parsed_with_explicit_warnings(): def test_declspec_calling_conventions_asm_labels_and_top_level_asm_are_tolerated(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -106,7 +106,7 @@ def test_declspec_calling_conventions_asm_labels_and_top_level_asm_are_tolerated def test_bare_compiler_extensions_and_comments_are_tolerated(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -129,7 +129,7 @@ def test_bare_compiler_extensions_and_comments_are_tolerated(): def test_compiler_extension_normalization_preserves_coordinates_and_source_states(): - from x2py.c_parser import CParser + from x2py.parsers.c import CParser source = ( 'const char *s = "__attribute__((packed))";\n' @@ -174,7 +174,7 @@ def test_compiler_extension_normalization_preserves_coordinates_and_source_state def test_compiler_extension_only_segment_does_not_stop_later_declarations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "__attribute__((deprecated));\nint kept;\n", @@ -187,7 +187,7 @@ def test_compiler_extension_only_segment_does_not_stop_later_declarations(): def test_abi_pointer_qualifiers_are_accepted_with_explicit_warning(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "int *__ptr64 global_ptr;\n", @@ -202,7 +202,7 @@ def test_abi_pointer_qualifiers_are_accepted_with_explicit_warning(): def test_double_bracket_attribute_scanner_ignores_quoted_closers(): - from x2py.c_parser import CParser + from x2py.parsers.c import CParser text = '[[vendor::attr("escaped \\" quote and ]] text")]] int value;' @@ -213,7 +213,7 @@ def test_double_bracket_attribute_scanner_ignores_quoted_closers(): def test_typeof_bitint_and_extended_scalars_remain_parseable_as_opaque_types(): - from x2py.c_parser import CTypedef, CUnknownType, parse_c_file + from x2py.parsers.c import CTypedef, CUnknownType, parse_c_file parsed = parse_c_file( """ @@ -242,7 +242,7 @@ def test_typeof_bitint_and_extended_scalars_remain_parseable_as_opaque_types(): def test_preprocessed_extension_diagnostics_and_declarations_use_linemarkers(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -267,7 +267,7 @@ def test_preprocessed_extension_diagnostics_and_declarations_use_linemarkers(): def test_gcc_preprocessed_standard_headers_remain_parseable(tmp_path: Path): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source compiler = shutil.which("cc") diff --git a/tests/parsing/c/test_c_corpus.py b/tests/parsing/c/test_c_corpus.py index 509ca4cc6..275f95c31 100644 --- a/tests/parsing/c/test_c_corpus.py +++ b/tests/parsing/c/test_c_corpus.py @@ -35,7 +35,7 @@ def test_cjson_regression_source_and_header_are_available(): def test_cjson_header_raw_parse_requires_preprocessing(): import pytest - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="require compiler preprocessing") as exc_info: parse_c_file(_CJSON_DIR / "cJSON.h") @@ -43,7 +43,7 @@ def test_cjson_header_raw_parse_requires_preprocessing(): def test_cjson_header_preprocessed_mode_has_no_error_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( _preprocessed_cjson_source("cJSON.h"), @@ -55,7 +55,7 @@ def test_cjson_header_preprocessed_mode_has_no_error_diagnostics(): def test_cjson_callback_hook_declarations_are_preprocessed_without_error_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( _preprocessed_cjson_source("cJSON.h"), @@ -68,7 +68,7 @@ def test_cjson_callback_hook_declarations_are_preprocessed_without_error_diagnos def test_cjson_source_file_parse_skips_function_bodies_safely(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( _preprocessed_cjson_source("cJSON.c"), @@ -81,7 +81,7 @@ def test_cjson_source_file_parse_skips_function_bodies_safely(): def test_cjson_project_parse_links_header_and_source(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project sources = {filename: _preprocessed_cjson_source(filename) for filename in ("cJSON.h", "cJSON.c")} project = parse_c_project(sources, preprocessing="compiler") diff --git a/tests/parsing/c/test_c_declarations_and_declarators.py b/tests/parsing/c/test_c_declarations_and_declarators.py index dfe90d34f..a1cb10448 100644 --- a/tests/parsing/c/test_c_declarations_and_declarators.py +++ b/tests/parsing/c/test_c_declarations_and_declarators.py @@ -4,7 +4,7 @@ def test_primitive_specifiers_create_concrete_primitive_types(): - from x2py.c_parser import CBool, CShort, CUnsignedLongLong, parse_c_file + from x2py.parsers.c import CBool, CShort, CUnsignedLongLong, parse_c_file parsed = parse_c_file( """ @@ -62,8 +62,8 @@ def test_primitive_specifiers_create_concrete_primitive_types(): ], ) def test_every_supported_primitive_spelling_creates_a_concrete_ctype(spelling, expected_name): - import x2py.c_parser as c_parser - from x2py.c_parser import CType, parse_c_file + import x2py.parsers.c as c_parser + from x2py.parsers.c import CType, parse_c_file function = parse_c_file(f"{spelling} primitive(void);\n", filename="primitive_table.h").functions[0] expected = getattr(c_parser, expected_name) @@ -82,8 +82,8 @@ def test_every_supported_primitive_spelling_creates_a_concrete_ctype(spelling, e ], ) def test_valid_reordered_primitive_specifiers_are_normalized(spelling, expected_name): - import x2py.c_parser as c_parser - from x2py.c_parser import parse_c_file + import x2py.parsers.c as c_parser + from x2py.parsers.c import parse_c_file function = parse_c_file(f"{spelling} primitive(void);\n", filename="reordered_primitives.h").functions[0] @@ -101,7 +101,7 @@ def test_valid_reordered_primitive_specifiers_are_normalized(spelling, expected_ ], ) def test_invalid_primitive_specifier_sequences_raise_parse_errors(source, expected_column): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="Invalid type specifier sequence") as error: parse_c_file(source, filename="invalid_specifiers.h") @@ -114,7 +114,7 @@ def test_invalid_primitive_specifier_sequences_raise_parse_errors(source, expect def test_unresolved_single_typedef_name_is_preserved_until_resolution(): - from x2py.c_parser import CTypedef, parse_c_file + from x2py.parsers.c import CTypedef, parse_c_file parsed = parse_c_file("external_type value;\n", filename="deferred_typedef.h") @@ -124,7 +124,7 @@ def test_unresolved_single_typedef_name_is_preserved_until_resolution(): def test_pointer_qualifiers_belong_to_the_component_they_qualify(): - from x2py.c_parser import CComposedType, CConst, CDouble, CPointer, CRestrict, parse_c_file + from x2py.parsers.c import CComposedType, CConst, CDouble, CPointer, CRestrict, parse_c_file parsed = parse_c_file( "void copy(const double * restrict src, double * restrict dst);\n", @@ -143,7 +143,7 @@ def test_pointer_qualifiers_belong_to_the_component_they_qualify(): def test_multi_level_qualifiers_stay_on_their_exact_type_components(): - from x2py.c_parser import CComposedType, CConst, CInt, CPointer, CVolatile, parse_c_file + from x2py.parsers.c import CComposedType, CConst, CInt, CPointer, CVolatile, parse_c_file parsed = parse_c_file( "const int * const * volatile chain;\n", @@ -159,7 +159,7 @@ def test_multi_level_qualifiers_stay_on_their_exact_type_components(): def test_array_parameters_preserve_declarations_and_expose_adjusted_pointer_types(): - from x2py.c_parser import CArray, CComposedType, CConst, CDouble, CInt, CPointer, parse_c_file + from x2py.parsers.c import CArray, CComposedType, CConst, CDouble, CInt, CPointer, parse_c_file parsed = parse_c_file( "void solve(size_t n, double a[static 4], const int shape[2], int work[const *], int matrix[3][4]);\n", @@ -193,7 +193,7 @@ def test_array_parameters_preserve_declarations_and_expose_adjusted_pointer_type def test_multiple_declarators_share_specifiers_but_have_distinct_compositions(): - from x2py.c_parser import CArray, CComposedType, CConst, CInt, CPointer, parse_c_file + from x2py.parsers.c import CArray, CComposedType, CConst, CInt, CPointer, parse_c_file parsed = parse_c_file("extern const int *left, right[4];\n", filename="variables.h") @@ -207,7 +207,7 @@ def test_multiple_declarators_share_specifiers_but_have_distinct_compositions(): def test_typedefs_and_typedef_references_are_concrete_types(): - from x2py.c_parser import CArray, CComposedType, CDouble, CPointer, CStruct, CTypedef, CUnsignedLong, parse_c_file + from x2py.parsers.c import CArray, CComposedType, CDouble, CPointer, CStruct, CTypedef, CUnsignedLong, parse_c_file parsed = parse_c_file( """ @@ -236,7 +236,7 @@ def test_typedefs_and_typedef_references_are_concrete_types(): def test_repeated_file_scope_tentative_variable_declarations_merge(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int i;\nint i;\n", filename="tentative.c") @@ -247,7 +247,7 @@ def test_repeated_file_scope_tentative_variable_declarations_merge(): def test_tentative_variable_declaration_followed_by_definition_prefers_definition(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int i;\nint i = 1;\n", filename="definition.c") @@ -259,7 +259,7 @@ def test_tentative_variable_declaration_followed_by_definition_prefers_definitio def test_duplicate_initialized_file_scope_variables_report_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int i = 1;\nint i = 2;\n", filename="duplicate_variables.c") @@ -269,7 +269,7 @@ def test_duplicate_initialized_file_scope_variables_report_diagnostic(): def test_conflicting_file_scope_variable_declarations_report_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int i;\ndouble i;\n", filename="conflicting_variables.c") @@ -278,7 +278,7 @@ def test_conflicting_file_scope_variable_declarations_report_diagnostic(): def test_type_key_preserves_seen_state_for_recursive_composed_types(): - from x2py.c_parser import CComposedType, CParser, CPointer, CTypedef + from x2py.parsers.c import CComposedType, CParser, CPointer, CTypedef typedef = CTypedef(name="node") recursive = CComposedType(components=[CPointer(), typedef]) @@ -295,7 +295,7 @@ def test_type_key_preserves_seen_state_for_recursive_composed_types(): def test_compatible_repeated_typedefs_merge_but_conflicting_typedefs_diagnose(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file compatible = parse_c_file("typedef int count_t;\ntypedef int count_t;\n", filename="typedefs.h") conflicting = parse_c_file("typedef int count_t;\ntypedef double count_t;\n", filename="bad_typedefs.h") @@ -307,7 +307,7 @@ def test_compatible_repeated_typedefs_merge_but_conflicting_typedefs_diagnose(): def test_variables_preserve_initializer_text_arrays_and_concrete_tag_types(): - from x2py.c_parser import CArray, CEnum, CInt, CStruct, CUnion, parse_c_file + from x2py.parsers.c import CArray, CEnum, CInt, CStruct, CUnion, parse_c_file parsed = parse_c_file( """ @@ -334,7 +334,7 @@ def test_variables_preserve_initializer_text_arrays_and_concrete_tag_types(): def test_parameters_preserve_concrete_struct_union_and_enum_uses(): - from x2py.c_parser import CEnum, CStruct, CUnion, parse_c_file + from x2py.parsers.c import CEnum, CStruct, CUnion, parse_c_file parsed = parse_c_file( "void consume(const struct state *s, union scalar *u, enum status status);\n", @@ -349,7 +349,7 @@ def test_parameters_preserve_concrete_struct_union_and_enum_uses(): def test_incomplete_structs_and_pointer_uses_are_concrete_objects(): - from x2py.c_parser import CComposedType, CPointer, CStruct, parse_c_file + from x2py.parsers.c import CComposedType, CPointer, CStruct, parse_c_file parsed = parse_c_file( """ @@ -375,7 +375,7 @@ def test_incomplete_structs_and_pointer_uses_are_concrete_objects(): def test_storage_is_declaration_metadata_and_qualifiers_are_type_metadata(): - from x2py.c_parser import CAtomic, CConst, CUnsignedLong, CVolatile, parse_c_file + from x2py.parsers.c import CAtomic, CConst, CUnsignedLong, CVolatile, parse_c_file parsed = parse_c_file( """ @@ -400,7 +400,7 @@ def test_storage_is_declaration_metadata_and_qualifiers_are_type_metadata(): def test_atomic_type_specifier_qualifies_the_declared_outermost_type(): - from x2py.c_parser import CAtomic, CComposedType, CInt, CPointer, parse_c_file + from x2py.parsers.c import CAtomic, CComposedType, CInt, CPointer, parse_c_file parsed = parse_c_file( """ @@ -437,7 +437,7 @@ def test_atomic_type_specifier_qualifies_the_declared_outermost_type(): ], ) def test_invalid_atomic_type_specifiers_raise_focused_errors(source, message): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match=message) as exc_info: parse_c_file(source, filename="invalid_atomic.h") @@ -446,7 +446,7 @@ def test_invalid_atomic_type_specifiers_raise_focused_errors(source, message): def test_function_bodies_do_not_contribute_local_variables(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -460,7 +460,7 @@ def test_function_bodies_do_not_contribute_local_variables(): def test_declarations_return_concrete_objects_instead_of_kind_fields(): - from x2py.c_parser import ( + from x2py.parsers.c import ( CArray, CFunction, CFunctionType, @@ -498,7 +498,7 @@ def test_declarations_return_concrete_objects_instead_of_kind_fields(): def test_composite_definitions_are_concrete_objects_and_static_assert_is_diagnostic(): - from x2py.c_parser import CEnum, CStruct, CUnion, CVariable, parse_c_file + from x2py.parsers.c import CEnum, CStruct, CUnion, CVariable, parse_c_file parsed = parse_c_file( """ @@ -519,7 +519,7 @@ def test_composite_definitions_are_concrete_objects_and_static_assert_is_diagnos def test_parenthesized_declarators_preserve_pointer_array_order(): - from x2py.c_parser import CArray, CInt, CPointer, parse_c_file + from x2py.parsers.c import CArray, CInt, CPointer, parse_c_file parsed = parse_c_file("extern int *values[4];\nextern int (*matrix)[4];\n", filename="paren_decl.h") variables = {variable.name: variable for variable in parsed.variables} @@ -529,7 +529,7 @@ def test_parenthesized_declarators_preserve_pointer_array_order(): def test_function_type_discards_placeholder_parameter_names(): - from x2py.c_parser import CFunctionType, CPointer, parse_c_file + from x2py.parsers.c import CFunctionType, CPointer, parse_c_file parsed = parse_c_file( "typedef int (*compare_fn)(const void *left, const void *right);\n", @@ -544,7 +544,7 @@ def test_function_type_discards_placeholder_parameter_names(): def test_conflicting_function_pointer_typedefs_report_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "typedef int (*callback_fn)(int);\ntypedef double (*callback_fn)(double);\n", @@ -558,7 +558,7 @@ def test_conflicting_function_pointer_typedefs_report_diagnostic(): def test_recursive_compositions_cover_tables_callback_arrays_and_function_results(): - from x2py.c_parser import CArray, CFunctionType, CInt, CPointer, parse_c_file + from x2py.parsers.c import CArray, CFunctionType, CInt, CPointer, parse_c_file parsed = parse_c_file( """ @@ -589,7 +589,7 @@ def test_recursive_compositions_cover_tables_callback_arrays_and_function_result def test_declaration_attributes_are_tolerated_and_layout_omissions_are_diagnosed(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -608,7 +608,7 @@ def test_declaration_attributes_are_tolerated_and_layout_omissions_are_diagnosed def test_unsupported_top_level_declarator_is_reported_with_source_location(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int value @@;\nint kept;\n", filename="bad_declarator.h") @@ -643,8 +643,8 @@ def test_unsupported_top_level_declarator_is_reported_with_source_location(): ], ) def test_unsupported_declaration_diagnostic_classifies_known_shapes(text, unit_kind, message): - from x2py.c_parser import CParser - from x2py.c_parser.lexer import CTopLevelSegment + from x2py.parsers.c import CParser + from x2py.parsers.c.lexer import CTopLevelSegment segment = CTopLevelSegment( text=text, @@ -671,8 +671,8 @@ def test_unsupported_declaration_diagnostic_classifies_known_shapes(text, unit_k def test_unsupported_declaration_diagnostic_ignores_empty_and_plain_declarations(): - from x2py.c_parser import CParser - from x2py.c_parser.lexer import CTopLevelSegment + from x2py.parsers.c import CParser + from x2py.parsers.c.lexer import CTopLevelSegment parser = CParser() @@ -688,7 +688,7 @@ def test_unsupported_declaration_diagnostic_ignores_empty_and_plain_declarations ], ) def test_non_c_top_level_grammar_is_rejected_without_language_guessing(source): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="Invalid C syntax") as exc_info: parse_c_file(source, filename="invalid_top_level.h") @@ -705,7 +705,7 @@ def test_non_c_top_level_grammar_is_rejected_without_language_guessing(source): ], ) def test_identifier_spelling_does_not_trigger_foreign_language_detection(source, name, type_name): - from x2py.c_parser import CTypedef, parse_c_file + from x2py.parsers.c import CTypedef, parse_c_file parsed = parse_c_file(source, filename="identifier_spelling.h") @@ -715,7 +715,7 @@ def test_identifier_spelling_does_not_trigger_foreign_language_detection(source, def test_braced_and_designated_initializer_declarations_preserve_source_text(): - from x2py.c_parser import CArray, CComposedType, parse_c_file + from x2py.parsers.c import CArray, CComposedType, parse_c_file parsed = parse_c_file( "struct config;\nint values[3] = {1, 2, 3};\nstruct config cfg = {.enabled = 1};\nint scalar = 1;\n", @@ -733,7 +733,7 @@ def test_braced_and_designated_initializer_declarations_preserve_source_text(): def test_asm_declarator_suffixes_are_tolerated_with_symbol_identity_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( 'extern int retained, pinned asm("r0");\nint run(int value asm("r0"));\n', @@ -750,7 +750,7 @@ def test_asm_declarator_suffixes_are_tolerated_with_symbol_identity_diagnostics( def test_storage_class_and_inline_specifiers_are_recorded_on_functions(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "static inline int local_add(int a, int b) { return a + b; }\nextern int exported_add(int a, int b);\n", diff --git a/tests/parsing/c/test_c_error_fixture_suite.py b/tests/parsing/c/test_c_error_fixture_suite.py index ced58cd5e..8f95d23b6 100644 --- a/tests/parsing/c/test_c_error_fixture_suite.py +++ b/tests/parsing/c/test_c_error_fixture_suite.py @@ -54,7 +54,7 @@ def test_c_error_fixtures_have_matching_expected_json(): def test_c_error_fixture_suite_reports_expected_diagnostics(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file for fixture in sorted(_ERRORS_DIR.glob("*")): if fixture.suffix.lower() not in _SOURCE_SUFFIXES: diff --git a/tests/parsing/c/test_c_fixture_suite.py b/tests/parsing/c/test_c_fixture_suite.py index 56dc0d844..da78b8fd0 100644 --- a/tests/parsing/c/test_c_fixture_suite.py +++ b/tests/parsing/c/test_c_fixture_suite.py @@ -75,7 +75,7 @@ def test_c_fixture_suite_has_inputs(data_subdir): ], ) def test_c_fixture_headers_with_macros_require_preprocessing(fixture): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="require compiler preprocessing") as exc_info: parse_c_file(fixture) @@ -98,7 +98,7 @@ def test_c_fixture_headers_with_macros_require_preprocessing(fixture): ], ) def test_c_fixture_headers_parse_after_compiler_preprocessing(fixture, defines): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source compiler = shutil.which("cc") @@ -126,7 +126,7 @@ def test_c_fixture_headers_parse_after_compiler_preprocessing(fixture, defines): def test_c_fixture_suite_keeps_source_locations_stable_for_plain_source(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( _DATA_DIR / "general" / "basic_array_update.c", diff --git a/tests/parsing/c/test_c_functions.py b/tests/parsing/c/test_c_functions.py index 8d3a28f3f..928aba5d4 100644 --- a/tests/parsing/c/test_c_functions.py +++ b/tests/parsing/c/test_c_functions.py @@ -4,7 +4,7 @@ def test_named_function_exposes_result_type_named_parameters_and_derived_type(): - from x2py.c_parser import CDouble, CFunctionType, CTypedef, parse_c_file + from x2py.parsers.c import CDouble, CFunctionType, CTypedef, parse_c_file parsed = parse_c_file( "double dot(size_t n, const double *x, const double *y);\n", @@ -21,7 +21,7 @@ def test_named_function_exposes_result_type_named_parameters_and_derived_type(): def test_function_definitions_skip_bodies_but_preserve_start_and_end_locations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -42,7 +42,7 @@ def test_function_definitions_skip_bodies_but_preserve_start_and_end_locations() def test_void_parameter_list_and_empty_parameter_list_are_distinguished(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int explicit_void(void);\nint unspecified();\n", filename="void_params.h") @@ -53,7 +53,7 @@ def test_void_parameter_list_and_empty_parameter_list_are_distinguished(): def test_variadic_functions_are_parsed_as_source_facts(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("int log_msg(const char *fmt, ...);\n", filename="variadic.h") @@ -62,7 +62,7 @@ def test_variadic_functions_are_parsed_as_source_facts(): def test_old_style_knr_function_definition_raises_unsupported_diagnostic(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file source = """ int add(a, b) @@ -85,7 +85,7 @@ def test_old_style_knr_function_definition_raises_unsupported_diagnostic(): def test_old_style_knr_detection_uses_linemarkers_and_normalized_headers(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file source = """# 40 "generated_api.c" __extension__ int exported(a) @@ -107,7 +107,7 @@ def test_old_style_knr_detection_uses_linemarkers_and_normalized_headers(): def test_modern_prototype_before_old_style_definition_does_not_stop_knr_detection(): - from x2py.c_parser import CParseError, CParser, parse_c_file + from x2py.parsers.c import CParseError, CParser, parse_c_file source = """ int modern(int value) @@ -141,7 +141,7 @@ def test_modern_prototype_before_old_style_definition_does_not_stop_knr_detectio def test_old_style_knr_scan_skips_directives_and_keeps_scanning(): - from x2py.c_parser import CParseError, CParser + from x2py.parsers.c import CParseError, CParser parser = CParser() parser._raise_for_unsupported_old_style_definitions( @@ -161,7 +161,7 @@ def test_old_style_knr_scan_skips_directives_and_keeps_scanning(): def test_find_parameter_list_returns_outer_function_signature_bounds(): - from x2py.c_parser import CParser + from x2py.parsers.c import CParser parser = CParser() text = "int run(int (*callback)(char ch), const char *label) " @@ -171,7 +171,7 @@ def test_find_parameter_list_returns_outer_function_signature_bounds(): def test_control_statement_parameter_lists_inside_function_bodies_are_not_knr_definitions(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -213,7 +213,7 @@ def test_control_statement_parameter_lists_inside_function_bodies_are_not_knr_de ], ) def test_c_parser_rejects_non_c_top_level_syntax(source): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="Invalid C syntax") as exc_info: parse_c_file(source, filename="mixed.h") @@ -222,7 +222,7 @@ def test_c_parser_rejects_non_c_top_level_syntax(source): def test_c_parser_invalid_syntax_error_maps_preprocessed_source_location(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError) as exc_info: parse_c_file( @@ -237,7 +237,7 @@ def test_c_parser_invalid_syntax_error_maps_preprocessed_source_location(): def test_c_parser_skips_non_c_tokens_inside_function_body(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -254,7 +254,7 @@ def test_c_parser_skips_non_c_tokens_inside_function_body(): def test_c_parser_does_not_classify_valid_c_from_typedef_identifier_spelling(): - from x2py.c_parser import CTypedef, parse_c_file + from x2py.parsers.c import CTypedef, parse_c_file parsed = parse_c_file("subroutine solve(void);\n", filename="identifier_spelling.h") @@ -265,7 +265,7 @@ def test_c_parser_does_not_classify_valid_c_from_typedef_identifier_spelling(): @pytest.mark.parametrize("source", ["@@@\n", "int run(void);\n@@@;\n"]) def test_c_parser_rejects_invalid_top_level_syntax(source): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="Invalid C syntax") as exc_info: parse_c_file(source, filename="invalid.c") @@ -274,7 +274,7 @@ def test_c_parser_rejects_invalid_top_level_syntax(source): def test_c_parser_ignores_invalid_syntax_inside_function_body(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -300,7 +300,7 @@ def test_c_parser_ignores_invalid_syntax_inside_function_body(): ], ) def test_c_parser_rejects_invalid_nested_grammar_units(source): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="Invalid C syntax") as exc_info: parse_c_file(source, filename="invalid_nested.h") @@ -309,7 +309,7 @@ def test_c_parser_rejects_invalid_nested_grammar_units(source): def test_control_flow_conditions_inside_function_body_do_not_look_like_knr_definitions(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -329,7 +329,7 @@ def test_control_flow_conditions_inside_function_body_do_not_look_like_knr_defin def test_function_pointer_parameter_is_a_callback_candidate_with_nameless_signature(): - from x2py.c_parser import CFunctionType, CInt, CPointer, parse_c_file + from x2py.parsers.c import CFunctionType, CInt, CPointer, parse_c_file parsed = parse_c_file( "void sort_items(void *items, int (*compare)(const void *, const void *));\n", @@ -346,7 +346,7 @@ def test_function_pointer_parameter_is_a_callback_candidate_with_nameless_signat def test_function_parameter_preserves_declaration_and_adjusts_to_callback_pointer(): - from x2py.c_parser import CComposedType, CFunctionType, CPointer, parse_c_file + from x2py.parsers.c import CComposedType, CFunctionType, CPointer, parse_c_file parsed = parse_c_file("void apply(int callback(int));\n", filename="adjusted_callback.h") @@ -360,7 +360,7 @@ def test_function_parameter_preserves_declaration_and_adjusts_to_callback_pointe def test_project_resolves_callback_typedef_parameter_to_typedef_signature(): - from x2py.c_parser import CFunctionType, CTypedef, parse_c_project + from x2py.parsers.c import CFunctionType, CTypedef, parse_c_project project = parse_c_project( { @@ -377,7 +377,7 @@ def test_project_resolves_callback_typedef_parameter_to_typedef_signature(): def test_function_returning_pointer_to_const_struct_is_preserved(): - from x2py.c_parser import CComposedType, CConst, CPointer, CStruct, parse_c_file + from x2py.parsers.c import CComposedType, CConst, CPointer, CStruct, parse_c_file parsed = parse_c_file( "struct state;\nconst struct state *current_state(void);\n", @@ -392,7 +392,7 @@ def test_function_returning_pointer_to_const_struct_is_preserved(): def test_matching_prototype_and_definition_merge_and_prefer_definition(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -414,7 +414,7 @@ def test_matching_prototype_and_definition_merge_and_prefer_definition(): def test_inline_function_body_in_header_is_recorded_as_definition(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "static inline int add_one(int value) { return value + 1; }\n", @@ -431,7 +431,7 @@ def test_inline_function_body_in_header_is_recorded_as_definition(): def test_function_declaration_attributes_are_tolerated_when_type_shape_is_unchanged(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( 'int exported(void) __attribute__((visibility("default")));\nint deprecated(void) [[deprecated]];\n', @@ -444,7 +444,7 @@ def test_function_declaration_attributes_are_tolerated_when_type_shape_is_unchan def test_unsupported_function_declarator_is_reported_and_later_declarations_continue(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "int broken @@ { return 0; }\nint kept;\n", @@ -464,7 +464,7 @@ def test_unsupported_function_declarator_is_reported_and_later_declarations_cont def test_conflicting_function_prototypes_report_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "int work(int value);\ndouble work(double value);\n", @@ -476,7 +476,7 @@ def test_conflicting_function_prototypes_report_diagnostic(): def test_function_conflicts_consider_parameters_and_variadic_marker(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -497,7 +497,7 @@ def test_function_conflicts_consider_parameters_and_variadic_marker(): def test_duplicate_function_definitions_report_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ diff --git a/tests/parsing/c/test_c_lexer_preprocessor.py b/tests/parsing/c/test_c_lexer_preprocessor.py index e1671d07f..bdfa02ee8 100644 --- a/tests/parsing/c/test_c_lexer_preprocessor.py +++ b/tests/parsing/c/test_c_lexer_preprocessor.py @@ -4,7 +4,7 @@ def test_lexer_removes_comments_without_changing_string_or_char_literals(): - from x2py.c_parser.lexer import lex_c_source + from x2py.parsers.c.lexer import lex_c_source tokens = lex_c_source( r""" @@ -23,7 +23,7 @@ def test_lexer_removes_comments_without_changing_string_or_char_literals(): def test_lexer_removes_multiline_block_comments_but_preserves_following_line_numbers(): - from x2py.c_parser.lexer import lex_c_source + from x2py.parsers.c.lexer import lex_c_source tokens = lex_c_source( "int first;\n/* removed\n block */\nint second;\n", @@ -37,7 +37,7 @@ def test_lexer_removes_multiline_block_comments_but_preserves_following_line_num def test_line_continuations_preserve_original_line_numbers(): - from x2py.c_parser.preprocessor import normalize_c_source + from x2py.parsers.c.preprocessor import normalize_c_source normalized = normalize_c_source( "#define SUM(a, b) \\\n ((a) + (b))\nint x;\n", @@ -50,7 +50,7 @@ def test_line_continuations_preserve_original_line_numbers(): def test_top_level_split_helpers_ignore_nested_commas_and_function_bodies(): - from x2py.c_parser.lexer import split_top_level_c_source, top_level_split + from x2py.parsers.c.lexer import split_top_level_c_source, top_level_split assert top_level_split("int (*cmp)(int, int), int value") == [ "int (*cmp)(int, int)", @@ -69,8 +69,8 @@ def test_top_level_split_helpers_ignore_nested_commas_and_function_bodies(): def test_c_lexer_covers_linemarker_escapes_top_level_strings_and_eof_records(): - from x2py.c_parser import parse_c_file - from x2py.c_parser.lexer import ( + from x2py.parsers.c import parse_c_file + from x2py.parsers.c.lexer import ( CLogicalRecord, _unescape_linemarker_filename, lex_c_source, @@ -78,7 +78,7 @@ def test_c_lexer_covers_linemarker_escapes_top_level_strings_and_eof_records(): normalize_c_source, split_top_level_c_source, ) - from x2py.c_parser.preprocessor import _record_location + from x2py.parsers.c.preprocessor import _record_location assert _unescape_linemarker_filename(r"a\nb\rc\td\\e\"f\x") == 'a\nb\rc\td\\e"fx' assert _unescape_linemarker_filename("tail\\") == "tail\\" @@ -126,7 +126,7 @@ def test_c_lexer_covers_linemarker_escapes_top_level_strings_and_eof_records(): def test_c_lexer_mapping_helpers_cover_boundaries_ranges_and_position_updates(): - from x2py.c_parser.lexer import ( + from x2py.parsers.c.lexer import ( CLineMapping, _advance_position, _line_mapping, @@ -161,7 +161,7 @@ def test_c_lexer_mapping_helpers_cover_boundaries_ranges_and_position_updates(): def test_c_lexer_linemarker_and_directive_helpers_cover_raw_and_preprocessed_modes(): - from x2py.c_parser.lexer import ( + from x2py.parsers.c.lexer import ( CLineMapping, _blank_preprocessor_directives, _parse_linemarker, @@ -201,7 +201,7 @@ def test_c_lexer_linemarker_and_directive_helpers_cover_raw_and_preprocessed_mod def test_c_lexer_delimiter_helpers_cover_literals_nesting_offsets_and_validation(): - from x2py.c_parser.lexer import ( + from x2py.parsers.c.lexer import ( _scan_code_states, top_level_partition, top_level_split, @@ -233,7 +233,7 @@ def test_c_lexer_delimiter_helpers_cover_literals_nesting_offsets_and_validation def test_c_lexer_aggregate_attribute_helpers_preserve_shape_and_classify_headers(): - from x2py.c_parser.lexer import ( + from x2py.parsers.c.lexer import ( _balanced_invocation_end, _is_aggregate_definition_header, _is_braced_declaration_header, @@ -271,7 +271,7 @@ def test_c_lexer_aggregate_attribute_helpers_preserve_shape_and_classify_headers def test_c_lexer_comment_normalization_and_tokens_preserve_source_accounting(): - from x2py.c_parser.lexer import lex_c_source, normalize_c_source, strip_c_comments + from x2py.parsers.c.lexer import lex_c_source, normalize_c_source, strip_c_comments source = 'int first; // removed\nchar *text = "/* kept */"; /* block\n removed */ int second;\n' stripped = strip_c_comments(source) @@ -310,7 +310,7 @@ def test_c_lexer_comment_normalization_and_tokens_preserve_source_accounting(): def test_raw_mode_records_includes_without_expanding_them(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( '#include "api_types.h"\n#include \nint run(void);\n', @@ -323,7 +323,7 @@ def test_raw_mode_records_includes_without_expanding_them(): def test_raw_mode_resolves_local_includes_relative_to_path_input(tmp_path): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file header = tmp_path / "api.h" types = tmp_path / "api_types.h" @@ -339,8 +339,8 @@ def test_raw_mode_resolves_local_includes_relative_to_path_input(tmp_path): def test_c_preprocessor_helpers_cover_include_dirs_and_filesystem_errors(tmp_path, monkeypatch): from pathlib import Path - from x2py.c_parser.lexer import CLogicalRecord - from x2py.c_parser.preprocessor import _record_location, _resolve_local_include + from x2py.parsers.c.lexer import CLogicalRecord + from x2py.parsers.c.preprocessor import _record_location, _resolve_local_include include_dir = tmp_path / "include" include_dir.mkdir() @@ -383,7 +383,7 @@ def raise_one_os_error(path): def test_collect_preprocessor_metadata_preserves_locations_and_diagnostics(tmp_path): - from x2py.c_parser.preprocessor import collect_preprocessor_metadata + from x2py.parsers.c.preprocessor import collect_preprocessor_metadata include_dir = tmp_path / "include" include_dir.mkdir() @@ -447,7 +447,7 @@ def test_collect_preprocessor_metadata_preserves_locations_and_diagnostics(tmp_p ], ) def test_raw_mode_rejects_directives_that_require_preprocessing(directive): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError, match="require compiler preprocessing") as exc_info: parse_c_file(f"{directive}\nint run(void);\n", filename="raw_macro.h") @@ -457,7 +457,7 @@ def test_raw_mode_rejects_directives_that_require_preprocessing(directive): def test_raw_mode_accepts_trivial_include_guards_without_preprocessing(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -478,7 +478,7 @@ def test_raw_mode_accepts_trivial_include_guards_without_preprocessing(): def test_raw_mode_records_pragmas_as_metadata_without_hiding_declarations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -499,7 +499,7 @@ def test_raw_mode_records_pragmas_as_metadata_without_hiding_declarations(): def test_raw_mode_openmp_declaration_pragmas_do_not_hide_declarations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -525,7 +525,7 @@ def test_raw_mode_openmp_declaration_pragmas_do_not_hide_declarations(): def test_compiler_preprocessed_mode_accepts_line_markers_and_expanded_declarations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -552,7 +552,7 @@ def test_compiler_preprocessed_mode_accepts_line_markers_and_expanded_declaratio def test_compiler_preprocessed_mode_maps_gcc_linemarkers_across_includes_and_line_jumps(): - from x2py.c_parser import CComposedType, CFunctionType, CPointer, parse_c_file + from x2py.parsers.c import CComposedType, CFunctionType, CPointer, parse_c_file parsed = parse_c_file( """ @@ -604,7 +604,7 @@ def test_compiler_preprocessed_mode_maps_gcc_linemarkers_across_includes_and_lin def test_compiler_preprocessed_mode_maps_nested_aggregate_members_to_original_file(): - from x2py.c_parser import CStruct, parse_c_file + from x2py.parsers.c import CStruct, parse_c_file parsed = parse_c_file( """ @@ -633,7 +633,7 @@ def test_compiler_preprocessed_mode_maps_nested_aggregate_members_to_original_fi def test_compiler_preprocessed_mode_maps_fatal_parse_errors_to_original_file(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError) as exc_info: parse_c_file( diff --git a/tests/parsing/c/test_c_model_serialization.py b/tests/parsing/c/test_c_model_serialization.py index 99f33faea..ff35b39cc 100644 --- a/tests/parsing/c/test_c_model_serialization.py +++ b/tests/parsing/c/test_c_model_serialization.py @@ -5,7 +5,7 @@ import inspect from types import SimpleNamespace -import x2py.c_parser.models as models +import x2py.parsers.c.models as models def _type_payload(model: str, **extra): diff --git a/tests/parsing/c/test_c_parser_developer_tutorial.py b/tests/parsing/c/test_c_parser_developer_tutorial.py index 15c1abc58..fd22fce11 100644 --- a/tests/parsing/c/test_c_parser_developer_tutorial.py +++ b/tests/parsing/c/test_c_parser_developer_tutorial.py @@ -7,7 +7,7 @@ def test_tutorial_shared_declarator_backend_builds_layered_variable_type(): - from x2py.c_parser import CArray, CConst, CInt, CParser, CPointer + from x2py.parsers.c import CArray, CConst, CInt, CParser, CPointer parser = CParser() specifiers, declarator = parser._split_declaration_specifiers("const int *values[4]") @@ -26,7 +26,7 @@ def test_tutorial_shared_declarator_backend_builds_layered_variable_type(): def test_tutorial_parse_file_dispatches_declaration_roles_through_one_model(): - from x2py.c_parser import CParser, CStruct + from x2py.parsers.c import CParser, CStruct parsed = CParser().parse_file( """ @@ -47,7 +47,7 @@ def test_tutorial_parse_file_dispatches_declaration_roles_through_one_model(): def test_tutorial_preprocessed_input_reuses_parsing_and_remaps_locations(): - from x2py.c_parser import CParser + from x2py.parsers.c import CParser parsed = CParser().parse_file( '# 24 "include/api.h"\nint expanded_api(void);\n', diff --git a/tests/parsing/c/test_c_project_resolution.py b/tests/parsing/c/test_c_project_resolution.py index fb0cfb213..ece5322b9 100644 --- a/tests/parsing/c/test_c_project_resolution.py +++ b/tests/parsing/c/test_c_project_resolution.py @@ -4,7 +4,7 @@ def test_project_include_graph_tracks_local_system_missing_and_cycles(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "a.h").write_text('#include "b.h"\n#include "missing.h"\n', encoding="utf-8") (tmp_path / "b.h").write_text('#include "a.h"\n#include \n', encoding="utf-8") @@ -19,7 +19,7 @@ def test_project_include_graph_tracks_local_system_missing_and_cycles(tmp_path: def test_project_resolves_quoted_includes_through_include_dirs(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project include_dir = tmp_path / "include" src_dir = tmp_path / "src" @@ -39,7 +39,7 @@ def test_project_resolves_quoted_includes_through_include_dirs(tmp_path: Path): def test_project_records_local_include_without_recursively_parsing_resolved_header(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project include_dir = tmp_path / "generated" include_dir.mkdir() @@ -57,7 +57,7 @@ def test_project_records_local_include_without_recursively_parsing_resolved_head def test_project_records_system_include_without_searching_or_parsing_local_copy(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project local_system_header = tmp_path / "stddef.h" api = tmp_path / "api.h" @@ -73,7 +73,7 @@ def test_project_records_system_include_without_searching_or_parsing_local_copy( def test_parse_c_project_directory_discovers_preprocessed_i_files(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.c").write_text("int from_source(void);\n", encoding="utf-8") (tmp_path / "generated.i").write_text( @@ -95,7 +95,7 @@ def test_parse_c_project_directory_discovers_preprocessed_i_files(tmp_path: Path def test_project_indexes_functions_by_file_and_enum_constants(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.h").write_text( "enum status { STATUS_OK = 0, STATUS_ERROR = -1 };\nint run(void);\nint stop(void);\n", @@ -110,7 +110,7 @@ def test_project_indexes_functions_by_file_and_enum_constants(tmp_path: Path): def test_project_indexes_file_scope_variables(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.h").write_text( "extern int global_count;\n", @@ -123,7 +123,7 @@ def test_project_indexes_file_scope_variables(tmp_path: Path): def test_project_function_index_prefers_definition_over_compatible_prototype(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.h").write_text("int solve(int value);\n", encoding="utf-8") (tmp_path / "api.c").write_text("int solve(int value) { return value; }\n", encoding="utf-8") @@ -136,7 +136,7 @@ def test_project_function_index_prefers_definition_over_compatible_prototype(tmp def test_project_reports_conflicting_function_declarations(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "a.h").write_text("int work(int value);\n", encoding="utf-8") (tmp_path / "b.h").write_text("double work(double value);\n", encoding="utf-8") @@ -147,7 +147,7 @@ def test_project_reports_conflicting_function_declarations(tmp_path: Path): def test_project_resolves_typedefs_and_struct_tags_across_files(tmp_path: Path): - from x2py.c_parser import CComposedType, CTypedef, parse_c_project + from x2py.parsers.c import CComposedType, CTypedef, parse_c_project (tmp_path / "types.h").write_text( "typedef unsigned long api_size;\nstruct state { int id; };\n", @@ -168,7 +168,7 @@ def test_project_resolves_typedefs_and_struct_tags_across_files(tmp_path: Path): def test_project_completes_forward_struct_tags_regardless_of_file_order(): - from x2py.c_parser import CComposedType, parse_c_project + from x2py.parsers.c import CComposedType, parse_c_project project = parse_c_project( { @@ -185,7 +185,7 @@ def test_project_completes_forward_struct_tags_regardless_of_file_order(): def test_project_keeps_complete_union_definition_when_forward_seen_later(): - from x2py.c_parser import CComposedType, parse_c_project + from x2py.parsers.c import CComposedType, parse_c_project project = parse_c_project( { @@ -202,7 +202,7 @@ def test_project_keeps_complete_union_definition_when_forward_seen_later(): def test_project_resolves_typedef_chains_while_preserving_alias_objects(tmp_path: Path): - from x2py.c_parser import CTypedef, CUnsignedLong, parse_c_project + from x2py.parsers.c import CTypedef, CUnsignedLong, parse_c_project (tmp_path / "types.h").write_text( "typedef unsigned long raw_size;\ntypedef raw_size api_size;\n", @@ -220,7 +220,7 @@ def test_project_resolves_typedef_chains_while_preserving_alias_objects(tmp_path def test_project_resolves_typedefs_for_variables_and_aggregate_members(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( { @@ -234,7 +234,7 @@ def test_project_resolves_typedefs_for_variables_and_aggregate_members(): def test_project_reports_each_typedef_cycle_once_with_structured_diagnostic(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project({"cycle.h": "typedef b a;\ntypedef a b;\n"}) @@ -249,7 +249,7 @@ def test_project_reports_each_typedef_cycle_once_with_structured_diagnostic(): def test_project_reports_prefixed_typedef_cycle_without_including_acyclic_alias(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( {"cycle.h": ("typedef inner_a alias;\ntypedef inner_b inner_a;\ntypedef inner_a inner_b;\n")} @@ -262,7 +262,7 @@ def test_project_reports_prefixed_typedef_cycle_without_including_acyclic_alias( def test_project_resolves_function_typedef_signature_references(): - from x2py.c_parser import CComposedType, CFunctionType, parse_c_project + from x2py.parsers.c import CComposedType, CFunctionType, parse_c_project project = parse_c_project( { @@ -284,7 +284,7 @@ def test_project_resolves_function_typedef_signature_references(): def test_project_resolves_parameter_declared_type_signature_references(): - from x2py.c_parser import CComposedType, CFunctionType, parse_c_project + from x2py.parsers.c import CComposedType, CFunctionType, parse_c_project project = parse_c_project( {"callbacks.h": ("typedef unsigned long api_size;\nvoid apply(api_size callback(api_size));\n")} @@ -300,7 +300,7 @@ def test_project_resolves_parameter_declared_type_signature_references(): def test_project_resolves_parameter_declared_array_references(): - from x2py.c_parser import CComposedType, parse_c_project + from x2py.parsers.c import CComposedType, parse_c_project project = parse_c_project({"arrays.h": ("typedef unsigned long api_size;\nvoid collect(api_size values[4]);\n")}) @@ -313,7 +313,7 @@ def test_project_resolves_parameter_declared_array_references(): def test_project_reuses_typedef_cycle_state_across_resolved_use_sites(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( { @@ -333,7 +333,7 @@ def test_project_reuses_typedef_cycle_state_across_resolved_use_sites(): def test_project_resolves_union_and_enum_tag_references(tmp_path: Path): - from x2py.c_parser import CComposedType, parse_c_project + from x2py.parsers.c import CComposedType, parse_c_project (tmp_path / "types.h").write_text( "union value { int i; };\nenum status { STATUS_OK = 0 };\n", @@ -353,7 +353,7 @@ def test_project_resolves_union_and_enum_tag_references(tmp_path: Path): def test_project_resolves_opaque_pointer_typedefs_across_files(tmp_path: Path): - from x2py.c_parser import CComposedType, CTypedef, parse_c_project + from x2py.parsers.c import CComposedType, CTypedef, parse_c_project (tmp_path / "types.h").write_text( "struct handle;\ntypedef struct handle *handle_t;\n", @@ -371,7 +371,7 @@ def test_project_resolves_opaque_pointer_typedefs_across_files(tmp_path: Path): def test_project_preserves_unresolved_type_references_for_later_diagnostics(): - from x2py.c_parser import CTypedef, parse_c_project + from x2py.parsers.c import CTypedef, parse_c_project project = parse_c_project({"api.h": "missing_type value(void);\n"}) @@ -381,7 +381,7 @@ def test_project_preserves_unresolved_type_references_for_later_diagnostics(): def test_project_preserves_unresolved_tag_references_for_later_diagnostics(): - from x2py.c_parser import CComposedType, CEnum, CStruct, CUnion, parse_c_project + from x2py.parsers.c import CComposedType, CEnum, CStruct, CUnion, parse_c_project project = parse_c_project( {"api.h": ("struct missing *get_struct(void);\nunion absent *get_union(void);\nenum unknown get_enum(void);\n")} @@ -401,7 +401,7 @@ def test_project_preserves_unresolved_tag_references_for_later_diagnostics(): def test_project_header_source_pairs_use_matching_stems_and_direct_includes(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "solver.h").write_text("int solve(void);\n", encoding="utf-8") (tmp_path / "solver.c").write_text('#include "solver.h"\n', encoding="utf-8") @@ -415,7 +415,7 @@ def test_project_header_source_pairs_use_matching_stems_and_direct_includes(tmp_ def test_project_header_source_pairs_preserve_many_to_many_relationships(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "a.h").write_text("int a(void);\n", encoding="utf-8") (tmp_path / "b.h").write_text("int b(void);\n", encoding="utf-8") @@ -429,7 +429,7 @@ def test_project_header_source_pairs_preserve_many_to_many_relationships(tmp_pat def test_project_serialization_keeps_include_indexes_json_stable(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.h").write_text("#include \nint run(void);\n", encoding="utf-8") diff --git a/tests/parsing/c/test_c_public_api_skeleton.py b/tests/parsing/c/test_c_public_api_skeleton.py index d32978e58..8871b940b 100644 --- a/tests/parsing/c/test_c_public_api_skeleton.py +++ b/tests/parsing/c/test_c_public_api_skeleton.py @@ -4,7 +4,7 @@ def test_c_parser_path_and_include_key_helpers_preserve_boundary_contracts(monkeypatch): - from x2py.c_parser.parser import _include_key_from_current, _looks_like_existing_source_path + from x2py.parsers.c.parser import _include_key_from_current, _looks_like_existing_source_path monkeypatch.setattr(Path, "is_file", lambda self: True) @@ -25,7 +25,7 @@ def raise_os_error(path): def test_c_parser_public_wrappers_forward_explicit_options(monkeypatch): - from x2py.c_parser import parse_c_file, parse_c_project + from x2py.parsers.c import parse_c_file, parse_c_project calls = [] @@ -38,7 +38,7 @@ def parse_project(self, *args, **kwargs): calls.append(("project", args, kwargs)) return "project-result" - monkeypatch.setattr("x2py.c_parser.parser._DEFAULT_PARSER", RecordingParser()) + monkeypatch.setattr("x2py.parsers.c.parser._DEFAULT_PARSER", RecordingParser()) include_dirs = [Path("include")] assert ( @@ -84,7 +84,7 @@ def parse_project(self, *args, **kwargs): def test_parse_c_file_accepts_inline_source_and_returns_typed_model(): - from x2py.c_parser import CFile, parse_c_file + from x2py.parsers.c import CFile, parse_c_file parsed = parse_c_file("int add(int a, int b);\n", filename="inline.h") @@ -106,7 +106,7 @@ def test_x2py_exports_c_file_and_project_entrypoints_like_fortran(): def test_parse_c_file_accepts_path_input_and_preserves_filename(tmp_path: Path): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file header = tmp_path / "api.h" header.write_text("double scale(double x);\n", encoding="utf-8") @@ -118,7 +118,7 @@ def test_parse_c_file_accepts_path_input_and_preserves_filename(tmp_path: Path): def test_parse_c_file_accepts_empty_source_and_unknown_suffix(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("", filename="empty.src") @@ -130,14 +130,14 @@ def test_parse_c_file_accepts_empty_source_and_unknown_suffix(): def test_parse_c_file_rejects_unknown_preprocessing_mode(): import pytest - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file with pytest.raises(ValueError, match="preprocessing mode"): parse_c_file("int answer(void);\n", filename="api.h", preprocessing="unknown") def test_parse_c_project_accepts_mapping_sources(): - from x2py.c_parser import CProject, parse_c_project + from x2py.parsers.c import CProject, parse_c_project project = parse_c_project( { @@ -153,7 +153,7 @@ def test_parse_c_project_accepts_mapping_sources(): def test_parse_c_project_accepts_single_file_path(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project source = tmp_path / "api.c" source.write_text("int answer(void);\n", encoding="utf-8") @@ -165,7 +165,7 @@ def test_parse_c_project_accepts_single_file_path(tmp_path: Path): def test_parse_c_project_indexes_forward_structs_by_tag_name(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( { @@ -180,7 +180,7 @@ def test_parse_c_project_indexes_forward_structs_by_tag_name(): def test_parse_c_project_indexes_named_union_and_enum_tags(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( { @@ -193,7 +193,7 @@ def test_parse_c_project_indexes_named_union_and_enum_tags(): def test_parse_c_project_accepts_directory_input_with_c_and_h_files(tmp_path: Path): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project (tmp_path / "api.h").write_text("int add(int a, int b);\n", encoding="utf-8") (tmp_path / "api.c").write_text('#include "api.h"\n', encoding="utf-8") @@ -205,7 +205,7 @@ def test_parse_c_project_accepts_directory_input_with_c_and_h_files(tmp_path: Pa def test_c_file_serialization_is_json_stable(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("", filename="empty.c") @@ -229,7 +229,7 @@ def test_c_file_serialization_is_json_stable(): def test_concrete_type_serialization_preserves_semantic_type_fields_and_locations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "typedef int (*compare_fn)(const void *, const void *);\ncompare_fn select_compare(void);\n", @@ -255,7 +255,7 @@ def test_concrete_type_serialization_preserves_semantic_type_fields_and_location def test_parameter_adjustment_serialization_preserves_declared_and_effective_types(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file payload = parse_c_file( "void process(int values[4], int callback(int));\n", @@ -272,7 +272,7 @@ def test_parameter_adjustment_serialization_preserves_declared_and_effective_typ def test_inline_aggregate_typedef_serialization_uses_references_without_cycles(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file payload = parse_c_file( "typedef struct node { struct node *next; } node_t;\n", @@ -286,7 +286,7 @@ def test_inline_aggregate_typedef_serialization_uses_references_without_cycles() def test_unresolved_typedef_reference_metadata_is_preserved_in_json(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file payload = parse_c_file("api_size count(void);\n", filename="unresolved.h").to_dict() @@ -297,7 +297,7 @@ def test_unresolved_typedef_reference_metadata_is_preserved_in_json(): def test_c_parser_instance_entrypoints_match_public_functions(): - from x2py.c_parser import CParser, parse_c_file, parse_c_project + from x2py.parsers.c import CParser, parse_c_file, parse_c_project source = "int answer(void);\n" parser = CParser() @@ -310,7 +310,7 @@ def test_c_parser_instance_entrypoints_match_public_functions(): def test_c_parse_error_attributes_and_diagnostic_formatting(): - from x2py.c_parser import CArray, CComposedType, CInt, CParseError, CPointer, CSourceLocation + from x2py.parsers.c import CArray, CComposedType, CInt, CParseError, CPointer, CSourceLocation err = CParseError( "unexpected token", @@ -338,7 +338,7 @@ def test_c_parse_error_attributes_and_diagnostic_formatting(): def test_c_parse_error_color_and_no_color_formatting(): - from x2py.c_parser import CParseError + from x2py.parsers.c import CParseError err = CParseError( "unexpected token", diff --git a/tests/parsing/c/test_c_structs_unions_enums_typedefs.py b/tests/parsing/c/test_c_structs_unions_enums_typedefs.py index 353bfb8fa..47c825e93 100644 --- a/tests/parsing/c/test_c_structs_unions_enums_typedefs.py +++ b/tests/parsing/c/test_c_structs_unions_enums_typedefs.py @@ -4,7 +4,7 @@ def test_named_struct_members_are_variables_in_source_order(): - from x2py.c_parser import CArray, CComposedType, CVariable, parse_c_file + from x2py.parsers.c import CArray, CComposedType, CVariable, parse_c_file parsed = parse_c_file( "struct point { double x; double y; double coordinates[2]; };\n", @@ -20,7 +20,7 @@ def test_named_struct_members_are_variables_in_source_order(): def test_typedef_struct_alias_refers_to_the_concrete_struct_object(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "typedef struct point { double x; double y; } point_t;\n", @@ -33,7 +33,7 @@ def test_typedef_struct_alias_refers_to_the_concrete_struct_object(): def test_forward_struct_declaration_is_completed_by_later_definition(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "struct state;\nstruct state { int id; };\n", @@ -47,7 +47,7 @@ def test_forward_struct_declaration_is_completed_by_later_definition(): def test_duplicate_complete_tag_definitions_report_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( "struct state { int id; };\nstruct state { int id; };\n", @@ -59,7 +59,7 @@ def test_duplicate_complete_tag_definitions_report_diagnostics(): def test_anonymous_struct_typedef_gets_stable_anonymous_id(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file("typedef struct { int code; } result_t;\n", filename="anon_struct.h") @@ -69,7 +69,7 @@ def test_anonymous_struct_typedef_gets_stable_anonymous_id(): def test_union_members_are_variables_without_struct_field_class(): - from x2py.c_parser import CUnion, CVariable, parse_c_file + from x2py.parsers.c import CUnion, CVariable, parse_c_file parsed = parse_c_file("union value { int i; double d; };\n", filename="union.h") @@ -80,7 +80,7 @@ def test_union_members_are_variables_without_struct_field_class(): def test_anonymous_union_typedef_refers_to_the_concrete_union_object(): - from x2py.c_parser import CUnion, parse_c_file + from x2py.parsers.c import CUnion, parse_c_file parsed = parse_c_file("typedef union { int i; double d; } value_t;\n", filename="anon_union.h") @@ -90,7 +90,7 @@ def test_anonymous_union_typedef_refers_to_the_concrete_union_object(): def test_function_signatures_using_unions_by_value_report_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -117,7 +117,7 @@ def test_function_signatures_using_unions_by_value_report_diagnostics(): def test_project_reports_union_by_value_through_resolved_typedefs(): - from x2py.c_parser import parse_c_project + from x2py.parsers.c import parse_c_project project = parse_c_project( { @@ -135,7 +135,7 @@ def test_project_reports_union_by_value_through_resolved_typedefs(): def test_incomplete_union_and_tag_typedef_aliases_use_concrete_tag_classes(): - from x2py.c_parser import CStruct, CUnion, parse_c_file + from x2py.parsers.c import CStruct, CUnion, parse_c_file parsed = parse_c_file( "struct handle;\nunion payload;\ntypedef struct handle handle_t;\ntypedef union payload payload_t;\n", @@ -152,7 +152,7 @@ def test_incomplete_union_and_tag_typedef_aliases_use_concrete_tag_classes(): def test_repeated_union_and_enum_tags_normalize_with_duplicate_diagnostics(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -174,7 +174,7 @@ def test_repeated_union_and_enum_tags_normalize_with_duplicate_diagnostics(): def test_enum_constants_preserve_explicit_implicit_and_symbolic_values(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """ @@ -197,7 +197,7 @@ def test_enum_constants_preserve_explicit_implicit_and_symbolic_values(): def test_typedef_enum_and_trailing_tag_variable_are_separate_objects(): - from x2py.c_parser import CEnum, CStruct, parse_c_file + from x2py.parsers.c import CEnum, CStruct, parse_c_file parsed = parse_c_file( "typedef enum { FLAG_NONE = 0, FLAG_READ = 1 } flag_t;\nstruct point { int x; } origin;\n", @@ -213,7 +213,7 @@ def test_typedef_enum_and_trailing_tag_variable_are_separate_objects(): def test_recursive_struct_pointer_uses_an_incomplete_struct_component_without_cycles(): - from x2py.c_parser import CComposedType, CPointer, CStruct, parse_c_file + from x2py.parsers.c import CComposedType, CPointer, CStruct, parse_c_file parsed = parse_c_file( "typedef struct node { int value; struct node *next; } node_t;\n", @@ -230,7 +230,7 @@ def test_recursive_struct_pointer_uses_an_incomplete_struct_component_without_cy def test_typedef_chains_preserve_typedef_objects_before_resolution(): - from x2py.c_parser import CTypedef, CUnsignedLong, parse_c_file + from x2py.parsers.c import CTypedef, CUnsignedLong, parse_c_file parsed = parse_c_file( "typedef unsigned long size_type;\ntypedef size_type api_size;\napi_size count(void);\n", @@ -245,7 +245,7 @@ def test_typedef_chains_preserve_typedef_objects_before_resolution(): def test_struct_members_use_same_components_for_callbacks_arrays_and_bitfields(): - from x2py.c_parser import CArray, CFunctionType, CPointer, parse_c_file + from x2py.parsers.c import CArray, CFunctionType, CPointer, parse_c_file parsed = parse_c_file( "struct hooks { int (*compare)(const void *, const void *); unsigned enabled : 1; int values[4]; };\n", @@ -261,7 +261,7 @@ def test_struct_members_use_same_components_for_callbacks_arrays_and_bitfields() def test_struct_members_preserve_precise_locations_and_legal_flexible_array_metadata(): - from x2py.c_parser import CArray, parse_c_file + from x2py.parsers.c import CArray, parse_c_file parsed = parse_c_file( """struct packet { @@ -315,7 +315,7 @@ def test_struct_members_preserve_precise_locations_and_legal_flexible_array_meta ], ) def test_invalid_flexible_array_members_are_diagnosed(source, owner_name, message): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file(source, filename="invalid_flexible.h") aggregate = getattr(parsed, owner_name)[0] @@ -331,7 +331,7 @@ def test_invalid_flexible_array_members_are_diagnosed(source, owner_name, messag def test_unnamed_and_zero_width_bitfields_preserve_source_facts_and_locations(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """struct flags { @@ -352,7 +352,7 @@ def test_unnamed_and_zero_width_bitfields_preserve_source_facts_and_locations(): def test_nested_aggregate_member_definition_builds_the_nested_type(): - from x2py.c_parser import CStruct, CUnion, parse_c_file + from x2py.parsers.c import CStruct, CUnion, parse_c_file parsed = parse_c_file( """struct outer { @@ -376,7 +376,7 @@ def test_nested_aggregate_member_definition_builds_the_nested_type(): def test_anonymous_aggregate_member_without_a_declarator_is_retained(): - from x2py.c_parser import CUnion, parse_c_file + from x2py.parsers.c import CUnion, parse_c_file parsed = parse_c_file( "struct flags { union { int integer; float real; }; int tag; };\n", @@ -392,7 +392,7 @@ def test_anonymous_aggregate_member_without_a_declarator_is_retained(): def test_struct_field_missing_semicolon_reports_syntax_location(): - from x2py.c_parser import CParseError, parse_c_file + from x2py.parsers.c import CParseError, parse_c_file with pytest.raises(CParseError) as exc_info: parse_c_file( @@ -412,7 +412,7 @@ def test_struct_field_missing_semicolon_reports_syntax_location(): def test_nested_aggregate_field_with_function_declarator_is_rejected(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """struct outer { @@ -434,7 +434,7 @@ def test_nested_aggregate_field_with_function_declarator_is_rejected(): def test_bad_field_declarator_does_not_stop_later_declarators(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """struct bad { @@ -450,7 +450,7 @@ def test_bad_field_declarator_does_not_stop_later_declarators(): def test_unnamed_field_type_without_bit_width_reports_diagnostic(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """struct bad { @@ -472,7 +472,7 @@ def test_unnamed_field_type_without_bit_width_reports_diagnostic(): def test_unsupported_field_declarator_is_reported_at_member_location(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file parsed = parse_c_file( """struct bad { diff --git a/tests/parsing/fortran/_procedure_support.py b/tests/parsing/fortran/_procedure_support.py index 98f15603b..41d02937e 100644 --- a/tests/parsing/fortran/_procedure_support.py +++ b/tests/parsing/fortran/_procedure_support.py @@ -1,6 +1,6 @@ import pytest -from x2py.fortran_parser.models import FortranFunctionCall, FortranSlice, FortranUseMapping, FortranVariable +from x2py.parsers.fortran.models import FortranFunctionCall, FortranSlice, FortranUseMapping, FortranVariable from x2py import FortranParseError, parse_fortran_file, parse_fortran_project diff --git a/tests/parsing/fortran/_regression_support.py b/tests/parsing/fortran/_regression_support.py index 86de5c988..95742acc4 100644 --- a/tests/parsing/fortran/_regression_support.py +++ b/tests/parsing/fortran/_regression_support.py @@ -6,9 +6,9 @@ import pytest -from x2py.fortran_parser.models import FortranArgument, FortranDerivedType, FortranModule, FortranProcedureSignature +from x2py.parsers.fortran.models import FortranArgument, FortranDerivedType, FortranModule, FortranProcedureSignature -from x2py.fortran_parser.parser import ( +from x2py.parsers.fortran.parser import ( FortranParser, SourceUnit, _ParserScope, diff --git a/tests/parsing/fortran/test_declaration_and_interface_edges.py b/tests/parsing/fortran/test_declaration_and_interface_edges.py index 5506c3ed4..6215b4b67 100644 --- a/tests/parsing/fortran/test_declaration_and_interface_edges.py +++ b/tests/parsing/fortran/test_declaration_and_interface_edges.py @@ -2,8 +2,8 @@ import pytest -from x2py.fortran_parser.models import FortranModule -from x2py.fortran_parser.parser import FortranParser, _ParserScope +from x2py.parsers.fortran.models import FortranModule +from x2py.parsers.fortran.parser import FortranParser, _ParserScope from x2py import FortranParseError, parse_fortran_file, parse_fortran_project diff --git a/tests/parsing/fortran/test_declarations_and_shapes.py b/tests/parsing/fortran/test_declarations_and_shapes.py index f4d7eda7a..17d595fdf 100644 --- a/tests/parsing/fortran/test_declarations_and_shapes.py +++ b/tests/parsing/fortran/test_declarations_and_shapes.py @@ -369,7 +369,7 @@ def test_shape_info_for_explicit_extent_dimension(): def test_structured_shape_handles_empty_dimensions_and_use_mapping_equality(): - from x2py.fortran_parser.type_resolver import extract_kind_from_type_spec + from x2py.parsers.fortran.type_resolver import extract_kind_from_type_spec var = FortranVariable(name="empty", shape=[""]) assert var.shape_info == [{"raw": "", "lower": None, "upper": None}] @@ -400,7 +400,7 @@ def test_structured_shape_handles_empty_dimensions_and_use_mapping_equality(): ], ) def test_extract_kind_from_type_spec_contract(base_type, type_spec, expected): - from x2py.fortran_parser.type_resolver import extract_kind_from_type_spec + from x2py.parsers.fortran.type_resolver import extract_kind_from_type_spec assert extract_kind_from_type_spec(base_type, type_spec) == expected diff --git a/tests/parsing/fortran/test_developer_tutorial.py b/tests/parsing/fortran/test_developer_tutorial.py index 59894617a..6d4140d7b 100644 --- a/tests/parsing/fortran/test_developer_tutorial.py +++ b/tests/parsing/fortran/test_developer_tutorial.py @@ -2,7 +2,7 @@ This test is intentionally written as a small walkthrough rather than as a black-box public API test. It shows the private visitor/helper sequence that -maintainers should follow when changing `x2py/fortran_parser/parser.py`: +maintainers should follow when changing `x2py/parsers/fortran/parser.py`: 1. preprocess and slice file-level source units, 2. split one unit into grammar parts, @@ -10,7 +10,7 @@ 4. recursively slice and inspect its direct children. """ -from x2py.fortran_parser.parser import FortranParser +from x2py.parsers.fortran.parser import FortranParser def test_developer_tutorial_recursive_unit_visitors_and_helpers(): diff --git a/tests/parsing/fortran/test_public_entrypoints.py b/tests/parsing/fortran/test_public_entrypoints.py index 05ea3e48a..9dd7a9828 100644 --- a/tests/parsing/fortran/test_public_entrypoints.py +++ b/tests/parsing/fortran/test_public_entrypoints.py @@ -2,10 +2,10 @@ import pytest -from x2py.fortran_parser.parser import FortranParser +from x2py.parsers.fortran.parser import FortranParser from x2py import FortranParseError, parse_fortran_file, parse_fortran_project -from x2py.c_parser.parser import parse_c_file -from x2py.fortran_parser.parser import FortranParser as PackageFortranParser +from x2py.parsers.c.parser import parse_c_file +from x2py.parsers.fortran.parser import FortranParser as PackageFortranParser from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules diff --git a/tests/parsing/fortran/test_scope_handling.py b/tests/parsing/fortran/test_scope_handling.py index b47fd9dba..730e6b9ee 100644 --- a/tests/parsing/fortran/test_scope_handling.py +++ b/tests/parsing/fortran/test_scope_handling.py @@ -1,6 +1,6 @@ import pytest -from x2py.fortran_parser.models import FortranParseError +from x2py.parsers.fortran.models import FortranParseError from x2py import parse_fortran_file diff --git a/tests/pipeline/preprocessing/test_parser_boundaries.py b/tests/pipeline/preprocessing/test_parser_boundaries.py index 04798b53e..0bf176374 100644 --- a/tests/pipeline/preprocessing/test_parser_boundaries.py +++ b/tests/pipeline/preprocessing/test_parser_boundaries.py @@ -10,7 +10,7 @@ def test_fortran_lexer_strip_comment_preserves_directives_and_quoted_bangs(): - from x2py.fortran_parser.lexer import strip_comment + from x2py.parsers.fortran.lexer import strip_comment assert strip_comment(" !$OMP parallel do", "free") == "!$OMP parallel do" assert strip_comment("C$OMP PARALLEL DO", "fixed") == "!$omp PARALLEL DO" @@ -28,7 +28,7 @@ def test_fortran_lexer_strip_comment_preserves_directives_and_quoted_bangs(): def test_fortran_lexer_preprocess_lines_folds_free_and_fixed_continuations(): - from x2py.fortran_parser.lexer import preprocess_lines + from x2py.parsers.fortran.lexer import preprocess_lines free = "alpha = one &\n & + two ! removed\n\nbeta = 3\n! removed\n" assert preprocess_lines(free, filename="free.f90") == [ diff --git a/tests/semantics/conversion/_property_support.py b/tests/semantics/conversion/_property_support.py index 5d37848f8..3de9842cb 100644 --- a/tests/semantics/conversion/_property_support.py +++ b/tests/semantics/conversion/_property_support.py @@ -10,7 +10,7 @@ from hypothesis import given, strategies as st -from x2py.c_parser import parse_c_file +from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules diff --git a/tests/semantics/conversion/c/_support.py b/tests/semantics/conversion/c/_support.py index ef341ea0e..ffafb65e0 100644 --- a/tests/semantics/conversion/c/_support.py +++ b/tests/semantics/conversion/c/_support.py @@ -6,9 +6,9 @@ import pytest -from x2py.c_parser import parse_c_file, parse_c_project +from x2py.parsers.c import parse_c_file, parse_c_project -from x2py.c_parser.models import ( +from x2py.parsers.c.models import ( CArray, CAtomic, CBool, diff --git a/tests/semantics/conversion/fortran/_support.py b/tests/semantics/conversion/fortran/_support.py index d81c8b308..54f1eeebb 100644 --- a/tests/semantics/conversion/fortran/_support.py +++ b/tests/semantics/conversion/fortran/_support.py @@ -8,7 +8,7 @@ import pytest -from x2py.fortran_parser.models import ( +from x2py.parsers.fortran.models import ( FortranArgument, FortranBlockData, FortranDerivedType, diff --git a/tests/semantics/policy/test_wrapper_policy.py b/tests/semantics/policy/test_wrapper_policy.py index 79a28bab6..a1416e9ff 100644 --- a/tests/semantics/policy/test_wrapper_policy.py +++ b/tests/semantics/policy/test_wrapper_policy.py @@ -3,7 +3,7 @@ import pytest from tests.wrapper.fortran._support import wrapper_source -from x2py.fortran_parser.parser import parse_fortran_project +from x2py.parsers.fortran.parser import parse_fortran_project from x2py.pipeline.build import _apply_source_python_exports, _fortran_source_for_pipeline, _merge_wrapper_modules from x2py.pipeline.preprocessing import PreprocessingConfig from x2py.pipeline.pyi import pyi_file_to_semantic_module diff --git a/tests/semantics/readiness/test_c_readiness.py b/tests/semantics/readiness/test_c_readiness.py index 9351287f9..443db8502 100644 --- a/tests/semantics/readiness/test_c_readiness.py +++ b/tests/semantics/readiness/test_c_readiness.py @@ -6,7 +6,7 @@ def test_c_semantic_readiness_accepts_plain_primitive_function_signatures(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -25,7 +25,7 @@ def test_c_semantic_readiness_accepts_plain_primitive_function_signatures(): def test_c_semantic_readiness_reports_unresolved_typedefs(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -38,7 +38,7 @@ def test_c_semantic_readiness_reports_unresolved_typedefs(): def test_c_semantic_readiness_reports_variadic_functions_as_blockers(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -51,7 +51,7 @@ def test_c_semantic_readiness_reports_variadic_functions_as_blockers(): def test_c_semantic_readiness_reports_callback_policy_required(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -93,7 +93,7 @@ def each_item( def test_c_semantic_readiness_reports_pointer_ownership_ambiguity(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -106,7 +106,7 @@ def test_c_semantic_readiness_reports_pointer_ownership_ambiguity(): def test_c_semantic_readiness_accepts_enum_values_and_blocks_mutable_enum_pointers(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness @@ -128,7 +128,7 @@ def test_c_semantic_readiness_accepts_enum_values_and_blocks_mutable_enum_pointe def test_c_semantic_readiness_aggregates_file_and_function_blockers(): - from x2py.c_parser import parse_c_file + from x2py.parsers.c import parse_c_file from x2py.semantics.c2ir import c_file_to_semantic_modules from x2py.semantics.readiness import assess_semantic_wrap_readiness diff --git a/tests/tools/test_check_radon_policy.py b/tests/tools/test_check_radon_policy.py index 04ec10fe8..a049f38ab 100644 --- a/tests/tools/test_check_radon_policy.py +++ b/tests/tools/test_check_radon_policy.py @@ -59,7 +59,7 @@ def test_policy_tracks_hotspot_average_without_changed_base(tmp_path: Path): def test_source_root_filter_uses_path_boundaries(): - assert is_under_source_roots("x2py/c_parser/parser.py", ("x2py",)) + assert is_under_source_roots("x2py/parsers/c/parser.py", ("x2py",)) assert is_under_source_roots("x2py", ("x2py",)) assert not is_under_source_roots("x2py_extra/parser.py", ("x2py",)) diff --git a/tests/wrapper/fortran/_support.py b/tests/wrapper/fortran/_support.py index 93fa3ac36..fd06881d9 100644 --- a/tests/wrapper/fortran/_support.py +++ b/tests/wrapper/fortran/_support.py @@ -15,7 +15,7 @@ from tests.wrapper.fortran.fmath_cases import fmath_cases from x2py import build_pyi_extension from x2py.compiling.basic import CompileObj -from x2py.fortran_parser.parser import parse_fortran_project +from x2py.parsers.fortran.parser import parse_fortran_project from x2py.pipeline.build import ( _apply_source_python_exports, _build_rendered_wrapper_extension, diff --git a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py index 71dca7f29..293536845 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py +++ b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py @@ -109,7 +109,6 @@ def _build_contract( "-m", "x2py", str(entry), - "--wrap", "--native-objects", str(native_object), "--native-include-dir", @@ -174,7 +173,6 @@ def test_source_build_preserves_modules_and_root_externals(tmp_path: Path): "-m", "x2py", str(source), - "--wrap", "--out-dir", str(build_dir), "--json", diff --git a/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py b/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py index 2f669bcbc..73d455e01 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py +++ b/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py @@ -102,7 +102,6 @@ def _build_pyi_cli(pyi_path: Path, native_object: Path, build_dir: Path): "-m", "x2py", str(pyi_path), - "--wrap", "--native-objects", str(native_object), "--native-include-dir", @@ -204,7 +203,7 @@ def test_pyi_cli_requires_a_native_link_input(tmp_path: Path): ) assert result.returncode == 2 - assert "--wrap from .pyi requires --native-fortran-sources" in result.stderr + assert "A .pyi wrapper build requires --native-fortran-sources" in result.stderr @pytest.mark.skipif( @@ -222,7 +221,6 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): "-m", "x2py", str(PYI_FIXTURE), - "--wrap", "--native-fortran-sources", str(native_source), "--native-fortran-flags=-O2 -g0", @@ -283,7 +281,6 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): "x2py", "--build-manifest", str(manifest_path), - "--wrap", "--makefile", "--json", ], @@ -304,7 +301,6 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): "x2py", "--build-manifest", str(manifest_path), - "--wrap", "--json", ], capture_output=True, @@ -328,7 +324,6 @@ def test_pyi_cli_accepts_exactly_one_entry_contract(tmp_path: Path): "x2py", str(PYI_FIXTURE), str(other), - "--wrap", "--native-objects", str(tmp_path / "unused.o"), ], @@ -337,7 +332,7 @@ def test_pyi_cli_accepts_exactly_one_entry_contract(tmp_path: Path): ) assert result.returncode == 2 - assert "--wrap from .pyi accepts exactly one entry contract" in result.stderr + assert "A .pyi wrapper build accepts exactly one entry contract" in result.stderr def test_pyi_python_api_rejects_a_missing_native_artifact(tmp_path: Path): @@ -435,7 +430,6 @@ def test_pyi_cli_preserves_explicit_ordered_link_items(tmp_path: Path): "-m", "x2py", str(PYI_FIXTURE), - "--wrap", "--native-link-item", "arg:-Wl,--start-group", f"object:{native_object}", diff --git a/tests/wrapper/fortran/build_from_source/test_runtime_abi.py b/tests/wrapper/fortran/build_from_source/test_runtime_abi.py index 97ee81437..12707c435 100644 --- a/tests/wrapper/fortran/build_from_source/test_runtime_abi.py +++ b/tests/wrapper/fortran/build_from_source/test_runtime_abi.py @@ -47,7 +47,6 @@ def test_debug_and_optimized_wrapper_builds_preserve_runtime_abi(tmp_path: Path) "-m", "x2py", str(optimized_source), - "--wrap", "--makefile", "--out-dir", str(optimized_dir), diff --git a/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py b/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py index 922ddd94b..2a703a0cf 100644 --- a/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py +++ b/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py @@ -145,7 +145,6 @@ def _build_sources(sources: tuple[Path, ...], build_dir: Path) -> tuple[object, "-m", "x2py", *(str(source) for source in sources), - "--wrap", "--out-dir", str(build_dir), "--json", @@ -319,7 +318,6 @@ def test_makefile_mode_reproduces_multi_source_build(tmp_path: Path): "x2py", str(first), str(second), - "--wrap", "--makefile", "--out-dir", str(tmp_path), diff --git a/tools/wrapper_plan_staged_walkthrough.py b/tools/wrapper_plan_staged_walkthrough.py index d1c2a6186..d650d46a1 100644 --- a/tools/wrapper_plan_staged_walkthrough.py +++ b/tools/wrapper_plan_staged_walkthrough.py @@ -11,7 +11,7 @@ import numpy as np -from x2py.fortran_parser.parser import parse_fortran_project +from x2py.parsers.fortran.parser import parse_fortran_project from x2py.pipeline import build as pipeline from x2py.pipeline.preprocessing import PreprocessingConfig from x2py.semantics.fortran2ir import fortran_project_to_semantic_modules diff --git a/x2py/README.md b/x2py/README.md index aab5b9ac8..37394fe18 100644 --- a/x2py/README.md +++ b/x2py/README.md @@ -14,7 +14,7 @@ jumping directly into generated-code internals. | `probes/` | C ABI facts, Fortran kind/storage facts, and type mapping reports. | | `runtime/` | Python runtime objects used by generated extensions. | | `types/` | Semantic-to-Python ecosystem type mappings. | -| `c_parser/` and `fortran_parser/` | Native source frontends and parser models. | +| `parsers/` | Parser namespace containing the `c`, `fortran`, and semantic `.pyi` frontends. | | `semantics/` | Language-neutral semantic IR, policy completion, readiness, and `.pyi` conversion. | | `wrapper_codegen/` | Canonical wrapper plans, direct native bridge/binding generation, and source printers. | | `compiling/` | Native compiler objects, wrapper compilation, runtime support installation, and linking. | @@ -24,7 +24,8 @@ The package root contains the public entrypoint modules plus the shared `stage_values.py` record support. Supported library symbols are flattened through `x2py.__init__`; internal modules import their canonical owner. `x2py.contracts` remains a deliberate public submodule because its import path -is part of semantic `.pyi` syntax. +is part of semantic `.pyi` syntax. Parser-specific imports use the public +`x2py.parsers.c`, `x2py.parsers.fortran`, and `x2py.parsers.pyi` namespaces. ## Source Navigation Docs diff --git a/x2py/__init__.py b/x2py/__init__.py index 516cf6405..9edd4dda5 100644 --- a/x2py/__init__.py +++ b/x2py/__init__.py @@ -2,9 +2,9 @@ from importlib import import_module -from x2py.c_parser.models import CFile, CParseError, CProject -from x2py.c_parser.parser import parse_c_file, parse_c_project -from x2py.fortran_parser.models import ( +from x2py.parsers.c.models import CFile, CParseError, CProject +from x2py.parsers.c.parser import parse_c_file, parse_c_project +from x2py.parsers.fortran.models import ( FortranArgument, FortranBlockData, FortranDerivedType, @@ -17,8 +17,8 @@ FortranProject, FortranSubmodule, ) -from x2py.fortran_parser.parser import parse_fortran_file, parse_fortran_project -from x2py.pyi_parser import parse_pyi_file, parse_pyi_text +from x2py.parsers.fortran.parser import parse_fortran_file, parse_fortran_project +from x2py.parsers.pyi import parse_pyi_file, parse_pyi_text from x2py.semantics.fortran2ir import ( collect_semantic_compile_time_requirements, fortran_file_to_semantic_modules, diff --git a/x2py/cli.py b/x2py/cli.py index 8cd476eca..988a9f027 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -10,12 +10,12 @@ from dataclasses import asdict, dataclass, fields, is_dataclass, replace from pathlib import Path -from x2py.c_parser.cli import attach_preprocessing_recipe, expand_c_paths, format_c_report, parse_c_report -from x2py.c_parser.models import CParseError -from x2py.c_parser.parser import CParser -from x2py.fortran_parser.cli import _format_report -from x2py.fortran_parser.models import FortranParseError -from x2py.fortran_parser.parser import FortranParser +from x2py.parsers.c.cli import attach_preprocessing_recipe, expand_c_paths, format_c_report, parse_c_report +from x2py.parsers.c.models import CParseError +from x2py.parsers.c.parser import CParser +from x2py.parsers.fortran.cli import _format_report +from x2py.parsers.fortran.models import FortranParseError +from x2py.parsers.fortran.parser import FortranParser from x2py.semantics.c2ir import c_project_to_semantic_modules from x2py.semantics.fortran2ir import fortran_file_to_semantic_modules from x2py.pipeline.pyi import pyi_paths_to_semantic_modules @@ -72,9 +72,9 @@ " Build wrappers:\n" " python3 -m x2py path/to/file.f\n" " python3 -m x2py path/to/file.f90 --out my_extension\n" - " python3 -m x2py dependency.f90 api.f90 --wrap --makefile --out-dir build\n" - " python3 -m x2py contracts/__init__.pyi --wrap --out my_extension --native-objects native.o\n" - " python3 -m x2py --build-manifest build/x2py-build.json --wrap\n" + " python3 -m x2py dependency.f90 api.f90 --makefile --out-dir build\n" + " python3 -m x2py contracts/__init__.pyi --out my_extension --native-objects native.o\n" + " python3 -m x2py --build-manifest build/x2py-build.json\n" "\n" " Write stage output:\n" " python3 -m x2py path/to/file.f90 --parse --json --out report.json\n" @@ -1009,11 +1009,17 @@ def _wrapper_compile_options_used(args: argparse.Namespace) -> bool: def _stage_defaults_to_wrap(args: argparse.Namespace) -> bool: + """Return whether wrapper-specific input selects the default build stage.""" return bool( args.language == "fortran" and not _has_stage(args) - and not getattr(args, "makefile", False) - and any(Path(path).is_dir() or _path_is_fortran_source(path) for path in args.paths) + and ( + getattr(args, "build_manifest", None) is not None + or any( + Path(path).is_dir() or _path_is_fortran_source(path) or _path_is_pyi_contract(path) + for path in args.paths + ) + ) ) @@ -1051,11 +1057,11 @@ def _fortran_type_probe_options_used(args: argparse.Namespace) -> bool: def _validate_pyi_wrap_options(args: argparse.Namespace, parser: argparse.ArgumentParser) -> None: if any(Path(path).is_dir() for path in args.paths): - parser.error("--wrap from .pyi expects semantic contract files, not directories") + parser.error("A .pyi wrapper build expects semantic contract files, not directories") if any(not _path_is_pyi_contract(path) for path in args.paths): - parser.error("--wrap from .pyi cannot mix positional native sources; pass native artifacts with flags") + parser.error("A .pyi wrapper build cannot mix positional native sources; pass native artifacts with flags") if len(args.paths) != 1: - parser.error("--wrap from .pyi accepts exactly one entry contract") + parser.error("A .pyi wrapper build accepts exactly one entry contract") if not ( getattr(args, "native_fortran_sources", None) or getattr(args, "native_objects", None) @@ -1063,7 +1069,7 @@ def _validate_pyi_wrap_options(args: argparse.Namespace, parser: argparse.Argume or getattr(args, "native_link_items", None) ): parser.error( - "--wrap from .pyi requires --native-fortran-sources, --native-objects, " + "A .pyi wrapper build requires --native-fortran-sources, --native-objects, " "--native-library, or --native-link-item" ) @@ -1079,11 +1085,11 @@ def _validate_manifest_wrap_options(args: argparse.Namespace, parser: argparse.A def _validate_source_wrap_options(args: argparse.Namespace, parser: argparse.ArgumentParser) -> None: if not args.paths: - parser.error("--wrap expects at least one Fortran source file or a semantic .pyi contract") + parser.error("A wrapper build expects at least one Fortran source file or a semantic .pyi contract") if _native_link_options_used(args): parser.error("Native artifact link flags are only supported for .pyi wrapper builds") if any(Path(path).is_dir() for path in args.paths): - parser.error("--wrap expects Fortran source files, not directories") + parser.error("A wrapper build expects Fortran source files, not directories") def _validate_wrapper_out(args: argparse.Namespace, parser: argparse.ArgumentParser) -> None: @@ -1156,14 +1162,10 @@ def _validate_stage_selection(args: argparse.Namespace, parser: argparse.Argumen selected = _selected_stage_flags(args) if len(selected) > 1: parser.error(f"Choose exactly one stage flag; cannot combine {', '.join(selected)}") - if getattr(args, "makefile", False) and not getattr(args, "wrap", False): - parser.error("--makefile requires --wrap") def _validate_main_options(args: argparse.Namespace, parser: argparse.ArgumentParser) -> int | None: _validate_stage_selection(args, parser) - if getattr(args, "build_manifest", None) is not None and not getattr(args, "wrap", False): - parser.error("--build-manifest requires --wrap") if not args.paths and getattr(args, "build_manifest", None) is None: parser.error("Source input is required unless --build-manifest is used") diff --git a/x2py/parsers/README.md b/x2py/parsers/README.md new file mode 100644 index 000000000..2a0b4e7b0 --- /dev/null +++ b/x2py/parsers/README.md @@ -0,0 +1,16 @@ +# Parser Frontends + +This namespace groups syntax-level frontends without flattening their +language-specific models: + +- `x2py.parsers.c` parses C source and headers. +- `x2py.parsers.fortran` parses Fortran source and provides parser reports. +- `x2py.parsers.pyi` parses semantic `.pyi` contracts to Python AST. + +Cross-language semantic interpretation belongs to `x2py.semantics`, while +preprocessing and build orchestration belong to `x2py.pipeline`. Stable parser +convenience functions remain exported from the `x2py` package root. + +See `docs/developer/source-map.md`, `docs/developer/feature-to-code-map.md`, +`docs/developer/c-parser-reference.md`, `docs/developer/fortran-parser-reference.md`, and +`docs/user/reference/semantic-pyi-format.md` for maintained behavior. diff --git a/x2py/parsers/__init__.py b/x2py/parsers/__init__.py new file mode 100644 index 000000000..156b8456a --- /dev/null +++ b/x2py/parsers/__init__.py @@ -0,0 +1,3 @@ +"""Language parser frontends for C, Fortran, and semantic `.pyi` contracts.""" + +__all__ = ("c", "fortran", "pyi") diff --git a/x2py/c_parser/README.md b/x2py/parsers/c/README.md similarity index 88% rename from x2py/c_parser/README.md rename to x2py/parsers/c/README.md index c4e9edac7..7fc61b4fe 100644 --- a/x2py/c_parser/README.md +++ b/x2py/parsers/c/README.md @@ -4,6 +4,10 @@ This package owns C source facts for inspection workflows. It parses C inputs, preserves declarations and diagnostics, and feeds semantic conversion. It does not own runtime wrapping of user-supplied C libraries. +Its canonical import namespace is `x2py.parsers.c`. The stable convenience +functions `x2py.parse_c_file` and `x2py.parse_c_project` remain available from +the package root. + ## Entry Points | File | Owns | diff --git a/x2py/c_parser/__init__.py b/x2py/parsers/c/__init__.py similarity index 100% rename from x2py/c_parser/__init__.py rename to x2py/parsers/c/__init__.py diff --git a/x2py/c_parser/__main__.py b/x2py/parsers/c/__main__.py similarity index 100% rename from x2py/c_parser/__main__.py rename to x2py/parsers/c/__main__.py diff --git a/x2py/c_parser/cli.py b/x2py/parsers/c/cli.py similarity index 100% rename from x2py/c_parser/cli.py rename to x2py/parsers/c/cli.py diff --git a/x2py/c_parser/lexer.py b/x2py/parsers/c/lexer.py similarity index 100% rename from x2py/c_parser/lexer.py rename to x2py/parsers/c/lexer.py diff --git a/x2py/c_parser/models.py b/x2py/parsers/c/models.py similarity index 100% rename from x2py/c_parser/models.py rename to x2py/parsers/c/models.py diff --git a/x2py/c_parser/parser.py b/x2py/parsers/c/parser.py similarity index 100% rename from x2py/c_parser/parser.py rename to x2py/parsers/c/parser.py diff --git a/x2py/c_parser/preprocessor.py b/x2py/parsers/c/preprocessor.py similarity index 100% rename from x2py/c_parser/preprocessor.py rename to x2py/parsers/c/preprocessor.py diff --git a/x2py/c_parser/type_resolver.py b/x2py/parsers/c/type_resolver.py similarity index 100% rename from x2py/c_parser/type_resolver.py rename to x2py/parsers/c/type_resolver.py diff --git a/x2py/fortran_parser/README.md b/x2py/parsers/fortran/README.md similarity index 88% rename from x2py/fortran_parser/README.md rename to x2py/parsers/fortran/README.md index ef2b12d88..1f465c714 100644 --- a/x2py/fortran_parser/README.md +++ b/x2py/parsers/fortran/README.md @@ -4,6 +4,10 @@ This package owns Fortran source facts before semantic conversion. It preserves modules, procedures, declarations, derived types, visibility, and diagnostics needed by wrapper and inspection workflows. +Its canonical implementation namespace is `x2py.parsers.fortran`. Public +callers may also use the stable parser functions and models exported from the +`x2py` package root. + ## Entry Points | File | Owns | diff --git a/x2py/fortran_parser/__init__.py b/x2py/parsers/fortran/__init__.py similarity index 100% rename from x2py/fortran_parser/__init__.py rename to x2py/parsers/fortran/__init__.py diff --git a/x2py/fortran_parser/__main__.py b/x2py/parsers/fortran/__main__.py similarity index 100% rename from x2py/fortran_parser/__main__.py rename to x2py/parsers/fortran/__main__.py diff --git a/x2py/fortran_parser/cli.py b/x2py/parsers/fortran/cli.py similarity index 99% rename from x2py/fortran_parser/cli.py rename to x2py/parsers/fortran/cli.py index 302576015..ae77920e8 100644 --- a/x2py/fortran_parser/cli.py +++ b/x2py/parsers/fortran/cli.py @@ -260,7 +260,7 @@ def _format_report( def main() -> int: - """CLI entrypoint for `python -m x2py.fortran_parser` and `x2py.fortran_parser.cli`. + """CLI entrypoint for `python -m x2py.parsers.fortran` and `x2py.parsers.fortran.cli`. The CLI supports: - parsing one or more paths (files and/or directories) diff --git a/x2py/fortran_parser/lexer.py b/x2py/parsers/fortran/lexer.py similarity index 100% rename from x2py/fortran_parser/lexer.py rename to x2py/parsers/fortran/lexer.py diff --git a/x2py/fortran_parser/models.py b/x2py/parsers/fortran/models.py similarity index 100% rename from x2py/fortran_parser/models.py rename to x2py/parsers/fortran/models.py diff --git a/x2py/fortran_parser/parser.py b/x2py/parsers/fortran/parser.py similarity index 100% rename from x2py/fortran_parser/parser.py rename to x2py/parsers/fortran/parser.py diff --git a/x2py/fortran_parser/type_resolver.py b/x2py/parsers/fortran/type_resolver.py similarity index 100% rename from x2py/fortran_parser/type_resolver.py rename to x2py/parsers/fortran/type_resolver.py diff --git a/x2py/fortran_parser/utils.py b/x2py/parsers/fortran/utils.py similarity index 100% rename from x2py/fortran_parser/utils.py rename to x2py/parsers/fortran/utils.py diff --git a/x2py/pyi_parser/README.md b/x2py/parsers/pyi/README.md similarity index 68% rename from x2py/pyi_parser/README.md rename to x2py/parsers/pyi/README.md index fa04262c7..3d239e99a 100644 --- a/x2py/pyi_parser/README.md +++ b/x2py/parsers/pyi/README.md @@ -4,3 +4,6 @@ This package owns syntax-only parsing for semantic `.pyi` contracts. It reads inline text or files and returns Python AST modules. Semantic interpretation is handled by `x2py/semantics/pyi2ir.py`, and combined file/text loading is handled by `x2py/pipeline/pyi.py`. + +Its canonical parser namespace is `x2py.parsers.pyi`; `parse_pyi_text` and +`parse_pyi_file` also remain stable root-level `x2py` exports. diff --git a/x2py/pyi_parser/__init__.py b/x2py/parsers/pyi/__init__.py similarity index 100% rename from x2py/pyi_parser/__init__.py rename to x2py/parsers/pyi/__init__.py diff --git a/x2py/pyi_parser/parser.py b/x2py/parsers/pyi/parser.py similarity index 100% rename from x2py/pyi_parser/parser.py rename to x2py/parsers/pyi/parser.py diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index fdb703e8e..56e0172f2 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -15,7 +15,7 @@ from x2py.compiling.basic import CompileObj from x2py.compiling.compilers import Compiler, get_condaless_search_path from x2py.compiling.runtime_support import install_runtime_support -from x2py.fortran_parser.parser import parse_fortran_project +from x2py.parsers.fortran.parser import parse_fortran_project from x2py.probes.fortran_types import evaluate_fortran_type_facts, evaluate_fortran_type_requirements from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source from x2py.pipeline.wrapper_artifacts import GeneratedSourceFile, RenderedGeneratedWrapperArtifacts diff --git a/x2py/pipeline/pyi.py b/x2py/pipeline/pyi.py index 8be8de970..c73f94dae 100644 --- a/x2py/pipeline/pyi.py +++ b/x2py/pipeline/pyi.py @@ -6,7 +6,7 @@ from dataclasses import dataclass, field from pathlib import Path -from x2py.pyi_parser import parse_pyi_text +from x2py.parsers.pyi import parse_pyi_text from x2py.semantics.models import SemanticModule from x2py.semantics.pyi_metadata import PYI_LOADED_METADATA from x2py.semantics.pyi2ir import convert_pyi_to_ir, reconcile_external_type_refs diff --git a/x2py/probes/report.py b/x2py/probes/report.py index f8157a526..a54ac5862 100644 --- a/x2py/probes/report.py +++ b/x2py/probes/report.py @@ -6,7 +6,7 @@ from collections.abc import Sequence import platform -from x2py.c_parser.models import ( +from x2py.parsers.c.models import ( CBool, CChar, CDouble, @@ -27,7 +27,7 @@ CUnsignedLongLong, CUnsignedShort, ) -from x2py.fortran_parser.models import FortranVariable +from x2py.parsers.fortran.models import FortranVariable from x2py.semantics.c2ir import CToIRConverter from x2py.semantics.fortran2ir import FortranToIRConverter, fortran_type_storage_expression diff --git a/x2py/semantics/c2ir.py b/x2py/semantics/c2ir.py index 246161406..77d2d9ae1 100644 --- a/x2py/semantics/c2ir.py +++ b/x2py/semantics/c2ir.py @@ -5,7 +5,7 @@ from pathlib import Path from typing import Any -from x2py.c_parser.models import ( +from x2py.parsers.c.models import ( CArray, CAtomic, CBool, diff --git a/x2py/semantics/fortran2ir.py b/x2py/semantics/fortran2ir.py index 984028945..5a611b9d7 100644 --- a/x2py/semantics/fortran2ir.py +++ b/x2py/semantics/fortran2ir.py @@ -7,7 +7,7 @@ import re from pathlib import Path -from x2py.fortran_parser.models import ( +from x2py.parsers.fortran.models import ( FortranArgument, FortranBlockData, FortranDerivedType, diff --git a/x2py/semantics/pyi2ir.py b/x2py/semantics/pyi2ir.py index 7497ca725..d13ed0585 100644 --- a/x2py/semantics/pyi2ir.py +++ b/x2py/semantics/pyi2ir.py @@ -76,7 +76,7 @@ def convert_pyi_to_ir(tree: ast.Module, *, module_name: str = "", source: s """Convert a parsed semantic `.pyi` AST into semantic IR.""" if not isinstance(tree, ast.Module): - raise TypeError("convert_pyi_to_ir expects a Python ast.Module parsed by x2py.pyi_parser") + raise TypeError("convert_pyi_to_ir expects a Python ast.Module parsed by x2py.parsers.pyi") module = _PyiAstParser(module_name=module_name, source=source).parse(tree) _annotate_imported_external_type_refs(module) return module From 611bdf87e8b6e22feac7c7a15e5c4f8befae2a33 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 13:03:52 +0100 Subject: [PATCH 22/30] remove default ORDER_F --- THIRD_PARTY_NOTICES.md | 34 ++++++-- docs/developer/development-workflow.md | 13 ++- docs/developer/quality-assurance.md | 4 +- .../maintainer/design/wrapper-design-notes.md | 2 +- docs/old_docs/pyi_format.md | 10 +-- docs/old_docs/quality.md | 4 +- docs/old_docs/semantics.md | 56 ++++++------ docs/user/guide/arrays.md | 20 ++--- docs/user/guide/data-types.md | 7 +- .../guide/editing-semantic-pyi-contracts.md | 15 +++- docs/user/guide/wrapping-derived-types.md | 7 ++ docs/user/reference/generated-functions.md | 8 +- docs/user/reference/semantic-pyi-format.md | 30 ++++--- tests/cli/test_readiness_reports.py | 15 ++-- .../parsing/pyi/test_python_ast_contracts.py | 2 +- .../pyi_builds/test_contract_fixtures.py | 1 + .../assumed_shape_and_derived_args.pyi | 6 +- .../modern_math_physics.pyi | 6 +- .../pyi/test_calls_and_projections.py | 5 +- .../pyi/test_classes_and_overloads.py | 14 ++- .../conversion/pyi/test_types_and_values.py | 60 +++++++++---- .../semantics/readiness/test_pyi_readiness.py | 13 +++ .../farray_contracts_f90.pyi | 86 +++++++++---------- .../farray_results_f90/farray_results_f90.pyi | 34 ++++---- .../contracts/multid_arrays/multid_arrays.pyi | 36 ++++---- .../fclasses_f90.pyi | 4 +- .../contracts/fclasses_f90/fclasses_f90.pyi | 22 ++--- .../finheritance_f90/finheritance_f90.pyi | 8 +- .../fpointer_handles_f90.pyi | 6 +- .../fnative_call_examples_f90.pyi | 6 +- .../test_native_order_contracts.py | 6 +- .../fnative_call_examples_f90.pyi | 8 +- .../contracts/foutputs_f90/foutputs_f90.pyi | 6 +- .../fallocatable_views_f90.pyi | 6 +- .../foverloads_f90.pyi | 6 +- .../foverloads_f90.pyi | 6 +- .../contracts/fnaming_f90/fnaming_f90.pyi | 10 +-- .../foperators_f90/foperators_f90.pyi | 4 +- .../foverloads_f90/foverloads_f90.pyi | 6 +- .../contracts/lapack/__init__.pyi | 40 ++++----- .../test_pyi_printer_imports_and_packages.py | 4 +- .../printers/test_types_and_declarations.py | 34 ++++++-- .../test_phase6b_dense_array_shapes.py | 4 +- .../test_phase6c_strided_arrays.py | 4 +- .../test_phase6g_raw_array_addresses.py | 4 +- x2py/cli.py | 7 +- x2py/contracts/__init__.py | 8 +- x2py/pipeline/pyi.py | 49 +++++++++-- x2py/semantics/pyi2ir.py | 44 +++++++--- x2py/semantics/readiness.py | 7 +- x2py/wrapper_codegen/printers/pyi_printer.py | 60 +++++++++---- 51 files changed, 537 insertions(+), 320 deletions(-) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index a314ba66e..80e185446 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -1,12 +1,36 @@ # Third-Party Notices -Some code in this repository was adapted from the Pyccel project. +The compiler configuration/execution modules `x2py/compiling/basic.py`, +`x2py/compiling/compilers.py`, and `x2py/compiling/default_compilers.py`; the +native NumPy/Python runtime in `x2py/stdlib/x2py_runtime/`; and small naming +utilities in `x2py/utilities/strings.py`, `x2py/utilities/metaclasses.py`, and +`x2py/naming/policy.py` contain code adapted from the Pyccel project. The +current semantic and wrapper source printers are independent implementations +and are not covered by this attribution. + +Upstream project: + +The surviving-file comparison was refreshed against Pyccel commit +`f3361939fdd736474e510d90d502e7bee7157e12`. Pyccel is licensed under the MIT License: Copyright (c) 2017-2020, Pyccel Developers. -The MIT License permits use, copying, modification, merging, publishing, -distribution, sublicensing, and selling copies of the software, provided that -the copyright notice and permission notice are included in copies or substantial -portions of the software. +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/docs/developer/development-workflow.md b/docs/developer/development-workflow.md index 05b94f012..381ccdc84 100644 --- a/docs/developer/development-workflow.md +++ b/docs/developer/development-workflow.md @@ -288,11 +288,17 @@ Important implementation rules: - `Addr(T)` and `Addr(T)` are storage contracts, not just pretty syntax. - Array subscriptions such as `Float64[n]` are semantic array contracts. -- `Annotated[..., ORDER_F]` and `ORDER_ANY` are array storage metadata. +- `Annotated[..., ORDER_F]` and `ORDER_ANY` are non-default array storage + metadata. Plain multidimensional Fortran `.pyi` arrays use `ORDER_F`; do not + print or retain that default marker in a generated contract. `Allocatable[T[...]]` and `Pointer[T[...]]` are descriptor-handle wrappers around the array storage contract. Output and writeback behavior is represented by writable storage plus `Returns["name", T]` when a Python result is projected. + - `Final[T]` is the public constant spelling. Do not reintroduce `Constant` as user-facing `.pyi` syntax. - `@native_call` is projection metadata. Use it only when the Python-visible @@ -300,6 +306,11 @@ Important implementation rules: - Generated stubs should preserve behavior-changing native contracts while staying compact; exact source intent that does not change execution can stay in semantic IR instead of the printed `.pyi`. +- Use `SourceName("...")` only when a source identifier cannot be used as the + Python target. Do not infer source identifiers from normalized Python names. +- Omit `Polymorphic` only for the passed-object dummy of a type-bound procedure, + where the binding itself restores that native fact. Ordinary `class(T)` + arguments must retain it. When changing `.pyi` syntax: diff --git a/docs/developer/quality-assurance.md b/docs/developer/quality-assurance.md index b5c3eb8da..a875c41b9 100644 --- a/docs/developer/quality-assurance.md +++ b/docs/developer/quality-assurance.md @@ -153,7 +153,7 @@ removed as redundant maintenance overhead. code generation. @@ -329,7 +329,7 @@ The `Fuzz` workflow runs deeper discovery every Monday and by manual dispatch: | --- | --- | --- | --- | | 2026-05-31 | Initial stack integration | Added configuration, CI, documentation, and Hypothesis tests. | Continue staged strictness rollout. | | 2026-05-31 | Bandit | Reviewed low-severity findings and confirmed no medium- or high-severity findings. | Re-review when command trust boundaries change. | -| 2026-05-31 | Hypothesis code generation | Added generated native-name escaping, stable synthetic-import ordering, and semantic-IR-to-Pyi parse-back invariants; fixed quoted `Name(...)` emission. | Keep storing minimized failures. | +| 2026-05-31 | Hypothesis code generation | Added generated native-name escaping, stable synthetic-import ordering, and semantic-IR-to-Pyi parse-back invariants; fixed quoted `SourceName(...)` emission. | Keep storing minimized failures. | | 2026-06-01 | Ruff formatting rollout | Formatted the historical Python tree and changed CI to `ruff format --check .`. | Continue complexity-policy ratchets. | | 2026-06-01 | Radon and Ruff complexity policy | Added `tools/check_radon_policy.py`, made the staged Radon policy blocking in CI, and lowered Ruff McCabe from `50` to `45`. | Continue hotspot refactors and later threshold ratchets toward `20`. | | 2026-06-02 | Historical mutation-derived tests | Added direct Fortran parser contracts and fixed the directory namespace encoding bug. | Keep the tests as normal regression coverage. | diff --git a/docs/maintainer/design/wrapper-design-notes.md b/docs/maintainer/design/wrapper-design-notes.md index 24bc0ccd5..548040eff 100644 --- a/docs/maintainer/design/wrapper-design-notes.md +++ b/docs/maintainer/design/wrapper-design-notes.md @@ -60,7 +60,7 @@ X2PY_C_DOCS_END --> diff --git a/docs/old_docs/pyi_format.md b/docs/old_docs/pyi_format.md index 191447c1b..1fd35bacf 100644 --- a/docs/old_docs/pyi_format.md +++ b/docs/old_docs/pyi_format.md @@ -451,7 +451,7 @@ Generated canonical metadata: | `Allocatable` | Fortran allocatable array storage | | `Pointer` | Fortran pointer array storage | | `PointerAssociation("runtime")` | pointer association is a runtime state rather than a declaration-time constant | -| `Name("native-name")` | source name cannot be represented directly as the Python target name | +| `SourceName("native-name")` | source name cannot be represented directly as the Python target name | | `FortranCharacterLength("n")` | Fortran character storage length for `String` contracts | | `FortranAllocatable` | Fortran scalar character storage is allocatable | | `Aliased` | native storage may be exposed across the Python boundary as an alias | @@ -464,7 +464,7 @@ Loaded compatibility metadata: | Metadata | Meaning | | --- | --- | -| `ORDER_C` | explicit C-oriented storage; this is also the default for plain multidimensional arrays | +| `ORDER_C` | explicit C-oriented storage in a Fortran contract | | `Contiguous` | source provenance says the array is contiguous | | `ArrayCategory("...")` | source array category provenance | | `SourceDims(...)` | source declaration dimensions | @@ -976,12 +976,12 @@ is a user contract applied to a declaration that was otherwise available to the wrapper, so the declaration remains printed and loadable as wrapper input. Names that are not valid Python identifiers are represented with `var[...]` for -data declarations, or with `Annotated[..., Name("native-name")]` for callable +data declarations, or with `Annotated[..., SourceName("native-name")]` for callable arguments: ```python var["class"]: Int32 -def f(class_: Annotated[Int32, Name("class")]) -> None: ... +def f(class_: Annotated[Int32, SourceName("class")]) -> None: ... ``` ## Projection Metadata @@ -1022,7 +1022,7 @@ Generated `.pyi` currently covers these exact-contract areas: | C primitive scalars | compiler-probed semantic dtype names when a target report is supplied | | Functions/subroutines | exact native argument order and direct return type | | Fortran scalar storage | `T`, `T[()]`, `Addr(Arg(...))`, `Returns[...]` | -| Arrays | shaped storage with extents, strided axes, `ORDER_F` for multidimensional Fortran arrays | +| Arrays | shaped storage with extents and strided axes; multidimensional order defaults from the selected native language | | Allocatable borrowed views | derived-type fields and target-backed module arrays, with `None` for unallocated storage | | Constants | `Final[T]` module variables | | C and Fortran enums | module-level `Final[...]` integer constants | diff --git a/docs/old_docs/quality.md b/docs/old_docs/quality.md index 3278f79a3..6f2ef0ebf 100644 --- a/docs/old_docs/quality.md +++ b/docs/old_docs/quality.md @@ -127,7 +127,7 @@ removed as redundant maintenance overhead. **Role:** generates edge cases for parsers, AST transforms, semantic IR, and code generation. -**Bugs found:** generated code-generation cases exposed quoted `Name(...)` +**Bugs found:** generated code-generation cases exposed quoted `SourceName(...)` emission. Generated preprocessing inputs also aligned raw Fortran and C macro handling around compiler-required errors. @@ -285,7 +285,7 @@ The `Fuzz` workflow runs deeper discovery every Monday and by manual dispatch: | --- | --- | --- | --- | | 2026-05-31 | Initial stack integration | Added configuration, CI, documentation, and Hypothesis tests. | Continue staged strictness rollout. | | 2026-05-31 | Bandit | Reviewed low-severity findings and confirmed no medium- or high-severity findings. | Re-review when command trust boundaries change. | -| 2026-05-31 | Hypothesis code generation | Added generated native-name escaping, stable synthetic-import ordering, and semantic-IR-to-Pyi parse-back invariants; fixed quoted `Name(...)` emission. | Keep storing minimized failures. | +| 2026-05-31 | Hypothesis code generation | Added generated native-name escaping, stable synthetic-import ordering, and semantic-IR-to-Pyi parse-back invariants; fixed quoted `SourceName(...)` emission. | Keep storing minimized failures. | | 2026-06-01 | Ruff formatting rollout | Formatted the historical Python tree and changed CI to `ruff format --check .`. | Continue complexity-policy ratchets. | | 2026-06-01 | Radon and Ruff complexity policy | Added `tools/check_radon_policy.py`, made the staged Radon policy blocking in CI, and lowered Ruff McCabe from `50` to `45`. | Continue hotspot refactors and later threshold ratchets toward `20`. | | 2026-06-02 | Historical mutation-derived tests | Added direct Fortran parser contracts and fixed the directory namespace encoding bug. | Keep the tests as normal regression coverage. | diff --git a/docs/old_docs/semantics.md b/docs/old_docs/semantics.md index 0d498ba5f..ae9c6a046 100644 --- a/docs/old_docs/semantics.md +++ b/docs/old_docs/semantics.md @@ -399,11 +399,11 @@ use `Annotated[T[...], Constraint, ...]`. replacement projection, where the argument remains visible and a `Returns["name", T]` item carries the post-call value. -Plain multidimensional array notation is C-oriented (`ORDER_C`) by default. -Under the current Fortran generation policy, every multidimensional Fortran -array contract emits `ORDER_F`, including stride-aware assumed-shape arrays. -Rank-one storage has no C-versus-Fortran order distinction, so no order marker -is emitted for vectors. +Plain multidimensional array notation follows the selected native language: +Fortran contracts default to `ORDER_F` and C contracts to `ORDER_C`. +Generated contracts omit that default order; an order annotation records only +an intentional non-default layout. Rank-one storage has no C-versus-Fortran +order distinction, so no order marker is emitted for vectors. `ArrayCategory(...)`, `SourceDims(...)`, `LowerBounds(...)` and `Contiguous` are not part of newly generated canonical array annotations. They described @@ -1032,20 +1032,17 @@ represents pointer-backed array storage; do not additionally wrap it in `Addr(...)`. For multidimensional storage, order is orthogonal to rank, dimensions and -stride capability. `Annotated[Float64[:, :], ORDER_F]` denotes a rank-two -dense Fortran-contiguous array, while -`Annotated[Float64[::, ::], ORDER_F]` denotes a rank-two -Fortran-oriented strided array. Bare `Float64[::, ::]` retains -the default `ORDER_C` orientation, and -`Annotated[Float64[::, ::], ORDER_ANY]` imposes no C/F -orientation restriction. `Annotated[Float64[...][1:4], ORDER_F]` expresses -the corresponding Fortran-oriented rank-polymorphic contract. These spellings -define the semantic format; they are explicit because `ORDER_F` and -`ORDER_ANY` are non-default in a C-origin stub. Accepting either in a -runnable C Phase 1 wrapper requires the corresponding native routine to -accept that storage layout directly. For a rank-one array, `ORDER_C` and -`ORDER_F` do not distinguish storage, contiguous or strided, so no order -constraint is written. +stride capability. In a C contract, `Annotated[Float64[:, :], ORDER_F]` +denotes a rank-two dense Fortran-contiguous array, while +`Annotated[Float64[::, ::], ORDER_F]` denotes a rank-two Fortran-oriented +strided array. Bare `Float64[::, ::]` uses the selected native language's +default orientation, and `Annotated[Float64[::, ::], ORDER_ANY]` imposes no +C/F orientation restriction. `Annotated[Float64[...][1:4], ORDER_F]` +expresses the corresponding Fortran-oriented rank-polymorphic contract. +These spellings define the semantic format; `ORDER_F`, `ORDER_C`, and +`ORDER_ANY` are written only when they differ from the selected language's +default. For a rank-one array, `ORDER_C` and `ORDER_F` do not distinguish +storage, contiguous or strided, so no order constraint is written. For a multidimensional strided annotation, `ORDER_F` is orientation metadata, not a requirement that NumPy report `F_CONTIGUOUS`; non-unit strides remain part of the contract. @@ -1293,16 +1290,17 @@ later Pythonic adaptations. #### 6.4 Contiguity Without an explicit layout or stride form, array annotations such as `T[:]`, -`T[:, :]`, `T[n]`, and `T[...]` require C-contiguous numeric storage; a -generated C stub does not repeat this as `ORDER_C`. Explicit non-default -forms such as `Annotated[T[:, :], ORDER_F]`, -`Annotated[T[::, ::], ORDER_F]`, or -`Annotated[T[::, ::], ORDER_ANY]` are exact interfaces when -the native routine accepts that layout and all required metadata remains -visible in the signature. A bare multidimensional stride form such as -`T[:, ::]` is also exact when native metadata is visible, but retains -the implicit `ORDER_C` orientation. Automatic packing, copy-back, or -derivation of native metadata is a later Pythonic transformation. +`T[:, :]`, `T[n]`, and `T[...]` require the selected native language's +default numeric storage order: Fortran-contiguous for Fortran and +C-contiguous for C. Generated stubs do not repeat that default. Explicit +non-default forms such as `Annotated[T[:, :], ORDER_F]` in a C contract, +`Annotated[T[:, :], ORDER_C]` in a Fortran contract, or +`Annotated[T[::, ::], ORDER_ANY]` are exact interfaces when the native +routine accepts that layout and all required metadata remains visible in the +signature. A bare multidimensional stride form such as `T[:, ::]` is also +exact when native metadata is visible, but retains the language-derived +default orientation. Automatic packing, copy-back, or derivation of native +metadata is a later Pythonic transformation. For rank one, `T[:]` and `T[n]` are also the canonical Fortran-contiguous spelling; write `T[::]` when contiguity is not required. diff --git a/docs/user/guide/arrays.md b/docs/user/guide/arrays.md index 23f3c4b6b..2e31a941f 100644 --- a/docs/user/guide/arrays.md +++ b/docs/user/guide/arrays.md @@ -48,13 +48,13 @@ end module array_ops Inspecting `arrays.f90` prints these array contracts: ```python -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, ORDER_F, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, native_call @native_call([Addr(Arg(0)), Addr(Arg(1)), Arg(2)]) def scale_matrix( rows: Int32, columns: Int32, - values: Annotated[Float64[rows, columns], ORDER_F] + values: Float64[rows, columns] ) -> None: ... @native_call([Addr(Arg(0)), Arg(1)]) @@ -134,14 +134,14 @@ runs. Use `numpy.asfortranarray` or `order="F"` for a multidimensional contract that requires Fortran orientation, as shown by `matrix` in the complete example. -Layout annotations describe the exact storage accepted by the native contract; -they do not request an automatic conversion. `ORDER_F` passes a -Fortran-contiguous array with its logical axes unchanged. `ORDER_C` passes the -same C-contiguous data address without copying and constructs the Fortran -bridge view with reversed axes. For example, a C-order Python shape `(2, 3)` -is a Fortran bridge shape `(3, 2)` over the same six elements. Use `ORDER_C` -only when the native operation intentionally accepts that transposed storage -view. +Layout annotations describe a deliberate non-default storage representation; +they do not request an automatic conversion. Plain multidimensional +Fortran-facing arrays already pass Fortran-contiguous storage with their logical +axes unchanged. `ORDER_C` passes the same C-contiguous data address without +copying and constructs the Fortran bridge view with reversed axes. For example, +a C-order Python shape `(2, 3)` is a Fortran bridge shape `(3, 2)` over the +same six elements. Use `ORDER_C` only when the native operation intentionally +accepts that transposed storage view. Add `COPY_F` when Python should accept C-contiguous storage but native Fortran must observe the same logical axes in Fortran order: diff --git a/docs/user/guide/data-types.md b/docs/user/guide/data-types.md index f12ac6834..a29b191e4 100644 --- a/docs/user/guide/data-types.md +++ b/docs/user/guide/data-types.md @@ -170,9 +170,10 @@ Array annotations combine an element dtype with rank and shape: Plain `:` records a dense axis. `::` records an axis where the contract allows runtime strides; x2py reads that exact slice spelling from the `.pyi` source -before Python AST normalization. Multidimensional dense arrays are validated -against their documented orientation, such as `ORDER_F` for Fortran-oriented -storage. +before Python AST normalization. Multidimensional dense arrays in a +Fortran-facing contract use Fortran orientation by default; generated contracts +omit `ORDER_F`. Explicit order metadata appears only when a contract +deliberately requests a non-default representation. The wrapper validates exact dtype, native byte order, rank, known extents, alignment, layout, and writeability before entering native code. It does not diff --git a/docs/user/guide/editing-semantic-pyi-contracts.md b/docs/user/guide/editing-semantic-pyi-contracts.md index ce4a8a186..61157fafe 100644 --- a/docs/user/guide/editing-semantic-pyi-contracts.md +++ b/docs/user/guide/editing-semantic-pyi-contracts.md @@ -404,6 +404,11 @@ and returns a different NumPy array. The original remains unchanged. The compiled evidence is [`test_policy_dispatch_contracts.py`](../../../tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py). +Mutability is a property of this argument boundary, not of `Float64`, +`String[n]`, or another datatype in isolation. The same datatype may be an +input value, caller-owned writable storage, or a replacement-only value in +different procedures, so datatype spelling cannot replace `Immutable`. + `Immutable` plus `Transfer("borrowed_view")` on a writable value is contradictory: one requests replacement-only semantics and the other requests a writable shared view. The contract fails instead of selecting one silently. @@ -414,20 +419,24 @@ The annotation is runtime policy, not merely an IDE hint. Supported edits can tighten or broaden validation without changing the native ABI: ```python -from x2py.contracts import Annotated, Float64, ORDER_F +from x2py.contracts import Float64 def solve( - matrix: Annotated[Float64[3, 3], ORDER_F], + matrix: Float64[3, 3], rhs: Float64[3], ) -> Float64[3]: ... ``` +Plain multidimensional arrays in a Fortran semantic `.pyi` use `ORDER_F` by +default. Generated contracts omit that default order. Use explicit layout +metadata only to request a non-default Python storage representation. + The wrapper validates exact dtype, rank, shape, layout, writeability, byte order, alignment, and zero-sized-array rules required by the selected backend path. Examples of supported changes include: - `Float64[:, :]` to `Float64[3, 3]` to require one shape; -- `ORDER_F` to `ORDER_ANY` when the native path is implemented for either +- `ORDER_ANY` when the native path is implemented for either contiguous orientation; - `T | None` or a default `= ...` for a genuinely optional native argument; - `Allocatable`, `Pointer`, `Aliased`, or `PointerPolicy(...)` when those diff --git a/docs/user/guide/wrapping-derived-types.md b/docs/user/guide/wrapping-derived-types.md index 288b9e277..acc337669 100644 --- a/docs/user/guide/wrapping-derived-types.md +++ b/docs/user/guide/wrapping-derived-types.md @@ -269,6 +269,13 @@ deallocation. Native termination from a finalizer terminates the process. Supported extension types form a matching Python inheritance hierarchy. A scalar polymorphic input over a known hierarchy dispatches descendant-first. +Ordinary native `class(T)` arguments retain `Annotated[T, Polymorphic]` in the +semantic `.pyi` because that source fact selects the accepted dynamic-type +dispatch. The passed-object dummy of a type-bound procedure is different: its +class binding already proves that it is polymorphic, so generated contracts use +the plain declared type for that one argument and restore the fact when loading +the binding. + Polymorphic results, mutable polymorphic arguments, arrays, allocatable or pointer polymorphic scalars, `class(*)`, abstract instantiation, and deferred bindings are blocked. diff --git a/docs/user/reference/generated-functions.md b/docs/user/reference/generated-functions.md index 17b50b6ed..e2935014b 100644 --- a/docs/user/reference/generated-functions.md +++ b/docs/user/reference/generated-functions.md @@ -45,7 +45,7 @@ contract uses `@native_call(...)` and `Returns[...]` to preserve the native call shape: ```python -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, ORDER_F, Returns, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call @native_call([Addr(Arg(0)), Arg(1)]) def fill_vector( @@ -57,9 +57,9 @@ def fill_vector( def shift_matrix( n: Int32, m: Int32, - values: Annotated[Float64[n, m], ORDER_F], - out: Annotated[Float64[n, m], ORDER_F] -) -> Returns["out", Annotated[Float64[n, m], ORDER_F]]: ... + values: Float64[n, m], + out: Float64[n, m] +) -> Returns["out", Float64[n, m]]: ... ``` `Returns["name", Type]` names a projected Python return. `tuple[...]` is used diff --git a/docs/user/reference/semantic-pyi-format.md b/docs/user/reference/semantic-pyi-format.md index 2e3c0cc8e..396d4e4d9 100644 --- a/docs/user/reference/semantic-pyi-format.md +++ b/docs/user/reference/semantic-pyi-format.md @@ -1083,17 +1083,22 @@ Use local constants or generated `Final[...]` names for shape symbols. `Annotated[...]` carries storage metadata and semantic constraints. It does not carry source-language argument direction or per-call value/reference -selection. Native call transport belongs to `@native_call`: a wrapped derived +selection. The Fortran semantic pipeline supplies `ORDER_F` as the default +multidimensional layout. Generated contracts omit that default. Write explicit +layout metadata only when the Python-visible storage deliberately differs from +that Fortran representation, such as a row-major input accepted by a Fortran +wrapper. +Native call transport belongs to `@native_call`: a wrapped derived object uses its normal reference handoff with `Arg(i)` and exact typed value handoff with `Value(Arg(i))`. The Python API accepts the same opaque wrapper object in both cases; the generated Fortran bridge performs the typed call, and the binding never exposes or guesses aggregate layout. ```python -from x2py.contracts import Annotated, COPY_F, Float64, ORDER_C, ORDER_F +from x2py.contracts import Annotated, COPY_F, Float64, ORDER_C def fill( - a: Annotated[Float64[:, :], ORDER_F], + a: Float64[:, :], c_input: Annotated[Float64[:, :], ORDER_C, COPY_F], out: Float64[()], ) -> None: ... @@ -1103,12 +1108,12 @@ Generated canonical metadata: | Metadata | Meaning | | --- | --- | -| `ORDER_F` | multidimensional Fortran-oriented storage | | `COPY_F` | accept the declared C-contiguous Python layout, create an F-contiguous temporary with the same logical axes, and copy back after visible native mutation | | `PointerAssociation("runtime")` | pointer association is a runtime state rather than a declaration-time constant | -| `Name("native-name")` | source name cannot be represented directly as the Python target name | +| `SourceName("native-name")` | source name cannot be represented directly as the Python target name | | `Aliased` | native storage may be exposed across the Python boundary as an alias | -| `Immutable` | Python-visible value must not be mutated in place; writable native calls require a completed copy-in/copy-out replacement policy or an explicit call-local discarded-mutation policy | +| `Immutable` | Python-visible value must not be mutated in place; this is a use-site boundary policy rather than an intrinsic datatype property, and writable native calls require a completed copy-in/copy-out replacement policy or an explicit call-local discarded-mutation policy | +| `Polymorphic` | an ordinary derived argument is a native polymorphic `class(T)` dummy; the passed-object dummy of a type-bound procedure omits this metadata because the binding already proves it | | `Ownership("python" | "native" | "wrapper" | "caller" | "temporary" | "unknown")` | explicit owner override for the wrapper ownership policy | | `Transfer("copy_return" | "snapshot_copy" | "borrowed_view" | "call_local" | "in_place" | "by_value" | "wrapper_instance" | "blocked")` | explicit boundary transfer override for the wrapper ownership policy | | `Destruction("python_refcount" | "wrapper_dealloc" | "native_owner" | "caller" | "call_local" | "none" | "blocked")` | explicit destruction override for the wrapper ownership policy | @@ -1127,7 +1132,10 @@ Loaded compatibility metadata: | `FortranAllocatable` | older scalar character allocatable metadata; generated contracts use `Allocatable[String]` | Without `COPY_F`, `ORDER_C` is zero-copy and native Fortran observes the @@ -2191,14 +2199,14 @@ is a user contract applied to a declaration that was otherwise available to the wrapper, so the declaration remains printed and loadable as wrapper input. Names that are not valid Python identifiers are represented with `var[...]` for -data declarations, or with `Annotated[..., Name("native-name")]` for callable +data declarations, or with `Annotated[..., SourceName("native-name")]` for callable arguments: ```python -from x2py.contracts import Annotated, Int32, Name +from x2py.contracts import Annotated, Int32, SourceName var["class"]: Int32 -def f(class_: Annotated[Int32, Name("class")]) -> None: ... +def f(class_: Annotated[Int32, SourceName("class")]) -> None: ... ``` ## Projection Metadata @@ -2268,7 +2276,7 @@ Generated `.pyi` currently covers these exact-contract areas: | Hidden Fortran outputs | Python returns plus generated `@native_call` in native argument order | | Scalar address inputs | Python-visible `T` plus `Addr(Arg(...))` native-call projection | | Writable scalar storage | `T[()]`, or visible `T` plus projected replacement `Returns["name", T]` | -| Arrays | shaped storage with extents, strided axes, `ORDER_F` for multidimensional Fortran arrays | +| Arrays | shaped storage with extents and strided axes; multidimensional order defaults from the selected native language | | Module variables | direct module-level annotations; native accessors remain internal | | Native array descriptor handles | `Allocatable[T[...]]` and `Pointer[T[...]]` handles for module variables, supported fields, and descriptor arguments; owned allocatable result handles; unallocated or unassociated state remains inside the handle | | Constants | `Final[T]` module variables | diff --git a/tests/cli/test_readiness_reports.py b/tests/cli/test_readiness_reports.py index d8c0894b6..6c1cb47a8 100644 --- a/tests/cli/test_readiness_reports.py +++ b/tests/cli/test_readiness_reports.py @@ -484,8 +484,9 @@ def assess(modules, *, source): assert source == str(path) return readiness - def pyi(paths): + def pyi(paths, *, native_language): assert paths == ["api"] + assert native_language == "c" return pyi_report monkeypatch.setattr(x2py_cli, "expand_c_paths", expand) @@ -556,8 +557,9 @@ def assess(modules, *, source): assert source == str(path) return readiness - def pyi(paths): + def pyi(paths, *, native_language): assert paths == ["api"] + assert native_language == "fortran" return pyi_report expected_compile_time_values = compile_time_values @@ -611,9 +613,10 @@ def expand(paths): calls.append(("expand", paths)) return [stub] - def load(paths): + def load(paths, *, native_language): assert paths == [str(package), str(stub)] - calls.append(("load", paths)) + assert native_language == "c" + calls.append(("load", paths, native_language)) return [module] def serialize(received): @@ -633,7 +636,7 @@ def assess(modules, *, source, require_native_contract): monkeypatch.setattr(x2py_cli, "asdict", serialize) monkeypatch.setattr(x2py_cli, "assess_semantic_wrap_readiness", assess) - assert x2py_cli._pyi_readiness_report([str(package), str(stub), str(ignored)]) == { + assert x2py_cli._pyi_readiness_report([str(package), str(stub), str(ignored)], native_language="c") == { str(stub): { "source_kind": "pyi", "semantic_modules": [{"name": "api"}], @@ -642,7 +645,7 @@ def assess(modules, *, source, require_native_contract): } assert calls == [ ("expand", [str(package), str(stub), str(ignored)]), - ("load", [str(package), str(stub)]), + ("load", [str(package), str(stub)], "c"), ("asdict", module), ("assess", [module], str(stub)), ] diff --git a/tests/parsing/pyi/test_python_ast_contracts.py b/tests/parsing/pyi/test_python_ast_contracts.py index e1d0bc12e..a9afd28ac 100644 --- a/tests/parsing/pyi/test_python_ast_contracts.py +++ b/tests/parsing/pyi/test_python_ast_contracts.py @@ -65,7 +65,7 @@ def test_pyi_parser_preserves_generic_constraints_as_annotation_metadata(): module = parse_pyi_text( """ value: Annotated[Int32, Bounded(1, 8), Finite] -alias: Annotated[Int32, Name("native_alias"), Finite] +alias: Annotated[Int32, SourceName("native_alias"), Finite] """, module_name="edited", ) diff --git a/tests/pipeline/pyi_builds/test_contract_fixtures.py b/tests/pipeline/pyi_builds/test_contract_fixtures.py index 2bf32fd12..cf58fe9f6 100644 --- a/tests/pipeline/pyi_builds/test_contract_fixtures.py +++ b/tests/pipeline/pyi_builds/test_contract_fixtures.py @@ -280,6 +280,7 @@ def test_c_pyi_fixtures_round_trip_through_semantic_ir(fixture: Path): expected, module_name=fixture.stem, filename=str(fixture), + native_language="c", ) assert module.name == fixture.stem diff --git a/tests/pyi/fixtures/general/assumed_shape_and_derived_args/assumed_shape_and_derived_args.pyi b/tests/pyi/fixtures/general/assumed_shape_and_derived_args/assumed_shape_and_derived_args.pyi index 7b492c107..36c351d3e 100644 --- a/tests/pyi/fixtures/general/assumed_shape_and_derived_args/assumed_shape_and_derived_args.pyi +++ b/tests/pyi/fixtures/general/assumed_shape_and_derived_args/assumed_shape_and_derived_args.pyi @@ -1,13 +1,13 @@ -from x2py.contracts import Annotated, Float32, Int32, ORDER_F, external +from x2py.contracts import Float32, Int32, external @external def fill_grid( - x: Annotated[Int32[::, ::], ORDER_F] + x: Int32[::, ::] ) -> None: ... @external def update_plane( - x: Annotated[Float32[::, ::], ORDER_F] + x: Float32[::, ::] ) -> None: ... @external diff --git a/tests/pyi/fixtures/general/modern_pyi_example/modern_math_physics.pyi b/tests/pyi/fixtures/general/modern_pyi_example/modern_math_physics.pyi index 728824356..be33224e7 100644 --- a/tests/pyi/fixtures/general/modern_pyi_example/modern_math_physics.pyi +++ b/tests/pyi/fixtures/general/modern_pyi_example/modern_math_physics.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, ORDER_F, Return, Returns, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Return, Returns, native_call class particle: def __init__( @@ -48,8 +48,8 @@ def dot3( ) -> Float64: ... def fill_identity3( - a: Annotated[Float64[3, 3], ORDER_F] -) -> Returns["a", Annotated[Float64[3, 3], ORDER_F]]: ... + a: Float64[3, 3] +) -> Returns["a", Float64[3, 3]]: ... def normalize_particle( p: particle diff --git a/tests/semantics/conversion/pyi/test_calls_and_projections.py b/tests/semantics/conversion/pyi/test_calls_and_projections.py index 9f941d508..6c7b141c3 100644 --- a/tests/semantics/conversion/pyi/test_calls_and_projections.py +++ b/tests/semantics/conversion/pyi/test_calls_and_projections.py @@ -741,7 +741,10 @@ class vector: "Annotated[...] for constraints or array metadata", ), ("foo.bar: Int32\n", "Unsupported annotation target: 'foo.bar'"), - ("value: Annotated[Int32, Name('x', 'y')]\n", "Name metadata expects one argument: \"Name('x', 'y')\""), + ( + "value: Annotated[Int32, SourceName('x', 'y')]\n", + "SourceName metadata expects one argument: \"SourceName('x', 'y')\"", + ), ("def f(x: Int32): ...\n", "Unsupported function header: 'def f(x: Int32):'"), ("def f(\n x: Int32,\n): ...\n", "Unterminated callable starting at line 2"), ("def f(*x: Int32) -> None: ...\n", "Unsupported function header: 'def f(*x: Int32) -> None:'"), diff --git a/tests/semantics/conversion/pyi/test_classes_and_overloads.py b/tests/semantics/conversion/pyi/test_classes_and_overloads.py index 996fcf006..f4e414d34 100644 --- a/tests/semantics/conversion/pyi/test_classes_and_overloads.py +++ b/tests/semantics/conversion/pyi/test_classes_and_overloads.py @@ -466,15 +466,17 @@ def shift( ) -> None: ... def scale( - self: Annotated[vector, Polymorphic], + self: vector, factor: Addr(Float64) ) -> None: ... def shift_vector( dx: Addr(Float64), - owner: Annotated[vector, Polymorphic], + owner: vector, dy: Addr(Float64) ) -> None: ... + +def inspect(value: Annotated[vector, Polymorphic]) -> None: ... """, module_name="edited", ) @@ -483,9 +485,17 @@ def shift_vector( assert functions["scale"].metadata["fortran_type_bound_target"] is True assert functions["scale"].metadata["fortran_passed_object_name"] == "self" assert functions["scale"].metadata["fortran_passed_object_position"] == 0 + assert functions["scale"].arguments[0].semantic_type.metadata["fortran_polymorphic"] is True assert functions["shift_vector"].metadata["fortran_type_bound_target"] is True assert functions["shift_vector"].metadata["fortran_passed_object_name"] == "owner" assert functions["shift_vector"].metadata["fortran_passed_object_position"] == 1 + assert functions["shift_vector"].arguments[1].semantic_type.metadata["fortran_polymorphic"] is True + + emitted = emit_module(from_pyi) + assert "self: vector" in emitted + assert "owner: vector" in emitted + assert "value: Annotated[vector, Polymorphic]" in emitted + assert parse_pyi_text(emitted, module_name="edited") == from_pyi def test_pyi_keyword_normalized_type_bound_method_keeps_native_binding_name(): diff --git a/tests/semantics/conversion/pyi/test_types_and_values.py b/tests/semantics/conversion/pyi/test_types_and_values.py index 8e90d4c8a..0e76bb2e4 100644 --- a/tests/semantics/conversion/pyi/test_types_and_values.py +++ b/tests/semantics/conversion/pyi/test_types_and_values.py @@ -342,7 +342,7 @@ def test_convert_pyi_to_ir_forwards_filename_to_syntax_errors(): def test_convert_pyi_to_ir_accepts_aliased_contract_wrapper_names(): module = pyi_text_to_semantic_module( """ -from x2py.contracts import Annotated as Metadata, Float64 as F64, Name as NativeName +from x2py.contracts import Annotated as Metadata, Float64 as F64, SourceName as NativeName from x2py.contracts import Returns as Gives alias: Metadata[F64[1:n], NativeName("native_alias")] @@ -477,7 +477,7 @@ def test_convert_pyi_to_ir_preserves_explicit_array_source_dimensions(): module = parse_pyi_text( """ def apply( - A: Annotated[Float64[LDA, N], ORDER_F], + A: Float64[LDA, N], work: Float64[::], scratch: Float64[:] ) -> None: ... @@ -516,27 +516,49 @@ def test_convert_pyi_to_ir_accepts_explicit_strided_marker_for_edited_contracts( assert [array.contiguous for array in arrays] == [False, False, False, False] -def test_convert_pyi_to_ir_accepts_c_and_fortran_order_constraints(): - module = parse_pyi_text( +def test_convert_pyi_to_ir_uses_the_selected_native_language_array_default(): + fortran = parse_pyi_text( """ def consume( a: Float64[:, :], - b: Annotated[Float64[:, :], ORDER_F], c: Annotated[Float64[:, :], ORDER_C], any_order: Annotated[Float64[:, :], ORDER_ANY] ) -> None: ... """, - module_name="edited", + module_name="fortran_contract", ) + c = parse_pyi_text( + """ +def consume( + a: Float64[:, :], + f: Annotated[Float64[:, :], ORDER_F], + any_order: Annotated[Float64[:, :], ORDER_ANY] +) -> None: ... +""", + module_name="c_contract", + native_language="c", + ) + + fortran_arrays = [arg.semantic_type.storage.array for arg in fortran.functions[0].arguments] + c_arrays = [arg.semantic_type.storage.array for arg in c.functions[0].arguments] + assert [array.order for array in fortran_arrays] == ["ORDER_F", "ORDER_C", "ORDER_ANY"] + assert [array.order for array in c_arrays] == ["ORDER_C", "ORDER_F", "ORDER_ANY"] + assert fortran_arrays[0].category is None + assert c_arrays[1].source_shape == [] + assert all(not arg.semantic_type.constraints for arg in fortran.functions[0].arguments) - arrays = [arg.semantic_type.storage.array for arg in module.functions[0].arguments] - assert arrays[0].order == "ORDER_C" - assert arrays[1].order == "ORDER_F" - assert arrays[2].order == "ORDER_C" - assert arrays[3].order == "ORDER_ANY" - assert arrays[0].category is None - assert arrays[1].source_shape == [] - assert all(not arg.semantic_type.constraints for arg in module.functions[0].arguments) + +@pytest.mark.parametrize( + ("native_language", "order"), + [("fortran", "ORDER_F"), ("c", "ORDER_C")], +) +def test_convert_pyi_to_ir_rejects_redundant_default_array_order(native_language: str, order: str): + with pytest.raises(ValueError, match=rf"{order} is implicit for {native_language}"): + parse_pyi_text( + f"value: Annotated[Float64[:, :], {order}]\n", + module_name="redundant_order", + native_language=native_language, + ) def test_convert_pyi_to_ir_records_explicit_c_to_fortran_copy_order(): @@ -602,7 +624,7 @@ def test_convert_pyi_to_ir_accepts_flat_array_dimension(): def test_convert_pyi_to_ir_preserves_extended_array_metadata_and_nested_selector(): module = parse_pyi_text( """ -value: Annotated[Float64[:, :], ORDER_F, Contiguous, ArrayCategory("deferred_shape")] +value: Annotated[Float64[:, :], Contiguous, ArrayCategory("deferred_shape")] nested: Float64[:, :][rank, kind] name: Annotated[String[16], FortranAllocatable] @@ -633,10 +655,10 @@ def fill(x: Float64[:]) -> None: ... def test_convert_pyi_to_ir_accepts_array_descriptor_handle_wrappers(): module = pyi_text_to_semantic_module( """ -from x2py.contracts import Allocatable as A, Annotated, Float64 as F64, Name, Pointer as P, String as Str +from x2py.contracts import Allocatable as A, Annotated, Float64 as F64, Pointer as P, SourceName, String as Str values: A[F64[:]] -target: Annotated[P[F64[:, :]], Name("target_values")] +target: Annotated[P[F64[:, :]], SourceName("target_values")] labels: P[Str[8][:]] plain_values: F64[:] @@ -833,7 +855,7 @@ def helper(value: Int32) -> None: ... "ORDER_C conflicts with ORDER_F implied by Flat placement", ), ( - "value: Annotated[Float64[:, :], ORDER_F, COPY_F]\n", + "value: Annotated[Float64[:, :], COPY_F]\n", "COPY_F requires a C-order Python array and targets Fortran order", ), ( @@ -841,7 +863,7 @@ def helper(value: Int32) -> None: ... "COPY_F requires a concrete multidimensional array rank", ), ( - "value: Annotated[Float64[::, ::], COPY_F]\n", + "value: Annotated[Float64[::, ::], ORDER_C, COPY_F]\n", "COPY_F initially supports only dense concrete-shape arrays", ), ( diff --git a/tests/semantics/readiness/test_pyi_readiness.py b/tests/semantics/readiness/test_pyi_readiness.py index 68d97e640..1d68511c1 100644 --- a/tests/semantics/readiness/test_pyi_readiness.py +++ b/tests/semantics/readiness/test_pyi_readiness.py @@ -119,6 +119,19 @@ def test_assess_pyi_wrap_readiness_honors_explicit_encoding(tmp_path: Path): assert _blocker_codes(report) == set() +def test_assess_pyi_wrap_readiness_uses_the_selected_native_language(tmp_path: Path): + pyi = tmp_path / "c_contract.pyi" + pyi.write_text( + f"{CONTRACT_IMPORT}matrix: Annotated[Float64[:, :], ORDER_F]\n", + encoding="utf-8", + ) + + report = assess_pyi_wrap_readiness(pyi, native_language="c") + + assert report["n_modules"] == 1 + assert report["source"] == [str(pyi)] + + def test_readiness_accepts_qualified_types_from_imported_modules_and_aliases(): report = _readiness_from_pyi( """ diff --git a/tests/wrapper/fortran/arrays/contracts/farray_contracts_f90/farray_contracts_f90.pyi b/tests/wrapper/fortran/arrays/contracts/farray_contracts_f90/farray_contracts_f90.pyi index d4df83e35..3df3d2218 100644 --- a/tests/wrapper/fortran/arrays/contracts/farray_contracts_f90/farray_contracts_f90.pyi +++ b/tests/wrapper/fortran/arrays/contracts/farray_contracts_f90/farray_contracts_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Flat, Float64, Int32, ORDER_F, Returns, native_call +from x2py.contracts import Addr, Arg, Flat, Float64, Int32, Returns, native_call @native_call([Addr(Arg(0)), Arg(1)]) def sum_assumed_size( @@ -30,71 +30,71 @@ def shift1( ) -> Returns["out", Float64[::]]: ... def shift2( - values: Annotated[Float64[::, ::], ORDER_F], - out: Annotated[Float64[::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::], ORDER_F]]: ... + values: Float64[::, ::], + out: Float64[::, ::] +) -> Returns["out", Float64[::, ::]]: ... def shift3( - values: Annotated[Float64[::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::], + out: Float64[::, ::, ::] +) -> Returns["out", Float64[::, ::, ::]]: ... def shift4( - values: Annotated[Float64[::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::], + out: Float64[::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::]]: ... def shift5( - values: Annotated[Float64[::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::]]: ... def shift6( - values: Annotated[Float64[::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::]]: ... def shift7( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::]]: ... def shift8( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift9( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift10( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift11( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift12( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift13( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift14( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... def shift15( - values: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], ORDER_F]]: ... + values: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::], + out: Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::] +) -> Returns["out", Float64[::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::, ::]]: ... diff --git a/tests/wrapper/fortran/arrays/contracts/farray_results_f90/farray_results_f90.pyi b/tests/wrapper/fortran/arrays/contracts/farray_results_f90/farray_results_f90.pyi index e7d8bdf69..651440ab0 100644 --- a/tests/wrapper/fortran/arrays/contracts/farray_results_f90/farray_results_f90.pyi +++ b/tests/wrapper/fortran/arrays/contracts/farray_results_f90/farray_results_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Allocatable, Annotated, Arg, Float64, Int32, ORDER_F, native_call +from x2py.contracts import Addr, Allocatable, Arg, Float64, Int32, native_call def fixed_vector() -> Float64[3]: ... @@ -11,44 +11,44 @@ def automatic_vector( def automatic_matrix( rows: Int32, cols: Int32 -) -> Annotated[Float64[rows, cols], ORDER_F]: ... +) -> Float64[rows, cols]: ... @native_call([Addr(Arg(0)), Addr(Arg(1)), Addr(Arg(2))]) def rank3_cube( n1: Int32, n2: Int32, n3: Int32 -) -> Annotated[Float64[n1, n2, n3], ORDER_F]: ... +) -> Float64[n1, n2, n3]: ... def rank1_result() -> Float64[2]: ... -def rank2_result() -> Annotated[Float64[2, 1], ORDER_F]: ... +def rank2_result() -> Float64[2, 1]: ... -def rank3_result() -> Annotated[Float64[2, 1, 1], ORDER_F]: ... +def rank3_result() -> Float64[2, 1, 1]: ... -def rank4_result() -> Annotated[Float64[2, 1, 1, 1], ORDER_F]: ... +def rank4_result() -> Float64[2, 1, 1, 1]: ... -def rank5_result() -> Annotated[Float64[2, 1, 1, 1, 1], ORDER_F]: ... +def rank5_result() -> Float64[2, 1, 1, 1, 1]: ... -def rank6_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank6_result() -> Float64[2, 1, 1, 1, 1, 1]: ... -def rank7_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank7_result() -> Float64[2, 1, 1, 1, 1, 1, 1]: ... -def rank8_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank8_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1]: ... -def rank9_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank9_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank10_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank10_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank11_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank11_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank12_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank12_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank13_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank13_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank14_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank14_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... -def rank15_result() -> Annotated[Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], ORDER_F]: ... +def rank15_result() -> Float64[2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]: ... def zero_vector() -> Float64[0]: ... diff --git a/tests/wrapper/fortran/arrays/contracts/multid_arrays/multid_arrays.pyi b/tests/wrapper/fortran/arrays/contracts/multid_arrays/multid_arrays.pyi index 4005a5c36..5f84b5ba7 100644 --- a/tests/wrapper/fortran/arrays/contracts/multid_arrays/multid_arrays.pyi +++ b/tests/wrapper/fortran/arrays/contracts/multid_arrays/multid_arrays.pyi @@ -1,17 +1,17 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, ORDER_F, Returns, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Returns, native_call def scale2_contiguous( - a: Annotated[Float64[:, :], ORDER_F], - out: Annotated[Float64[:, :], ORDER_F] -) -> Returns["out", Annotated[Float64[:, :], ORDER_F]]: ... + a: Float64[:, :], + out: Float64[:, :] +) -> Returns["out", Float64[:, :]]: ... def scale2_strided( - a: Annotated[Float64[::, ::], ORDER_F], - out: Annotated[Float64[::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::], ORDER_F]]: ... + a: Float64[::, ::], + out: Float64[::, ::] +) -> Returns["out", Float64[::, ::]]: ... def checksum2_strided( - a: Annotated[Float64[::, ::], ORDER_F], + a: Float64[::, ::], checksum: Float64[1] ) -> Returns["checksum", Float64[1]]: ... @@ -19,21 +19,21 @@ def checksum2_strided( def scale2_explicit( rows: Int32, cols: Int32, - a: Annotated[Float64[rows, cols], ORDER_F], - out: Annotated[Float64[rows, cols], ORDER_F] -) -> Returns["out", Annotated[Float64[rows, cols], ORDER_F]]: ... + a: Float64[rows, cols], + out: Float64[rows, cols] +) -> Returns["out", Float64[rows, cols]]: ... def shift3_contiguous( - a: Annotated[Float64[:, :, :], ORDER_F], - out: Annotated[Float64[:, :, :], ORDER_F] -) -> Returns["out", Annotated[Float64[:, :, :], ORDER_F]]: ... + a: Float64[:, :, :], + out: Float64[:, :, :] +) -> Returns["out", Float64[:, :, :]]: ... def shift3_strided( - a: Annotated[Float64[::, ::, ::], ORDER_F], - out: Annotated[Float64[::, ::, ::], ORDER_F] -) -> Returns["out", Annotated[Float64[::, ::, ::], ORDER_F]]: ... + a: Float64[::, ::, ::], + out: Float64[::, ::, ::] +) -> Returns["out", Float64[::, ::, ::]]: ... def checksum3_strided( - a: Annotated[Float64[::, ::, ::], ORDER_F], + a: Float64[::, ::, ::], checksum: Float64[1] ) -> Returns["checksum", Float64[1]]: ... diff --git a/tests/wrapper/fortran/derived_types/contracts/fbound_constructor_phase9/fclasses_f90.pyi b/tests/wrapper/fortran/derived_types/contracts/fbound_constructor_phase9/fclasses_f90.pyi index 351626265..8bab34c05 100644 --- a/tests/wrapper/fortran/derived_types/contracts/fbound_constructor_phase9/fclasses_f90.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/fbound_constructor_phase9/fclasses_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Pass, Polymorphic, bind, native_call +from x2py.contracts import Addr, Arg, Float64, Pass, bind, native_call class vector: @@ -16,6 +16,6 @@ class vector: @native_call([Addr(Arg(0)), Arg(1), Addr(Arg(2))]) def shift_vector( dx: Float64, - owner: Annotated[vector, Polymorphic], + owner: vector, dy: Float64, ) -> None: ... diff --git a/tests/wrapper/fortran/derived_types/contracts/fclasses_f90/fclasses_f90.pyi b/tests/wrapper/fortran/derived_types/contracts/fclasses_f90/fclasses_f90.pyi index 30633008a..1aa56ee3b 100644 --- a/tests/wrapper/fortran/derived_types/contracts/fclasses_f90/fclasses_f90.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/fclasses_f90/fclasses_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Allocatable, Annotated, Arg, Float64, Int64, ORDER_F, Pass, Polymorphic, bind, native_call +from x2py.contracts import Addr, Allocatable, Arg, Float64, Int64, Pass, bind, native_call class vector: def __init__( @@ -31,7 +31,7 @@ class vector_store: def __init__(self) -> None: ... values: Allocatable[Float64[:]] - matrix: Allocatable[Annotated[Float64[:, :], ORDER_F]] + matrix: Allocatable[Float64[:, :]] @native_call([Pass(), Addr(Arg(0))]) def allocate_values( @@ -53,7 +53,7 @@ class vector_store: def set_matrix( self, - source: Annotated[Float64[::, ::], ORDER_F] + source: Float64[::, ::] ) -> None: ... @staticmethod @@ -66,42 +66,42 @@ class vector_store: @native_call([Arg(0), Addr(Arg(1))]) def scale( - self: Annotated[vector, Polymorphic], + self: vector, factor: Float64 ) -> None: ... @native_call([Addr(Arg(0)), Arg(1), Addr(Arg(2))]) def shift_vector( dx: Float64, - owner: Annotated[vector, Polymorphic], + owner: vector, dy: Float64 ) -> None: ... def magnitude( - self: Annotated[vector, Polymorphic] + self: vector ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1))]) def allocate_values( - self: Annotated[vector_store, Polymorphic], + self: vector_store, n: Int64 ) -> None: ... def set_values( - self: Annotated[vector_store, Polymorphic], + self: vector_store, source: Float64[::] ) -> None: ... @native_call([Arg(0), Addr(Arg(1)), Addr(Arg(2))]) def allocate_matrix( - self: Annotated[vector_store, Polymorphic], + self: vector_store, rows: Int64, cols: Int64 ) -> None: ... def set_matrix( - self: Annotated[vector_store, Polymorphic], - source: Annotated[Float64[::, ::], ORDER_F] + self: vector_store, + source: Float64[::, ::] ) -> None: ... @native_call([Addr(Arg(0)), Addr(Arg(1))]) diff --git a/tests/wrapper/fortran/derived_types/contracts/finheritance_f90/finheritance_f90.pyi b/tests/wrapper/fortran/derived_types/contracts/finheritance_f90/finheritance_f90.pyi index 57f826f6b..39ae37d17 100644 --- a/tests/wrapper/fortran/derived_types/contracts/finheritance_f90/finheritance_f90.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/finheritance_f90/finheritance_f90.pyi @@ -44,21 +44,21 @@ class box(base_shape): def area(self) -> Float64: ... def base_area( - self: Annotated[base_shape, Polymorphic] + self: base_shape ) -> Float64: ... @native_call([Arg(0), Addr(Arg(1))]) def base_set_size( - self: Annotated[base_shape, Polymorphic], + self: base_shape, value: Float64 ) -> None: ... def circle_area( - self: Annotated[circle, Polymorphic] + self: circle ) -> Float64: ... def box_area( - self: Annotated[box, Polymorphic] + self: box ) -> Float64: ... def describe_shape( diff --git a/tests/wrapper/fortran/derived_types/contracts/fpointer_handles_policy/fpointer_handles_f90.pyi b/tests/wrapper/fortran/derived_types/contracts/fpointer_handles_policy/fpointer_handles_f90.pyi index bd43cef44..a25dec28d 100644 --- a/tests/wrapper/fortran/derived_types/contracts/fpointer_handles_policy/fpointer_handles_f90.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/fpointer_handles_policy/fpointer_handles_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Aliased, Allocatable, Annotated, Float64, Pointer, PointerAssociation, PointerPolicy, Polymorphic, bind +from x2py.contracts import Aliased, Allocatable, Annotated, Float64, Pointer, PointerAssociation, PointerPolicy, bind class pointer_box: def __init__(self) -> None: ... @@ -50,8 +50,8 @@ def associate_module_slice() -> None: ... def associate_module_contiguous() -> None: ... def allocate_module_values() -> None: ... -def box_associate_values(self: Annotated[pointer_box, Polymorphic]) -> None: ... -def box_associate_values_strided(self: Annotated[pointer_box, Polymorphic]) -> None: ... +def box_associate_values(self: pointer_box) -> None: ... +def box_associate_values_strided(self: pointer_box) -> None: ... def sum_values(values: Float64[::]) -> Float64: ... def sum_pointer_descriptor(values: Pointer[Float64[:]]) -> Float64: ... diff --git a/tests/wrapper/fortran/edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi b/tests/wrapper/fortran/edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi index f82aa2376..8afcb5dc2 100644 --- a/tests/wrapper/fortran/edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi +++ b/tests/wrapper/fortran/edit_pyi_contracts/modified_contracts/fnative_call_examples_native_order/fnative_call_examples_f90.pyi @@ -1,6 +1,6 @@ # Intentional difference: no native-call decorators. Output slots stay # visible in native dummy-argument order. -from x2py.contracts import Addr, Annotated, Float64, Int32, ORDER_F, String, bind +from x2py.contracts import Addr, Float64, Int32, String, bind class summary_point: def __init__( @@ -38,8 +38,8 @@ def fill_vector_raw( def shift_matrix( n: Int32[()], m: Int32[()], - values: Annotated[Float64[n, m], ORDER_F], - out: Annotated[Float64[n, m], ORDER_F] + values: Float64[n, m], + out: Float64[n, m] ) -> None: ... def scale_with_status( diff --git a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py index fe1d9bf5d..0f3ac2da6 100644 --- a/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py +++ b/tests/wrapper/fortran/edit_pyi_contracts/test_native_order_contracts.py @@ -110,7 +110,7 @@ def test_raw_array_addresses_use_canonical_plan(tmp_path: Path): encoding="utf-8", ) (contract_package / "fnative_call_examples_f90.pyi").write_text( - """from x2py.contracts import Addr, Annotated, Float64, Int32, ORDER_F, bind + """from x2py.contracts import Addr, Float64, Int32, bind @bind("fill_vector") def fill_vector_raw(n: Int32[()], values: Addr(Float64[n])) -> None: ... @@ -127,8 +127,8 @@ def shift_matrix_raw_c( def shift_matrix_raw_f( n: Int32[()], m: Int32[()], - values: Annotated[Addr(Float64[n, m]), ORDER_F], - out: Annotated[Addr(Float64[n, m]), ORDER_F] + values: Addr(Float64[n, m]), + out: Addr(Float64[n, m]) ) -> None: ... """, encoding="utf-8", diff --git a/tests/wrapper/fortran/function_calls/contracts/fnative_call_examples_f90/fnative_call_examples_f90.pyi b/tests/wrapper/fortran/function_calls/contracts/fnative_call_examples_f90/fnative_call_examples_f90.pyi index 8e3ce65c7..43a16203d 100644 --- a/tests/wrapper/fortran/function_calls/contracts/fnative_call_examples_f90/fnative_call_examples_f90.pyi +++ b/tests/wrapper/fortran/function_calls/contracts/fnative_call_examples_f90/fnative_call_examples_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, ORDER_F, Return, Returns, String, native_call +from x2py.contracts import Addr, Arg, Float64, Int32, Return, Returns, String, native_call class summary_point: def __init__( @@ -26,9 +26,9 @@ def fill_vector( def shift_matrix( n: Int32, m: Int32, - values: Annotated[Float64[n, m], ORDER_F], - out: Annotated[Float64[n, m], ORDER_F] -) -> Returns["out", Annotated[Float64[n, m], ORDER_F]]: ... + values: Float64[n, m], + out: Float64[n, m] +) -> Returns["out", Float64[n, m]]: ... @native_call([Arg(0), Return('status', 0)]) def scale_with_status( diff --git a/tests/wrapper/fortran/function_calls/contracts/foutputs_f90/foutputs_f90.pyi b/tests/wrapper/fortran/function_calls/contracts/foutputs_f90/foutputs_f90.pyi index dc9e7bf12..3422a4b1c 100644 --- a/tests/wrapper/fortran/function_calls/contracts/foutputs_f90/foutputs_f90.pyi +++ b/tests/wrapper/fortran/function_calls/contracts/foutputs_f90/foutputs_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Allocatable, Annotated, Arg, Float64, Int32, ORDER_F, Return, Returns, String, native_call +from x2py.contracts import Addr, Allocatable, Arg, Float64, Int32, Return, Returns, String, native_call class output_point: def __init__( @@ -26,8 +26,8 @@ def fill_vector( def fill_matrix( n: Int32, m: Int32, - values: Annotated[Float64[n, m], ORDER_F] -) -> Returns["values", Annotated[Float64[n, m], ORDER_F]]: ... + values: Float64[n, m] +) -> Returns["values", Float64[n, m]]: ... @native_call([Addr(Arg(0)), Return('values', 0)]) def build_alloc( diff --git a/tests/wrapper/fortran/module_state/contracts/fallocatable_views_f90/fallocatable_views_f90.pyi b/tests/wrapper/fortran/module_state/contracts/fallocatable_views_f90/fallocatable_views_f90.pyi index 72fe5dfae..bdab7e9b8 100644 --- a/tests/wrapper/fortran/module_state/contracts/fallocatable_views_f90/fallocatable_views_f90.pyi +++ b/tests/wrapper/fortran/module_state/contracts/fallocatable_views_f90/fallocatable_views_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Aliased, Allocatable, Annotated, Arg, Float64, Int32, ORDER_F, Pass, Return, native_call +from x2py.contracts import Addr, Aliased, Allocatable, Annotated, Arg, Float64, Int32, Pass, Return, native_call class buffer: def __init__(self) -> None: ... @@ -46,7 +46,7 @@ def build_values( def build_matrix( n: Int32, m: Int32 -) -> Allocatable[Annotated[Float64[:, :], ORDER_F]]: ... +) -> Allocatable[Float64[:, :]]: ... @native_call([Addr(Arg(0))]) def make_values( @@ -57,4 +57,4 @@ def make_values( def make_matrix( n: Int32, m: Int32 -) -> Allocatable[Annotated[Float64[:, :], ORDER_F]]: ... +) -> Allocatable[Float64[:, :]]: ... diff --git a/tests/wrapper/fortran/naming/contracts/fclass_overloads_phase9/foverloads_f90.pyi b/tests/wrapper/fortran/naming/contracts/fclass_overloads_phase9/foverloads_f90.pyi index 2407c3fb8..1a093ac8f 100644 --- a/tests/wrapper/fortran/naming/contracts/fclass_overloads_phase9/foverloads_f90.pyi +++ b/tests/wrapper/fortran/naming/contracts/fclass_overloads_phase9/foverloads_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, Pass, Polymorphic, bind, native_call, overload, private +from x2py.contracts import Addr, Arg, Float64, Int32, Pass, bind, native_call, overload, private class accumulator: @@ -26,7 +26,7 @@ class accumulator: @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_integer( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Int32, ) -> None: ... @@ -34,6 +34,6 @@ def accumulator_add_integer( @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_real( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Float64, ) -> None: ... diff --git a/tests/wrapper/fortran/naming/contracts/fconstructor_overloads_phase9/foverloads_f90.pyi b/tests/wrapper/fortran/naming/contracts/fconstructor_overloads_phase9/foverloads_f90.pyi index 1e95bf72e..f79788b17 100644 --- a/tests/wrapper/fortran/naming/contracts/fconstructor_overloads_phase9/foverloads_f90.pyi +++ b/tests/wrapper/fortran/naming/contracts/fconstructor_overloads_phase9/foverloads_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Float64, Int32, Pass, Polymorphic, bind, native_call, overload, private +from x2py.contracts import Addr, Arg, Float64, Int32, Pass, bind, native_call, overload, private class accumulator: @@ -32,7 +32,7 @@ class accumulator: @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_integer( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Int32, ) -> None: ... @@ -40,6 +40,6 @@ def accumulator_add_integer( @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_real( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Float64, ) -> None: ... diff --git a/tests/wrapper/fortran/naming/contracts/fnaming_f90/fnaming_f90.pyi b/tests/wrapper/fortran/naming/contracts/fnaming_f90/fnaming_f90.pyi index 2d436182b..0b072c2f6 100644 --- a/tests/wrapper/fortran/naming/contracts/fnaming_f90/fnaming_f90.pyi +++ b/tests/wrapper/fortran/naming/contracts/fnaming_f90/fnaming_f90.pyi @@ -1,15 +1,15 @@ -from x2py.contracts import Addr, Annotated, Arg, Int32, Name, bind, native_call +from x2py.contracts import Addr, Annotated, Arg, Int32, SourceName, bind, native_call class visible_t: def __init__( self, *, - lambda_: Annotated[Int32, Name("lambda")] = 3, - lambda__2: Annotated[Int32, Name("lambda_")] = 4 + lambda_: Annotated[Int32, SourceName("lambda")] = 3, + lambda__2: Annotated[Int32, SourceName("lambda_")] = 4 ) -> None: ... - lambda_: Annotated[Int32, Name("lambda")] = 3 - lambda__2: Annotated[Int32, Name("lambda_")] = 4 + lambda_: Annotated[Int32, SourceName("lambda")] = 3 + lambda__2: Annotated[Int32, SourceName("lambda_")] = 4 @bind("visible_from") def from_(self) -> Int32: ... diff --git a/tests/wrapper/fortran/naming/contracts/foperators_f90/foperators_f90.pyi b/tests/wrapper/fortran/naming/contracts/foperators_f90/foperators_f90.pyi index 0bd0102a1..f7688635d 100644 --- a/tests/wrapper/fortran/naming/contracts/foperators_f90/foperators_f90.pyi +++ b/tests/wrapper/fortran/naming/contracts/foperators_f90/foperators_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Bool, Float64, Int32, Pass, Polymorphic, Returns, bind, native_call, overload, private +from x2py.contracts import Addr, Arg, Bool, Float64, Int32, Pass, Returns, bind, native_call, overload, private class vector: def __init__( @@ -430,7 +430,7 @@ def assign_vector_real( @private @native_call([Arg(0), Addr(Arg(1))]) def counter_add_integer( - self: Annotated[counter, Polymorphic], + self: counter, right: Int32 ) -> counter: ... diff --git a/tests/wrapper/fortran/naming/contracts/foverloads_f90/foverloads_f90.pyi b/tests/wrapper/fortran/naming/contracts/foverloads_f90/foverloads_f90.pyi index 56fd11b24..677fd7887 100644 --- a/tests/wrapper/fortran/naming/contracts/foverloads_f90/foverloads_f90.pyi +++ b/tests/wrapper/fortran/naming/contracts/foverloads_f90/foverloads_f90.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Complex128, Float64, Int32, Pass, Polymorphic, bind, native_call, overload, private +from x2py.contracts import Addr, Arg, Complex128, Float64, Int32, Pass, bind, native_call, overload, private class accumulator: def __init__( @@ -88,14 +88,14 @@ def inspect_sample( @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_integer( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Int32 ) -> None: ... @private @native_call([Arg(0), Addr(Arg(1))]) def accumulator_add_real( - self: Annotated[accumulator, Polymorphic], + self: accumulator, value: Float64 ) -> None: ... diff --git a/tests/wrapper/fortran/real_libraries/contracts/lapack/__init__.pyi b/tests/wrapper/fortran/real_libraries/contracts/lapack/__init__.pyi index a245f2c67..415845176 100644 --- a/tests/wrapper/fortran/real_libraries/contracts/lapack/__init__.pyi +++ b/tests/wrapper/fortran/real_libraries/contracts/lapack/__init__.pyi @@ -1,4 +1,4 @@ -from x2py.contracts import Addr, Annotated, Arg, Bool, Complex128, Complex64, Flat, Float32, Float64, Int32, ORDER_F, Return, Returns, String, bind, external, native_call +from x2py.contracts import Addr, Arg, Bool, Complex128, Complex64, Flat, Float32, Float64, Int32, Return, Returns, String, bind, external, native_call from . import LA_CONSTANTS from . import LA_XISNAN @@ -2879,7 +2879,7 @@ def chesvxx( def cheswapr( UPLO: String[1], N: Int32, - A: Annotated[Complex64[LDA, N], ORDER_F], + A: Complex64[LDA, N], LDA: Int32, I1: Int32, I2: Int32 @@ -3934,8 +3934,8 @@ def cla_lin_berr( N: Int32, NZ: Int32, NRHS: Int32, - RES: Annotated[Complex64[N, NRHS], ORDER_F], - AYB: Annotated[Float32[N, NRHS], ORDER_F], + RES: Complex64[N, NRHS], + AYB: Float32[N, NRHS], BERR: Float32[NRHS] ) -> None: ... @@ -4505,9 +4505,9 @@ def clahr2( A: Complex64[LDA, Flat], LDA: Int32, TAU: Complex64[NB], - T: Annotated[Complex64[LDT, NB], ORDER_F], + T: Complex64[LDT, NB], LDT: Int32, - Y: Annotated[Complex64[LDY, NB], ORDER_F], + Y: Complex64[LDY, NB], LDY: Int32 ) -> None: ... @@ -7308,7 +7308,7 @@ def csysvxx( def csyswapr( UPLO: String[1], N: Int32, - A: Annotated[Complex64[LDA, N], ORDER_F], + A: Complex64[LDA, N], LDA: Int32, I1: Int32, I2: Int32 @@ -11463,8 +11463,8 @@ def dla_lin_berr( N: Int32, NZ: Int32, NRHS: Int32, - RES: Annotated[Float64[N, NRHS], ORDER_F], - AYB: Annotated[Float64[N, NRHS], ORDER_F], + RES: Float64[N, NRHS], + AYB: Float64[N, NRHS], BERR: Float64[NRHS] ) -> None: ... @@ -12172,9 +12172,9 @@ def dlahr2( A: Float64[LDA, Flat], LDA: Int32, TAU: Float64[NB], - T: Annotated[Float64[LDT, NB], ORDER_F], + T: Float64[LDT, NB], LDT: Int32, - Y: Annotated[Float64[LDY, NB], ORDER_F], + Y: Float64[LDY, NB], LDY: Int32 ) -> None: ... @@ -20918,8 +20918,8 @@ def sla_lin_berr( N: Int32, NZ: Int32, NRHS: Int32, - RES: Annotated[Float32[N, NRHS], ORDER_F], - AYB: Annotated[Float32[N, NRHS], ORDER_F], + RES: Float32[N, NRHS], + AYB: Float32[N, NRHS], BERR: Float32[NRHS] ) -> None: ... @@ -21627,9 +21627,9 @@ def slahr2( A: Float32[LDA, Flat], LDA: Int32, TAU: Float32[NB], - T: Annotated[Float32[LDT, NB], ORDER_F], + T: Float32[LDT, NB], LDT: Int32, - Y: Annotated[Float32[LDY, NB], ORDER_F], + Y: Float32[LDY, NB], LDY: Int32 ) -> None: ... @@ -30724,7 +30724,7 @@ def zhesvxx( def zheswapr( UPLO: String[1], N: Int32, - A: Annotated[Complex128[LDA, N], ORDER_F], + A: Complex128[LDA, N], LDA: Int32, I1: Int32, I2: Int32 @@ -31772,8 +31772,8 @@ def zla_lin_berr( N: Int32, NZ: Int32, NRHS: Int32, - RES: Annotated[Complex128[N, NRHS], ORDER_F], - AYB: Annotated[Float64[N, NRHS], ORDER_F], + RES: Complex128[N, NRHS], + AYB: Float64[N, NRHS], BERR: Float64[NRHS] ) -> None: ... @@ -32343,9 +32343,9 @@ def zlahr2( A: Complex128[LDA, Flat], LDA: Int32, TAU: Complex128[NB], - T: Annotated[Complex128[LDT, NB], ORDER_F], + T: Complex128[LDT, NB], LDT: Int32, - Y: Annotated[Complex128[LDY, NB], ORDER_F], + Y: Complex128[LDY, NB], LDY: Int32 ) -> None: ... diff --git a/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py b/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py index ce5f27368..71f8ef3ae 100644 --- a/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py +++ b/tests/wrapper_codegen/printers/test_pyi_printer_imports_and_packages.py @@ -55,8 +55,8 @@ def test_fortran_generated_contracts_reserve_colliding_public_names_by_namespace code = emit_module(module, normalize_fortran_public_names=True) - assert 'lambda_: Annotated[Int32, Name("lambda")]' in code - assert 'lambda__2: Annotated[Int32, Name("lambda_")]' in code + assert 'lambda_: Annotated[Int32, SourceName("lambda")]' in code + assert 'lambda__2: Annotated[Int32, SourceName("lambda_")]' in code assert '@bind("lambda")\ndef lambda_() -> Int32: ...' in code assert '@bind("lambda_")\ndef lambda__2() -> Int32: ...' in code assert "def lambda__3" not in code diff --git a/tests/wrapper_codegen/printers/test_types_and_declarations.py b/tests/wrapper_codegen/printers/test_types_and_declarations.py index aefc66c59..4061f26d6 100644 --- a/tests/wrapper_codegen/printers/test_types_and_declarations.py +++ b/tests/wrapper_codegen/printers/test_types_and_declarations.py @@ -149,7 +149,7 @@ def test_emit_matrix_shapes(): code = generate_pyi(source) - assert "A: Annotated[Float64[::, ::], ORDER_F" in code + assert "A: Float64[::, ::]" in code assert "Shape" not in code assert "x: Float64[::]" in code assert "y: Float64[::]" in code @@ -270,7 +270,7 @@ def test_emit_explicit_shape(): code = generate_pyi(source) - assert "A: Annotated[Float64[10, 20], ORDER_F]" in code + assert "A: Float64[10, 20]" in code def test_parameter_target_sanitizes_non_identifier_names(): @@ -285,7 +285,7 @@ def test_emit_argument_escapes_original_name_metadata(): emitted = PyiPrinter().emit(SemanticArgument('quote"name', SemanticType("Int32"))) reparsed = parse_pyi_text(f"def consume({emitted}) -> None: ...\n", module_name="quoted") - assert emitted == 'quote_name: Annotated[Int32, Name("quote\\"name")]' + assert emitted == 'quote_name: Annotated[Int32, SourceName("quote\\"name")]' assert reparsed.functions[0].arguments[0].name == 'quote"name' @@ -371,12 +371,12 @@ def test_emit_complex_fem_module(): # Matrix annotations # -------------------------------------------------------- - assert "K: Annotated[Float64[::, ::], ORDER_F" in code - assert 'Returns["K", Annotated[Float64[::, ::], ORDER_F]]' in code + assert "K: Float64[::, ::]" in code + assert 'Returns["K", Float64[::, ::]]' in code - assert "coords: Annotated[Float64[::, ::], ORDER_F" in code + assert "coords: Float64[::, ::]" in code - assert "connectivity: Annotated[Int32[::, ::], ORDER_F" in code + assert "connectivity: Int32[::, ::]" in code # -------------------------------------------------------- # Return type @@ -445,7 +445,7 @@ def test_printer_emit_visitor_dispatches_semantic_models(): assert printer.emit(constraint) == "Finite" assert printer.emit(semantic_type) == "Float64[:]" - assert printer.emit(argument) == 'class_: Annotated[Float64[:], Name("class")] = ...' + assert printer.emit(argument) == 'class_: Annotated[Float64[:], SourceName("class")] = ...' assert "def reset(self) -> None: ..." in printer.emit(method) assert "@private\nclass thing:" in printer.emit(cls) assert "var['bad-name']: Float64[:]" in printer.emit(cls) @@ -496,6 +496,24 @@ def test_printer_emits_flat_dimension_for_assumed_size_arrays(): assert PyiPrinter().emit(fortran_type) == "Float64[3, Flat]" assert PyiPrinter().emit(c_type) == "Annotated[Float64[Flat, 3], ORDER_C]" + nondefault_c_type = SemanticType( + "Float64", + dtype="Float64", + rank=2, + shape=[":", ":"], + storage=SemanticStorageContract( + kind="array", + array=SemanticArrayContract( + rank=2, + shape=[":", ":"], + source_shape=[":", ":"], + order="ORDER_C", + contiguous=True, + ), + ), + ) + assert PyiPrinter().emit(nondefault_c_type) == "Annotated[Float64[:, :], ORDER_C]" + lower_bound_assumed_size = SemanticType( "Float64", dtype="Float64", diff --git a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py index 13446ef85..afe75184a 100644 --- a/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py +++ b/tests/wrapper_codegen/test_phase6b_dense_array_shapes.py @@ -17,9 +17,9 @@ def _dense_plan(): module = parse_pyi_text( """ -from x2py.contracts import Annotated, Flat, Float64, Int32, ORDER_C, ORDER_F, external +from x2py.contracts import Annotated, Flat, Float64, Int32, ORDER_C, external -def dense_f(rows: Int32, cols: Int32, values: Annotated[Float64[rows, cols], ORDER_F]) -> None: ... +def dense_f(rows: Int32, cols: Int32, values: Float64[rows, cols]) -> None: ... def dense_c(rows: Int32, cols: Int32, values: Annotated[Float64[rows, cols], ORDER_C]) -> None: ... def flat(n: Int32, values: Float64[Flat]) -> None: ... diff --git a/tests/wrapper_codegen/test_phase6c_strided_arrays.py b/tests/wrapper_codegen/test_phase6c_strided_arrays.py index b66876045..338564e57 100644 --- a/tests/wrapper_codegen/test_phase6c_strided_arrays.py +++ b/tests/wrapper_codegen/test_phase6c_strided_arrays.py @@ -12,9 +12,9 @@ def _strided_plan(): module = parse_pyi_text( """ -from x2py.contracts import Annotated, Float64, ORDER_F +from x2py.contracts import Float64 -def strided(values: Annotated[Float64[::, ::], ORDER_F]) -> None: ... +def strided(values: Float64[::, ::]) -> None: ... """, module_name="strided_arrays", ) diff --git a/tests/wrapper_codegen/test_phase6g_raw_array_addresses.py b/tests/wrapper_codegen/test_phase6g_raw_array_addresses.py index 492db342f..1d553146b 100644 --- a/tests/wrapper_codegen/test_phase6g_raw_array_addresses.py +++ b/tests/wrapper_codegen/test_phase6g_raw_array_addresses.py @@ -25,11 +25,11 @@ def _raw_array_module(): module = parse_pyi_text( """ def raw_vector(n: Int32[()], values: Addr(Float64[n])) -> None: ... -def raw_matrix_c(n: Int32, m: Int32, values: Addr(Float64[n, m])) -> None: ... +def raw_matrix_c(n: Int32, m: Int32, values: Annotated[Addr(Float64[n, m]), ORDER_C]) -> None: ... def raw_matrix_f( n: Int32, m: Int32, - values: Annotated[Addr(Float64[n, m]), ORDER_F] + values: Addr(Float64[n, m]) ) -> None: ... def raw_labels(n: Int32, labels: Addr(String[8][n])) -> None: ... def raw_literal(values: Addr(Float64[4])) -> None: ... diff --git a/x2py/cli.py b/x2py/cli.py index 988a9f027..e7739978e 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -745,18 +745,19 @@ def _wrap_readiness_report( } for path, modules in converted_files } - out.update(_pyi_readiness_report(paths)) + out.update(_pyi_readiness_report(paths, native_language=language)) return out -def _pyi_readiness_report(paths: list[str]) -> dict[str, dict]: +def _pyi_readiness_report(paths: list[str], *, native_language: str = "fortran") -> dict[str, dict]: """Load one edited `.pyi` file set and report each interface path.""" pyi_paths = _expand_pyi_paths(paths) if not pyi_paths: return {} modules = pyi_paths_to_semantic_modules( - [raw for raw in paths if Path(raw).is_dir() or Path(raw).suffix.lower() == ".pyi"] + [raw for raw in paths if Path(raw).is_dir() or Path(raw).suffix.lower() == ".pyi"], + native_language=native_language, ) return { str(path): { diff --git a/x2py/contracts/__init__.py b/x2py/contracts/__init__.py index 449246d16..c9bb79aef 100644 --- a/x2py/contracts/__init__.py +++ b/x2py/contracts/__init__.py @@ -104,15 +104,15 @@ def apply(target): Finite = _expression IsPresent = _expression Len = _expression -Name = _expression Ownership = _expression Pass = _expression PointerAssociation = _expression PointerPolicy = _expression -Return = _expression -Value = _expression Range = _expression +Return = _expression +SourceName = _expression Transfer = _expression +Value = _expression Work = _expression bind = _decorator @@ -175,7 +175,6 @@ def apply(target): "IsPresent", "Len", "Matrix", - "Name", "Opaque", "OpaqueHandle", "ORDER_ANY", @@ -191,6 +190,7 @@ def apply(target): "Return", "Returns", "SizeT", + "SourceName", "Strided", "String", "Transfer", diff --git a/x2py/pipeline/pyi.py b/x2py/pipeline/pyi.py index c73f94dae..23a541b32 100644 --- a/x2py/pipeline/pyi.py +++ b/x2py/pipeline/pyi.py @@ -16,7 +16,7 @@ @dataclass class _PyiSemanticModuleCache: - modules: dict[tuple[Path, str, str], SemanticModule] = field(default_factory=dict) + modules: dict[tuple[Path, str, str, str], SemanticModule] = field(default_factory=dict) def file_to_semantic_module( self, @@ -24,10 +24,11 @@ def file_to_semantic_module( *, module_name: str | None = None, encoding: str = "utf-8", + native_language: str = "fortran", ) -> SemanticModule: pyi_path = Path(path) resolved_module_name = module_name or pyi_path.stem - key = (pyi_path.resolve(), resolved_module_name, encoding) + key = (pyi_path.resolve(), resolved_module_name, encoding, native_language) cached = self.modules.get(key) if cached is not None: return cached @@ -37,6 +38,7 @@ def file_to_semantic_module( source, module_name=resolved_module_name, filename=str(pyi_path), + native_language=native_language, ) except ValueError as exc: raise ValueError(f"{pyi_path}: {exc}") from exc @@ -48,6 +50,7 @@ def paths_to_semantic_modules( paths: str | Path | Iterable[str | Path], *, encoding: str = "utf-8", + native_language: str = "fortran", ) -> list[SemanticModule]: raw_paths = [paths] if isinstance(paths, str | Path) else list(paths) expanded: dict[Path, str | None] = {} @@ -66,32 +69,62 @@ def paths_to_semantic_modules( expanded.setdefault(path, None) return reconcile_external_type_refs( [ - self.file_to_semantic_module(path, module_name=module_name, encoding=encoding) + self.file_to_semantic_module( + path, + module_name=module_name, + encoding=encoding, + native_language=native_language, + ) for path, module_name in sorted(expanded.items()) ] ) -def pyi_text_to_semantic_module(source: str, *, module_name: str = "", filename: str = "") -> SemanticModule: +def pyi_text_to_semantic_module( + source: str, + *, + module_name: str = "", + filename: str = "", + native_language: str = "fortran", +) -> SemanticModule: """Parse inline semantic `.pyi` text and convert it to semantic IR.""" tree = parse_pyi_text(source, filename=filename) - module = convert_pyi_to_ir(tree, module_name=module_name, source=source) + module = convert_pyi_to_ir( + tree, + module_name=module_name, + source=source, + native_language=native_language, + ) module.metadata[PYI_LOADED_METADATA] = True return module def pyi_file_to_semantic_module( - path: str | Path, *, module_name: str | None = None, encoding: str = "utf-8" + path: str | Path, + *, + module_name: str | None = None, + encoding: str = "utf-8", + native_language: str = "fortran", ) -> SemanticModule: """Convert one semantic `.pyi` file to semantic IR.""" - return _PyiSemanticModuleCache().file_to_semantic_module(path, module_name=module_name, encoding=encoding) + return _PyiSemanticModuleCache().file_to_semantic_module( + path, + module_name=module_name, + encoding=encoding, + native_language=native_language, + ) def pyi_paths_to_semantic_modules( paths: str | Path | Iterable[str | Path], *, encoding: str = "utf-8", + native_language: str = "fortran", ) -> list[SemanticModule]: """Convert semantic `.pyi` files or directories and reconcile external types.""" - return _PyiSemanticModuleCache().paths_to_semantic_modules(paths, encoding=encoding) + return _PyiSemanticModuleCache().paths_to_semantic_modules( + paths, + encoding=encoding, + native_language=native_language, + ) diff --git a/x2py/semantics/pyi2ir.py b/x2py/semantics/pyi2ir.py index d13ed0585..303738a30 100644 --- a/x2py/semantics/pyi2ir.py +++ b/x2py/semantics/pyi2ir.py @@ -72,12 +72,22 @@ class _CallbackArgumentSpec: passes_by_value: bool -def convert_pyi_to_ir(tree: ast.Module, *, module_name: str = "", source: str = "") -> SemanticModule: +def convert_pyi_to_ir( + tree: ast.Module, + *, + module_name: str = "", + source: str = "", + native_language: str = "fortran", +) -> SemanticModule: """Convert a parsed semantic `.pyi` AST into semantic IR.""" if not isinstance(tree, ast.Module): raise TypeError("convert_pyi_to_ir expects a Python ast.Module parsed by x2py.parsers.pyi") - module = _PyiAstParser(module_name=module_name, source=source).parse(tree) + module = _PyiAstParser( + module_name=module_name, + source=source, + native_language=native_language, + ).parse(tree) _annotate_imported_external_type_refs(module) return module @@ -108,9 +118,13 @@ class _PendingOverload: class _PyiAstParser: - def __init__(self, *, module_name: str, source: str = ""): - self.module = SemanticModule(name=module_name) + def __init__(self, *, module_name: str, source: str = "", native_language: str = "fortran"): + native_language = native_language.casefold() + if native_language not in {"c", "fortran"}: + raise ValueError(f"Unsupported semantic .pyi native language: {native_language!r}") + self.module = SemanticModule(name=module_name, origin=SemanticOrigin(source_language=native_language)) self.source = source + self.native_language = native_language self._pending_overloads: list[_PendingOverload] = [] self._contract_bindings: dict[str, str] = {} self._user_type_names: set[str] = set() @@ -698,6 +712,7 @@ def _restore_type_bound_targets(self) -> None: target.metadata["fortran_type_bound_target"] = True target.metadata["fortran_passed_object_name"] = target.arguments[passed_position].name target.metadata["fortran_passed_object_position"] = passed_position + target.arguments[passed_position].semantic_type.metadata["fortran_polymorphic"] = True @classmethod def _iter_classes(cls, classes: list[SemanticClass]): @@ -1377,6 +1392,7 @@ def _address_type(self, node: ast.Call) -> SemanticType: pointee = self.semantic_type(node.args[0]) read_only = pointee.storage.read_only if pointee.storage is not None else False metadata = dict(pointee.storage.metadata) if pointee.storage is not None else {} + array = pointee.storage.array if pointee.storage is not None else None metadata[ADDRESS_ROLE_METADATA] = ADDRESS_ROLE_RAW pointer_depth = self._addr_depth(node.func) pointee.storage = SemanticStorageContract( @@ -1384,6 +1400,7 @@ def _address_type(self, node: ast.Call) -> SemanticType: read_only=read_only, mutable=not read_only, pointer_depth=pointer_depth, + array=array, metadata=metadata, ) pointee.ownership.mutable = not read_only @@ -1479,8 +1496,8 @@ def _strided_dimension_text(text: str) -> str: lower, upper, _step = text.split(":", 2) return f"{lower.strip()}:{upper.strip()}:{_STRIDED_DIMENSION_SENTINEL}" - @staticmethod def _array_type_from_dimensions( + self, name: str, dims: list[str], *, @@ -1498,7 +1515,7 @@ def _array_type_from_dimensions( array = SemanticArrayContract( rank=rank, shape=list(dims), - order=_PyiAstParser._array_order_for_dimensions(category, rank, source_shape), + order=self._array_order_for_dimensions(category, rank, source_shape), axes=["strided" if strided else "dense" for strided in strided_axes], contiguous=not any(strided_axes), category=category, @@ -1551,8 +1568,8 @@ def _flat_array_dimensions( upper_bounds, ) - @staticmethod def _array_order_for_dimensions( + self, category: str | None, rank: int | None, source_shape: list[str], @@ -1561,7 +1578,7 @@ def _array_order_for_dimensions( return None if category == "assumed_size": return _PyiAstParser._flat_array_order(source_shape, rank) - return "ORDER_C" + return "ORDER_F" if self.native_language == "fortran" else "ORDER_C" @staticmethod def _flat_array_order(source_shape: list[str], rank: int | None) -> str | None: @@ -1696,9 +1713,16 @@ def _apply_ownership_annotation_metadata(self, semantic_type: SemanticType, node def _apply_metadata_name(self, semantic_type: SemanticType, name: str) -> bool: if name in {"ORDER_C", "ORDER_F", "ORDER_ANY"}: array = self._require_array_storage(semantic_type) + if array.rank is None or array.rank <= 1: + raise ValueError(f"{name} requires a multidimensional array") expected_order = self._flat_array_order(array.source_shape, array.rank) if expected_order is not None and name != expected_order: raise ValueError(f"{name} conflicts with {expected_order} implied by Flat placement") + default_order = self._array_order_for_dimensions(array.category, array.rank, array.source_shape) + if expected_order is None and name == default_order: + raise ValueError( + f"{name} is implicit for {self.native_language} semantic .pyi contracts; remove the annotation" + ) array.order = name return True if name == "COPY_F": @@ -2048,9 +2072,9 @@ def returned_argument(self, node: ast.expr) -> SemanticArgument | None: ) def name_metadata(self, node: ast.expr) -> str | None: - if isinstance(node, ast.Call) and self.matches_name(node.func, "Name"): + if isinstance(node, ast.Call) and self.matches_name(node.func, "SourceName"): if len(node.args) != 1: - raise ValueError(f"Name metadata expects one argument: {ast.unparse(node)!r}") + raise ValueError(f"SourceName metadata expects one argument: {ast.unparse(node)!r}") return str(ast.literal_eval(node.args[0])) return None diff --git a/x2py/semantics/readiness.py b/x2py/semantics/readiness.py index 07ba98faa..f2f7e1203 100644 --- a/x2py/semantics/readiness.py +++ b/x2py/semantics/readiness.py @@ -90,13 +90,18 @@ def assess_pyi_wrap_readiness( paths: str | Path | Iterable[str | Path], *, encoding: str = "utf-8", + native_language: str = "fortran", ) -> dict: """Load one or more edited .pyi files and assess semantic wrap-readiness.""" from x2py.pipeline.pyi import pyi_paths_to_semantic_modules raw_paths = [paths] if isinstance(paths, str | Path) else list(paths) expanded = _expand_pyi_paths(raw_paths) - modules = pyi_paths_to_semantic_modules(raw_paths, encoding=encoding) + modules = pyi_paths_to_semantic_modules( + raw_paths, + encoding=encoding, + native_language=native_language, + ) return assess_semantic_wrap_readiness( modules, source=[str(path) for path in expanded], diff --git a/x2py/wrapper_codegen/printers/pyi_printer.py b/x2py/wrapper_codegen/printers/pyi_printer.py index 372d774f7..9f45f5458 100644 --- a/x2py/wrapper_codegen/printers/pyi_printer.py +++ b/x2py/wrapper_codegen/printers/pyi_printer.py @@ -82,10 +82,17 @@ def __init__(self, *, normalize_fortran_public_names: bool = False): self._semantic_class_names: set[str] = set() self._contract_imports: set[str] = set() self._contract_aliases: dict[str, str] = {} + self._default_array_order: str | None = None def emit(self, node) -> str: """Emit the supported semantic model passed by the caller.""" - if self._normalize_fortran_public_names and isinstance(node, SemanticModule): + if not isinstance(node, SemanticModule): + return self._visit(node) + previous_default_order = self._default_array_order + self._default_array_order = self._native_default_array_order(node.origin.source_language) + try: + if not self._normalize_fortran_public_names: + return self._visit(node) previous_policy = self._naming_policy previous_namespace = self._public_namespace previous_reserved = self._reserved_public_names @@ -98,7 +105,8 @@ def emit(self, node) -> str: self._naming_policy = previous_policy self._public_namespace = previous_namespace self._reserved_public_names = previous_reserved - return self._visit(node) + finally: + self._default_array_order = previous_default_order @staticmethod def _visit_not_supported(node): @@ -190,13 +198,26 @@ def _emit_function(self, func: SemanticFunction, *, name_owner: object | None = decorator = self._decorators(func, emitted_name=name) return self._emit_callable( name=name, - arguments=[self._emit_call_argument(func, arg) for arg in self._call_arguments(func)], + arguments=[self._emit_contract_argument(func, arg) for arg in self._call_arguments(func)], return_type=return_type, decorator=decorator, def_indent="", parameter_indent=" ", ) + def _emit_contract_argument(self, func: SemanticFunction, arg: SemanticArgument) -> str: + """Omit native facts already implied by the surrounding callable contract.""" + passed_object_name = func.metadata.get("fortran_passed_object_name") + if ( + func.metadata.get("fortran_type_bound_target") + and isinstance(passed_object_name, str) + and arg.name == passed_object_name + and arg.semantic_type.metadata.get("fortran_polymorphic") + ): + arg = deepcopy(arg) + arg.semantic_type.metadata.pop("fortran_polymorphic", None) + return self._emit_call_argument(func, arg) + def _visit_SemanticMethod(self, method: SemanticMethod) -> str: """Emit method syntax.""" return self._emit_method(method) @@ -447,12 +468,16 @@ def _array_annotation_metadata(self, array: SemanticArrayContract | None) -> lis """Handle array annotation metadata for the current generation context.""" if array is None: return [] + if array.rank is None or array.rank <= 1: + return [] metadata: list[str] = [] - if array.order in {"ORDER_F", "ORDER_ANY"} and not ( - array.category == "assumed_size" and array.order == "ORDER_F" + if ( + array.order in {"ORDER_F", "ORDER_ANY"} + and array.order != self._default_array_order + and not (array.category == "assumed_size" and array.order == "ORDER_F") ): metadata.append(self._contract(array.order)) - if array.order == "ORDER_C" and PyiPrinter._is_c_order_flat_array(array): + if array.order == "ORDER_C" and array.order != self._default_array_order: metadata.append(self._contract("ORDER_C")) if array.copy_order == "ORDER_F": metadata.append(self._contract("COPY_F")) @@ -463,15 +488,13 @@ def _array_annotation_metadata(self, array: SemanticArrayContract | None) -> lis return metadata @staticmethod - def _is_c_order_flat_array(array: SemanticArrayContract) -> bool: - """Return whether an assumed-size contract uses leading Flat storage.""" - return ( - array.category == "assumed_size" - and array.rank is not None - and array.rank > 1 - and bool(array.source_shape) - and PyiPrinter._assumed_size_array_dimension(array.source_shape[0]) == _FLAT_DIMENSION_PRINT_SENTINEL - ) + def _native_default_array_order(native_language: str | None) -> str | None: + """Return the implicit multidimensional storage order for one native language.""" + if native_language == "fortran": + return "ORDER_F" + if native_language == "c": + return "ORDER_C" + return "ORDER_F" def _semantic_annotation_metadata(self, semantic_type: SemanticType) -> list[str]: """Handle semantic annotation metadata for the current generation context.""" @@ -633,7 +656,7 @@ def _emit_typed_name( type_text = self._visit(semantic_type) annotation_metadata = [] if original_name is not None: - annotation_metadata.append(f"{self._contract('Name')}({json.dumps(original_name)})") + annotation_metadata.append(f"{self._contract('SourceName')}({json.dumps(original_name)})") if annotation_metadata: type_text = self._annotated_type_text(type_text, annotation_metadata) if self._is_constant(arg.semantic_type): @@ -940,7 +963,10 @@ def _constructor_argument(self, field: SemanticVariable) -> str: or "..." ) if name != field.name: - type_text = self._annotated_type_text(type_text, [f"{self._contract('Name')}({json.dumps(field.name)})"]) + type_text = self._annotated_type_text( + type_text, + [f"{self._contract('SourceName')}({json.dumps(field.name)})"], + ) return f"{name}: {type_text} = {default_value}" @staticmethod From 9d29449204618d66e05b3fa1cec840bbde885933 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 18:08:23 +0100 Subject: [PATCH 23/30] clean compiling --- README.md | 27 +- THIRD_PARTY_NOTICES.md | 12 +- docs/developer/source-map.md | 4 +- .../internal-architecture/pipeline-map.md | 2 +- .../wrapper-plan-migration-checklist.md | 5 +- docs/old_docs/examples.md | 6 +- docs/old_docs/fortran_wrapper.md | 6 +- docs/old_docs/tutorial.md | 6 +- docs/user/guide/fortran-wrapper.md | 29 +- docs/user/reference/cli-commands.md | 8 +- tests/docs/test_structure.py | 2 +- .../test_rendered_wrapper_artifact_build.py | 110 ++- tests/wrapper/CHECKLIST_COVERAGE.md | 2 +- tests/wrapper/fortran/_support.py | 22 +- .../test_contract_package_runtime.py | 4 +- .../build_from_pyi/test_pyi_wrapper_builds.py | 9 +- .../build_from_source/test_build_modes.py | 103 ++- .../test_compiler_verbose.py | 92 +- .../test_multi_source_builds.py | 2 + .../test_phase1a_wrapper_assembly.py | 18 + tools/wrapper_plan_staged_walkthrough.py | 2 +- x2py/cli.py | 23 +- x2py/compiling/README.md | 32 +- x2py/compiling/basic.py | 294 ------- x2py/compiling/compiler_profiles.py | 270 ++++++ x2py/compiling/compilers.py | 809 +++++------------- x2py/compiling/default_compilers.py | 409 --------- x2py/compiling/objects.py | 37 + x2py/compiling/runtime_support.py | 32 +- x2py/pipeline/build.py | 360 +++++--- x2py/stdlib/x2py_runtime/CMakeLists.txt | 11 - x2py/stdlib/x2py_runtime/meson.build | 8 - x2py/wrapper_codegen/c/binding.py | 12 +- x2py/wrapper_codegen/generator.py | 40 +- 34 files changed, 1222 insertions(+), 1586 deletions(-) delete mode 100644 x2py/compiling/basic.py create mode 100644 x2py/compiling/compiler_profiles.py delete mode 100644 x2py/compiling/default_compilers.py create mode 100644 x2py/compiling/objects.py delete mode 100644 x2py/stdlib/x2py_runtime/CMakeLists.txt delete mode 100644 x2py/stdlib/x2py_runtime/meson.build diff --git a/README.md b/README.md index c9e1e1b7a..f0d8e2af1 100644 --- a/README.md +++ b/README.md @@ -47,14 +47,16 @@ Build it with the default output locations: python3 -m x2py scale.f90 ``` -By default, x2py writes the importable `.so` beside the input source and keeps -generated build intermediates under `__x2py__/`: +By default, x2py writes generated build artifacts, including the ABI-suffixed +extension, under `__x2py__/` in the directory where you run the command. A +direct CLI build also writes a stable `.so` import alias there: ```text . scale.f90 scale.so __x2py__/ + scale..so generated-wrapper sources x2py_runtime/ ``` @@ -72,6 +74,7 @@ Expected result: scale.f90 SCALE.so __x2py__/ + SCALE..so generated-wrapper sources x2py_runtime/ ``` @@ -80,8 +83,8 @@ For a wrapper build, `--out SCALE` selects the Python module name and the final shared-library filename. This first example is a standalone procedure, so it is exposed directly at the extension root. -Use `--out-dir` when you want the shared library and generated intermediates in -an explicit build directory: +Use `--out-dir` when you want the ABI-specific shared library and generated +intermediates in an explicit build directory: ```bash python3 -m x2py scale.f90 \ @@ -92,10 +95,12 @@ python3 -m x2py scale.f90 \ Expected result: ```text -build/SCALE/ +. SCALE.so - generated-wrapper sources - x2py_runtime/ + build/SCALE/ + SCALE..so + generated-wrapper sources + x2py_runtime/ ``` Generate the semantic `.pyi` contract for the same source: @@ -148,10 +153,12 @@ Use `--out NAME` with wrapper builds when you want the import name and final The `.pyi` build produces the same importable extension shape: ```text -build/SCALE_from_pyi/ +. SCALE.so - generated-wrapper sources - x2py_runtime/ + build/SCALE_from_pyi/ + SCALE..so + generated-wrapper sources + x2py_runtime/ ``` The direct source build exposes the standalone procedure at the extension root: diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 80e185446..c6a3692fb 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -1,12 +1,10 @@ # Third-Party Notices -The compiler configuration/execution modules `x2py/compiling/basic.py`, -`x2py/compiling/compilers.py`, and `x2py/compiling/default_compilers.py`; the -native NumPy/Python runtime in `x2py/stdlib/x2py_runtime/`; and small naming -utilities in `x2py/utilities/strings.py`, `x2py/utilities/metaclasses.py`, and -`x2py/naming/policy.py` contain code adapted from the Pyccel project. The -current semantic and wrapper source printers are independent implementations -and are not covered by this attribution. +The native NumPy/Python runtime in `x2py/stdlib/x2py_runtime/` and small +naming utilities in `x2py/utilities/strings.py`, `x2py/utilities/metaclasses.py`, +and `x2py/naming/policy.py` contain code adapted from the Pyccel project. The +current compilation package, semantic printer, and wrapper source printers are +independent implementations and are not covered by this attribution. Upstream project: diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index 978f135da..b7958a4b8 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -71,7 +71,7 @@ X2PY_C_DOCS_END --> | `x2py/types/` | Semantic-to-Python ecosystem type mappings | `numpy.py` | `tests/types/test_numpy.py` | | `x2py/parsers/` | Public namespace for language and semantic `.pyi` frontends | child parser packages | `tests/parsing/`, parser references, semantic `.pyi` reference | | `x2py/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | -| `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `basic.py`, `compilers.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | +| `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | | `x2py/stdlib/` | Native runtime support files copied into generated wrapper builds | `x2py_runtime/` | wrapper runtime tests | | `x2py/utilities/` | Small shared Python utilities | `metaclasses.py`, `strings.py` | tests that exercise callers | @@ -115,7 +115,7 @@ update this table, the package README files, and the mechanical checks in | `x2py/wrapper_codegen/c/binding.py` | Direct Python-extension binding lowering from typed plans. | | `x2py/wrapper_codegen/printers/source_printers.py` | Native binding, header, and Fortran source printing. | | `x2py/wrapper_codegen/printers/pyi_printer.py` | Semantic `.pyi` printing. | -| `x2py/compiling/basic.py` | Native compile object model. | +| `x2py/compiling/objects.py` | Native compile object model. | | `x2py/compiling/compilers.py` | Compiler command execution and tool lookup. | | `x2py/compiling/runtime_support.py` | Runtime support installation for generated wrappers. | | `x2py/naming/policy.py` | Public wrapper names and generated target-language symbols. | diff --git a/docs/maintainer/internal-architecture/pipeline-map.md b/docs/maintainer/internal-architecture/pipeline-map.md index 36518ebc2..ce7b0d7e0 100644 --- a/docs/maintainer/internal-architecture/pipeline-map.md +++ b/docs/maintainer/internal-architecture/pipeline-map.md @@ -48,7 +48,7 @@ X2PY_C_DOCS_END --> | Wrapper planning | `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/plan.py` | policy-completed semantic modules | typed wrapper plans consuming completed decisions without re-inferring policy | `tests/wrapper_codegen/`, wrapper tests | | Direct bridge and binding lowering | `x2py/wrapper_codegen/fortran/bridge.py`, `x2py/wrapper_codegen/c/binding.py`, `x2py/wrapper_codegen/generator.py` | validated typed wrapper plans | Fortran, C, and header syntax nodes | `tests/wrapper_codegen/`, wrapper tests | | Wrapper and semantic-contract printing | `x2py/wrapper_codegen/printers/` | wrapper syntax nodes or semantic IR | wrapper source files or semantic `.pyi` text | printer, generated-contract, and wrapper artifact tests | -| Compile and link | `x2py/compiling/` | user objects, wrapper sources, runtime support | shared library | wrapper runtime tests | +| Compile and link | `x2py/compiling/`, `x2py/pipeline/build.py` | explicit native objects, then generated bridge objects, then runtime/binding objects and ordered link inputs | shared library | wrapper runtime and build-mode tests | diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 3202a67dc..17583cce6 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -593,7 +593,7 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 346 | +| `wrapper-plan` | 347 | | `dual-route` | 0 | | `legacy` | 0 | | `not-applicable` | 75 | @@ -692,9 +692,8 @@ already covered by the new generator. | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_reduced_entry_generates_only_reachable_module_variable_bindings` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_scale_runtime_contract[*]` | source/generated-.pyi parity or parametrized route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py::test_source_named_root_discovers_and_builds_module_leaf` | direct wrapper/build route | semantic .pyi generation/parsing; build/compile/link orchestration; scalar inputs/results; module variables/state | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_module_name_does_not_collide_with_root_function` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | -| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_extension_beside_source` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | +| `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_default_places_artifacts_in_invocation_directory` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_fortran_wrapper_out_names_importable_shared_library` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper` | direct wrapper/build route | build/compile/link orchestration; scalar inputs/results | `wrapper-plan` | | `tests/wrapper/fortran/build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | direct wrapper/build route | build/compile/link orchestration | `wrapper-plan` | diff --git a/docs/old_docs/examples.md b/docs/old_docs/examples.md index d2cd61dc3..53e04a29d 100644 --- a/docs/old_docs/examples.md +++ b/docs/old_docs/examples.md @@ -203,8 +203,10 @@ Exact NumPy scalars are part of the native contract. Passing ordinary Python numbers where a specific native dtype is required raises `TypeError` rather than silently changing the ABI conversion. -With no `--out-dir`, x2py writes intermediates under `__x2py__` beside the -first source and writes the extension beside that source. Use `--verbose` to +With no `--out-dir`, x2py writes intermediates and the ABI-suffixed extension +under `__x2py__` in the current working directory, while a direct CLI build +writes its stable `.so` alias there unless `--out` gives it an explicit path. +Use `--verbose` to print the direct compiler and linker commands. Use `--strict-wrapper-names` to reject public names that need Python keyword escaping or collision suffixes. diff --git a/docs/old_docs/fortran_wrapper.md b/docs/old_docs/fortran_wrapper.md index c5c72707a..5a8d1cc1d 100644 --- a/docs/old_docs/fortran_wrapper.md +++ b/docs/old_docs/fortran_wrapper.md @@ -132,8 +132,10 @@ The extension name comes from the first generated semantic module. For a multi-source build, x2py merges the public surface into that extension and compiles sources in caller-supplied order. -Without `--out-dir`, x2py uses a private `__x2py__` build directory beside the -source and places the importable extension beside the source file. Generated +Without `--out-dir`, x2py writes generated artifacts, including the ABI-suffixed +extension, in a private `__x2py__` build directory in the current working +directory. A direct CLI build writes its stable `.so` import alias in +the current working directory unless `--out` gives it an explicit path. Generated Fortran and C wrapper sources remain build artifacts; users do not edit them to change the Python API. diff --git a/docs/old_docs/tutorial.md b/docs/old_docs/tutorial.md index cb053f6a3..75d4b4e44 100644 --- a/docs/old_docs/tutorial.md +++ b/docs/old_docs/tutorial.md @@ -264,8 +264,10 @@ assert value == np.float64(7.5) The exact NumPy scalar types are intentional. The wrapper validates the native ABI contract instead of silently converting arbitrary Python numeric objects. -Without `--out-dir`, intermediate files go into `__x2py__` beside the first -source and the extension is placed beside that source. Use `--verbose` to print +Without `--out-dir`, intermediate files and the ABI-suffixed extension go into +`__x2py__` in the current working directory, while a direct CLI build writes +its stable `.so` alias there unless `--out` gives it an explicit path. Use +`--verbose` to print the executed compiler and linker commands. ### 6. Understand The Generated Boundary diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 91f5f16e2..843b30b3c 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -190,14 +190,18 @@ When a folder contains only standalone BLAS/LAPACK-style procedures, separate artifacts. -Without `--out-dir`, x2py uses a private `__x2py__` build directory beside the -source and places the importable extension beside the source file. Generated +Without `--out-dir`, x2py writes generated artifacts, including the ABI-suffixed +extension, in a private `__x2py__` build directory in the current working +directory. A direct CLI build writes its stable `.so` import alias in +the current working directory unless `--out` gives it an explicit path. Generated wrapper sources remain build artifacts; users do not edit them to change the Python API. @@ -281,10 +285,19 @@ Runtime tests: [`test_pyi_wrapper_builds.py`](../../../tests/wrapper/fortran/bui [`test_policy_dispatch_contracts.py`](../../../tests/wrapper/fortran/edit_pyi_contracts/test_policy_dispatch_contracts.py). Use `--verbose` to execute a build while printing every exact, shell-escaped -compiler and linker command. Verbose builds also print elapsed time for each -compiler/linker command and for the wrapper creation, printing, and compilation -stages. Use `--makefile` to generate an editable `Makefile.x2py` without -compiling. These modes are mutually exclusive. +compiler and linker command. It first announces binding, bridge, and header +source-text generation on separate lines without paths, because those files do +not exist yet. Each line is printed immediately before its separate lowering +and printing operation, followed by `Timing: ...` for that operation. It then announces each written artifact with its output path +(`Write bridge source: ...` and `Write runtime support: ...`), each native, bridge, runtime, and binding compilation with its +source and object path (`Compile bridge source: source -> object`), and the final +extension path before linking (`Create shared library: ...`). The exact +shell-escaped command follows each compilation or link announcement, so it can +be copied to reproduce that step. Verbose builds print elapsed time for policy +completion, each source-text generation, every compilation, +and linking, followed by total build time; writing generated files has no separate +timing. Use `--makefile` to generate an editable +`Makefile.x2py` without compiling. These modes are mutually exclusive. | `--json` | Prints JSON to stdout for inspection stages and wrapper build results. | | `--out [PATH]` | Writes inspection-stage output, selects the generated Fortran `.pyi` package directory, or names the wrapper Python module and final `.so`. | | `--out-dir DIR` | Selects the wrapper build output directory. | -| `--verbose` | Prints wrapper compiler commands, build steps, and elapsed time for each compiler/linker command and wrapper stage. | +| `--verbose` | Announces and completes binding, bridge, and header source-text generation in order, then each written artifact, source/object compilation pair, and final extension path before printing the exact compiler or linker command; it times each non-writing operation and reports total build time last. | | `--wrapper-compiler-debug` | Uses the compiler debug profile for direct wrapper builds instead of the default release profile. | | `--wrapper-fortran-flags FLAG...` | Appends flags to generated Fortran bridge compilation commands. | | `--no-color` | Disables ANSI color in parse diagnostics. | diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index 24b206f3b..62a0d1794 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -246,7 +246,7 @@ "x2py/wrapper_codegen/fortran/bridge.py", "x2py/wrapper_codegen/printers/pyi_printer.py", "x2py/wrapper_codegen/printers/source_printers.py", - "x2py/compiling/basic.py", + "x2py/compiling/objects.py", "x2py/compiling/compilers.py", "x2py/compiling/runtime_support.py", "x2py/naming/policy.py", diff --git a/tests/pipeline/test_rendered_wrapper_artifact_build.py b/tests/pipeline/test_rendered_wrapper_artifact_build.py index d3d795c1d..b4d2ddcfe 100644 --- a/tests/pipeline/test_rendered_wrapper_artifact_build.py +++ b/tests/pipeline/test_rendered_wrapper_artifact_build.py @@ -7,7 +7,7 @@ import pytest from tests._shared.ownership_policy_support import parse_pyi_text -from x2py.compiling.basic import CompileObj +from x2py.compiling.objects import ObjectFile from x2py.pipeline.build import ( NativeBuildPlan, NativeLinkItem, @@ -34,20 +34,45 @@ def __init__(self): self.compiled = [] self.linked = None - def compile_module(self, compile_obj, output_folder, language, verbose): - self.compiled.append((compile_obj, Path(output_folder), language, verbose)) - compile_obj.module_target.parent.mkdir(parents=True, exist_ok=True) - compile_obj.module_target.write_text(f"{language} object\n", encoding="utf-8") - if language == "fortran": - module_file = Path(output_folder) / f"{compile_obj.python_module}.mod" + def compile_object(self, object_file, *, verbose=False): + self.compiled.append((object_file, verbose)) + object_file.object_path.parent.mkdir(parents=True, exist_ok=True) + object_file.object_path.write_text(f"{object_file.language} object\n", encoding="utf-8") + if object_file.language == "fortran": + module_file = object_file.object_path.parent / f"{object_file.source.stem}.mod" module_file.write_text("fortran module\n", encoding="utf-8") - def compile_shared_library(self, compile_obj, output_folder, language, verbose, sharedlib_modname=None): - name = sharedlib_modname or compile_obj.python_module - shared_library = Path(output_folder) / f"{name}.so" + def link_extension( + self, + *, + module_name, + output_dir, + language, + objects, + link_args=(), + library_dirs=(), + libraries=(), + flags=(), + tools=("python",), + verbose=False, + ): + shared_library = Path(output_dir) / f"{module_name}.so" + if verbose: + print(f">> Create shared library: {shared_library}") shared_library.write_text("shared library\n", encoding="utf-8") - self.linked = (compile_obj, Path(output_folder), language, verbose, sharedlib_modname) - return str(shared_library) + self.linked = ( + module_name, + Path(output_dir), + language, + tuple(objects), + tuple(link_args), + tuple(library_dirs), + tuple(libraries), + tuple(flags), + tuple(tools), + verbose, + ) + return shared_library def _module_plan(source: str, *, module_name: str) -> ModulePlan: @@ -60,7 +85,7 @@ def _rendered_artifacts(source: str, *, module_name: str) -> RenderedGeneratedWr return WrapperCodeGenerator().generate(_module_plan(source, module_name=module_name)) -def test_build_rendered_wrapper_extension_writes_compiles_runtime_and_links(tmp_path: Path): +def test_build_rendered_wrapper_extension_writes_compiles_runtime_and_links(tmp_path: Path, capsys): rendered = _rendered_artifacts( """ @bind("SCALE") @@ -73,11 +98,16 @@ def scale(x: Float64) -> Float64: ... native_dir.mkdir() native_source = native_dir / "scale_native.f90" native_source.write_text("native source placeholder\n", encoding="utf-8") - native_obj = CompileObj(native_source.name, native_dir) - native_obj.module_target.write_text("native object\n", encoding="utf-8") + native_obj = ObjectFile( + source=native_source, + object_path=native_dir / "scale_native.o", + language="fortran", + include_dirs=(native_dir,), + ) + native_obj.object_path.write_text("native object\n", encoding="utf-8") native_plan = NativeBuildPlan( - produced_objects=(native_obj.module_target,), - link_items=(NativeLinkItem("object", native_obj.module_target),), + produced_objects=(native_obj.object_path,), + link_items=(NativeLinkItem("object", native_obj.object_path),), module_dirs=(native_dir,), library_dirs=(native_dir,), ) @@ -90,10 +120,11 @@ def scale(x: Float64) -> Float64: ... sources=(Path("scale_contract.pyi"),), native_build_plan=native_plan, native_dependencies=(native_obj,), - native_link_args=(str(native_obj.module_target),), + native_link_args=("-lm",), wrapper_fortran_flags=("-O2",), wrapper_c_flags=("-O3",), compiler=compiler, + verbose=True, ) build_dir = tmp_path / "build" @@ -108,18 +139,26 @@ def scale(x: Float64) -> Float64: ... assert header.read_text(encoding="utf-8") == rendered.sources[2].text assert runtime_source.exists() assert runtime_object.exists() - assert [language for _obj, _output, language, _verbose in compiler.compiled] == ["fortran", "c", "c"] + assert [object_file.language for object_file, _verbose in compiler.compiled] == ["fortran", "c", "c"] bridge_obj = compiler.compiled[0][0] runtime_obj = compiler.compiled[1][0] binding_obj = compiler.compiled[2][0] - assert tuple(bridge_obj.dependencies) == (native_obj,) - assert native_dir in bridge_obj.include - assert tuple(binding_obj.dependencies) == (native_obj, bridge_obj, runtime_obj) - assert binding_obj.link_args == (str(native_obj.module_target),) - assert native_dir in binding_obj.libdir - assert binding_obj.extra_compilation_tools == {"python"} - assert compiler.linked == (binding_obj, tmp_path / "extension", "fortran", False, "plan_scalar_build") + assert native_dir in bridge_obj.include_dirs + assert runtime_obj.tools == frozenset({"python"}) + assert binding_obj.tools == frozenset({"python"}) + assert compiler.linked == ( + "plan_scalar_build", + tmp_path / "extension", + "fortran", + (native_obj, bridge_obj, runtime_obj, binding_obj), + ("-lm",), + (native_dir,), + (), + ("-O3",), + ("python",), + True, + ) assert result.sources == (Path("scale_contract.pyi"),) assert result.module_name == "plan_scalar_build" @@ -130,10 +169,25 @@ def scale(x: Float64) -> Float64: ... assert result.build_makefile is None assert result.native_build_plan == native_plan assert result.generated_sources == (bridge_source, binding_source, header) - assert bridge_obj.module_target in result.generated_files - assert binding_obj.module_target in result.generated_files + assert bridge_obj.object_path in result.generated_files + assert binding_obj.object_path in result.generated_files assert runtime_source in result.generated_files assert runtime_object in result.generated_files + step_lines = [ + line.removeprefix(">> ") + for line in capsys.readouterr().out.splitlines() + if line.startswith(">> ") and not line.startswith(">> Timing") + ] + assert step_lines == [ + f"Write bridge source: {bridge_source}", + f"Write binding source: {binding_source}", + f"Write binding header: {header}", + f"Write runtime support: {build_dir / 'x2py_runtime'}", + f"Compile bridge source: {bridge_source} -> {bridge_obj.object_path}", + f"Compile runtime source: {runtime_source} -> {runtime_obj.object_path}", + f"Compile binding source: {binding_source} -> {binding_obj.object_path}", + f"Create shared library: {result.shared_library}", + ] def test_build_rendered_wrapper_extension_rejects_unknown_runtime_support_key(tmp_path: Path): diff --git a/tests/wrapper/CHECKLIST_COVERAGE.md b/tests/wrapper/CHECKLIST_COVERAGE.md index 31e21941b..2036b1f00 100644 --- a/tests/wrapper/CHECKLIST_COVERAGE.md +++ b/tests/wrapper/CHECKLIST_COVERAGE.md @@ -28,7 +28,7 @@ recorded progression, not in the live ledger. | --- | --- | | Structured extension-level native build plan | `build_from_source/test_build_modes.py::test_source_build_result_records_structured_native_plan`, `build_from_pyi/test_pyi_wrapper_builds.py::test_generated_pyi_fixture_builds_from_native_object_without_source_reparse` | | Ordered link item model across native item kinds | `build_from_source/test_build_modes.py::test_native_link_plan_serializes_interleaved_item_kinds` | -| Lower compiler dependency order preserves caller order | `build_from_source/test_build_modes.py::test_compile_object_dependency_modules_keep_caller_order` | +| Explicit extension link order preserves declared objects and arguments | `build_from_source/test_compiler_verbose.py::test_link_keeps_the_declared_object_and_link_argument_order` | ## Stage 3 — Multi-Source Combined Contract Generation diff --git a/tests/wrapper/fortran/_support.py b/tests/wrapper/fortran/_support.py index fd06881d9..dda8cf6cf 100644 --- a/tests/wrapper/fortran/_support.py +++ b/tests/wrapper/fortran/_support.py @@ -14,9 +14,10 @@ from tests._shared.pyi_fixture_packages import assert_generated_pyi_package_matches_fixture from tests.wrapper.fortran.fmath_cases import fmath_cases from x2py import build_pyi_extension -from x2py.compiling.basic import CompileObj +from x2py.compiling.objects import ObjectFile from x2py.parsers.fortran.parser import parse_fortran_project from x2py.pipeline.build import ( + NativeBuildPlan, _apply_source_python_exports, _build_rendered_wrapper_extension, _fortran_source_for_pipeline, @@ -83,7 +84,7 @@ def _build_and_import(source_template: Path, workdir: Path, expected_generated_s str(workdir), "--json", ] - result = subprocess.run(cmd, capture_output=True, text=True, check=True) + result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=workdir) payload = json.loads(result.stdout) shared_library = Path(payload["shared_library"]) @@ -221,7 +222,12 @@ def _build_source_wrapper_plan_and_import( shutil.copyfile(source_template, source) native_object = _compile_native_object(source, workdir / "native") - native_compile_obj = CompileObj(source.name, native_object.parent) + native_compile_obj = ObjectFile( + source=source, + object_path=native_object, + language="fortran", + include_dirs=(native_object.parent,), + ) parsed = parse_fortran_project( { str(source): _fortran_source_for_pipeline( @@ -237,10 +243,16 @@ def _build_source_wrapper_plan_and_import( plan = WrapperPlanner().build(module) rendered = WrapperCodeGenerator().generate(plan) + native_build_plan = NativeBuildPlan( + produced_objects=(native_object,), + module_dirs=(native_object.parent,), + include_dirs=(native_object.parent,), + ) result = _build_rendered_wrapper_extension( rendered, output_dir=workdir / "wrapper_plan_build", sources=(source,), + native_build_plan=native_build_plan, native_dependencies=(native_compile_obj,), ) module = _import_from_build_dir(result.module_name, result.output_dir) @@ -261,7 +273,7 @@ def _build_text_and_import(source_text: str, filename: str, workdir: Path, expec str(workdir), "--json", ] - result = subprocess.run(cmd, capture_output=True, text=True, check=True) + result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=workdir) payload = json.loads(result.stdout) shared_library = Path(payload["shared_library"]) @@ -293,7 +305,7 @@ def _build_sources_and_import(source_texts: list[tuple[str, str]], workdir: Path str(workdir), "--json", ] - result = subprocess.run(cmd, capture_output=True, text=True, check=True) + result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=workdir) payload = json.loads(result.stdout) module_name = payload["module_name"] diff --git a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py index 293536845..73f50609e 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py +++ b/tests/wrapper/fortran/build_from_pyi/test_contract_package_runtime.py @@ -104,6 +104,8 @@ def _build_contract( cwd: Path | None = None, output_name: str | None = None, ): + invocation_dir = cwd or build_dir + invocation_dir.mkdir(parents=True, exist_ok=True) command = [ sys.executable, "-m", @@ -119,7 +121,7 @@ def _build_contract( ] if output_name is not None: command.extend(("--out", output_name)) - payload = _run_json(command, cwd=cwd) + payload = _run_json(command, cwd=invocation_dir) module = _import_extension(str(payload["module_name"]), build_dir) return module, payload diff --git a/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py b/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py index 73d455e01..67e61b18e 100644 --- a/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py +++ b/tests/wrapper/fortran/build_from_pyi/test_pyi_wrapper_builds.py @@ -97,6 +97,7 @@ def _import_from_build_dir(module_name: str, build_dir: Path): def _build_pyi_cli(pyi_path: Path, native_object: Path, build_dir: Path): + build_dir.mkdir(parents=True, exist_ok=True) cmd = [ sys.executable, "-m", @@ -110,7 +111,7 @@ def _build_pyi_cli(pyi_path: Path, native_object: Path, build_dir: Path): str(build_dir), "--json", ] - result = subprocess.run(cmd, capture_output=True, text=True, check=True) + result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=build_dir) payload = json.loads(result.stdout) return _import_from_build_dir(payload["module_name"], build_dir), payload @@ -213,6 +214,7 @@ def test_pyi_cli_requires_a_native_link_input(tmp_path: Path): def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): native_source = tmp_path / SOURCE.name build_dir = tmp_path / "pyi_build" + build_dir.mkdir() shutil.copyfile(SOURCE, native_source) generated = subprocess.run( @@ -235,6 +237,7 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=build_dir, ) payload = json.loads(generated.stdout) manifest_path = Path(payload["build_manifest"]) @@ -287,6 +290,7 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=build_dir, ) regenerated_payload = json.loads(regenerated.stdout) assert regenerated_payload["compiled"] is False @@ -306,6 +310,7 @@ def test_pyi_makefile_manifest_and_replay_workflows(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=build_dir, ) replayed_payload = json.loads(replayed.stdout) assert replayed_payload["compiled"] is True @@ -424,6 +429,7 @@ def test_generated_pyi_fixture_builds_from_native_object_without_source_reparse( def test_pyi_cli_preserves_explicit_ordered_link_items(tmp_path: Path): native_object = _compile_native_object(SOURCE, tmp_path / "native") build_dir = tmp_path / "pyi_build" + build_dir.mkdir() result = subprocess.run( [ sys.executable, @@ -441,6 +447,7 @@ def test_pyi_cli_preserves_explicit_ordered_link_items(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=build_dir, ) payload = json.loads(result.stdout) native_plan = payload["native_build_plan"] diff --git a/tests/wrapper/fortran/build_from_source/test_build_modes.py b/tests/wrapper/fortran/build_from_source/test_build_modes.py index 7d02acaea..c423b4aef 100644 --- a/tests/wrapper/fortran/build_from_source/test_build_modes.py +++ b/tests/wrapper/fortran/build_from_source/test_build_modes.py @@ -12,7 +12,6 @@ import pytest from tests.wrapper.fortran._support import _assert_fmath_examples, _sole_native_module, wrapper_source -from x2py.compiling.basic import CompileObj from x2py.pipeline.preprocessing import PreprocessingConfig from x2py.pipeline.build import NativeBuildPlan, NativeLinkItem, build_fortran_extension @@ -39,6 +38,7 @@ def test_verbose_mode_prints_full_direct_build_commands(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=tmp_path, ) command_lines = result.stdout.splitlines() @@ -50,11 +50,43 @@ def test_verbose_mode_prints_full_direct_build_commands(tmp_path: Path): assert "-O3" in c_wrapper_parts assert "-DNDEBUG" in c_wrapper_parts assert "-g" not in c_wrapper_parts - assert any("-shared" in line and "verbose_api" in line for line in command_lines) - assert any(line.startswith(">> Command completed in ") for line in command_lines) - assert any(line.startswith(">> Timing :: Wrapper creation: ") for line in command_lines) - assert any(line.startswith(">> Timing :: Wrapper printing: ") for line in command_lines) - assert any(line.startswith(">> Timing :: Wrapper compilation: ") for line in command_lines) + link_command = next(line for line in command_lines if "-shared" in line and "verbose_api" in line) + link_parts = shlex.split(link_command) + link_output = link_parts[link_parts.index("-o") + 1] + step_lines = [ + line.removeprefix(">> ") + for line in command_lines + if line.startswith(">> ") and not line.startswith((">> Timing", ">> Total build time")) + ] + bridge_source = tmp_path / "bind_c_verbose_api_wrapper.f90" + binding_source = tmp_path / "verbose_api_wrapper.c" + header = tmp_path / "verbose_api_wrapper.h" + native_object = tmp_path / "verbose_api.o" + bridge_object = tmp_path / "bind_c_verbose_api_wrapper.o" + binding_object = tmp_path / "verbose_api_wrapper.o" + assert step_lines[:4] == [ + "Complete wrapper policies", + "Generate binding source", + "Generate bridge source", + "Generate binding header", + ] + binding_generation = command_lines.index(">> Generate binding source") + bridge_generation = command_lines.index(">> Generate bridge source") + header_generation = command_lines.index(">> Generate binding header") + assert bridge_generation == binding_generation + 2 + assert header_generation == bridge_generation + 2 + assert command_lines[binding_generation + 1].startswith(">> Timing: ") + assert command_lines[bridge_generation + 1].startswith(">> Timing: ") + assert command_lines[header_generation + 1].startswith(">> Timing: ") + assert f"Compile native source: {source} -> {native_object}" in step_lines + assert f"Write bridge source: {bridge_source}" in step_lines + assert f"Write binding source: {binding_source}" in step_lines + assert f"Write binding header: {header}" in step_lines + assert f"Compile bridge source: {bridge_source} -> {bridge_object}" in step_lines + assert f"Compile binding source: {binding_source} -> {binding_object}" in step_lines + assert f"Create shared library: {link_output}" in step_lines + assert any(line.startswith(">> Timing: ") for line in command_lines) + assert command_lines[-1].startswith(">> Total build time: ") assert "Built extension:" in result.stdout @@ -81,6 +113,7 @@ def test_verbose_mode_prints_custom_wrapper_flags(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=tmp_path, ) command_lines = result.stdout.splitlines() @@ -94,22 +127,51 @@ def test_verbose_mode_prints_custom_wrapper_flags(tmp_path: Path): assert "-O2" in shlex.split(link_command) -def test_fortran_wrapper_default_places_extension_beside_source(tmp_path: Path): - source = tmp_path / DEFAULT_OUTPUT_SOURCE.name +def test_fortran_wrapper_default_places_artifacts_in_invocation_directory(tmp_path: Path): + source_dir = tmp_path / "source" + source_dir.mkdir() + run_dir = tmp_path / "run" + run_dir.mkdir() + source = source_dir / DEFAULT_OUTPUT_SOURCE.name shutil.copyfile(DEFAULT_OUTPUT_SOURCE, source) cmd = [sys.executable, "-m", "x2py", str(source), "--json"] - result = subprocess.run(cmd, capture_output=True, text=True, check=True) + result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=run_dir) payload = json.loads(result.stdout) - build_dir = tmp_path / "__x2py__" + build_dir = run_dir / "__x2py__" shared_library = Path(payload["shared_library"]) - assert shared_library.parent == tmp_path + assert shared_library.parent == run_dir assert shared_library.name == "fdefault_output.so" assert shared_library.exists() assert Path(payload["output_dir"]) == build_dir assert (build_dir / "bind_c_fdefault_output_wrapper.f90").exists() - assert not list(tmp_path.glob("*_wrapper.c")) + assert len(tuple(build_dir.glob("fdefault_output.*.so"))) == 1 + assert not list(source_dir.glob("*_wrapper.c")) + + +def test_fortran_wrapper_out_dir_separates_abi_artifact_from_cli_alias(tmp_path: Path): + source_dir = tmp_path / "source" + source_dir.mkdir() + run_dir = tmp_path / "run" + run_dir.mkdir() + source = source_dir / DEFAULT_OUTPUT_SOURCE.name + shutil.copyfile(DEFAULT_OUTPUT_SOURCE, source) + + result = subprocess.run( + [sys.executable, "-m", "x2py", str(source), "--out-dir", "build", "--json"], + capture_output=True, + text=True, + check=True, + cwd=run_dir, + ) + payload = json.loads(result.stdout) + + build_dir = run_dir / "build" + assert Path(payload["shared_library"]) == run_dir / "fdefault_output.so" + assert (run_dir / "fdefault_output.so").is_file() + assert len(tuple(build_dir.glob("fdefault_output.*.so"))) == 1 + assert not (build_dir / "fdefault_output.so").exists() def test_fortran_wrapper_default_module_name_does_not_collide_with_root_function(tmp_path: Path): @@ -121,6 +183,7 @@ def test_fortran_wrapper_default_module_name_does_not_collide_with_root_function capture_output=True, text=True, check=True, + cwd=tmp_path, ) payload = json.loads(result.stdout) @@ -157,6 +220,7 @@ def test_fortran_wrapper_out_names_importable_shared_library(tmp_path: Path): capture_output=True, text=True, check=True, + cwd=tmp_path, ) payload = json.loads(result.stdout) @@ -164,7 +228,7 @@ def test_fortran_wrapper_out_names_importable_shared_library(tmp_path: Path): assert shared_library == output_name.with_suffix(".so") assert shared_library.is_file() assert payload["module_name"] == "SCALE" - assert any(path.name.startswith("SCALE.") and path.suffix == ".so" for path in tmp_path.iterdir()) + assert any(path.name.startswith("SCALE.") and path.suffix == ".so" for path in (tmp_path / "__x2py__").iterdir()) assert str(shared_library) in payload["generated_files"] sys.modules.pop("SCALE", None) @@ -245,19 +309,6 @@ def test_native_link_plan_serializes_interleaved_item_kinds(): ] -def test_compile_object_dependency_modules_keep_caller_order(tmp_path: Path): - first = CompileObj("first.f90", tmp_path) - archive = CompileObj("libsolver.a", tmp_path) - shared = CompileObj("libsupport.so", tmp_path) - main = CompileObj("wrapper.c", tmp_path, dependencies=(first, archive, shared)) - - assert main.extra_modules == ( - first.module_target, - archive.module_target, - shared.module_target, - ) - - def test_wrapper_build_rejects_empty_source_list(tmp_path: Path): with pytest.raises(ValueError, match="at least one Fortran source"): build_fortran_extension([], output_dir=tmp_path) diff --git a/tests/wrapper/fortran/build_from_source/test_compiler_verbose.py b/tests/wrapper/fortran/build_from_source/test_compiler_verbose.py index 1b7cfbfd6..0cdf604ea 100644 --- a/tests/wrapper/fortran/build_from_source/test_compiler_verbose.py +++ b/tests/wrapper/fortran/build_from_source/test_compiler_verbose.py @@ -1,8 +1,10 @@ -import re import shlex import sys +from pathlib import Path +from x2py.compiling.objects import ObjectFile from x2py.compiling.compilers import Compiler +from x2py.compiling.compiler_profiles import available_compilers, vendors def test_run_command_verbose_prints_replayable_command(capsys): @@ -10,37 +12,97 @@ def test_run_command_verbose_prints_replayable_command(capsys): returned = Compiler.run_command(cmd, verbose=1) - assert returned == cmd + assert returned == tuple(cmd) output = capsys.readouterr().out.splitlines() - assert output[0] == shlex.join(cmd) - assert re.fullmatch(r">> Command completed in \d+\.\d{3}s", output[1]) + assert output == [shlex.join(cmd)] -def test_record_only_compiler_keeps_exact_command_without_executing(monkeypatch): +def test_record_only_compiler_keeps_object_command_without_executing(monkeypatch, tmp_path: Path): compiler = Compiler("GNU", execute_commands=False) + monkeypatch.setattr(compiler, "_executable", lambda _language, _tools: "gcc") monkeypatch.setattr( Compiler, "run_command", lambda *_args, **_kwargs: (_ for _ in ()).throw(AssertionError("command executed")), ) + object_file = ObjectFile( + source=tmp_path / "source.c", + object_path=tmp_path / "source.o", + language="c", + ) - command = ["gfortran", "-O3", "source.f90", "-o", "source.o"] + compiler.compile_object(object_file) - assert compiler._run_or_record_command(command, verbose=0) == command - assert compiler.command_log == (tuple(command),) + command = compiler.command_log[0] + assert command[0] == "gcc" + assert command[-4:] == ("-c", str(object_file.source), "-o", str(object_file.object_path)) -def test_user_compile_flags_are_appended_after_default_profile_flags(): +def test_user_compile_flags_follow_default_profile_flags(monkeypatch, tmp_path: Path): compiler = Compiler("GNU", debug=False, execute_commands=False) - compiler._language_info = compiler._compiler_info["c"] + monkeypatch.setattr(compiler, "_executable", lambda _language, _tools: "gcc") + object_file = ObjectFile( + source=tmp_path / "source.c", + object_path=tmp_path / "source.o", + language="c", + flags=("-O0", "-g0"), + ) - flags = compiler._get_flags(["-O0", "-g0"]) + compiler.compile_object(object_file) - assert flags.index("-O3") < flags.index("-O0") - assert flags.index("-DNDEBUG") < flags.index("-g0") + command = compiler.command_log[0] + assert command.index("-O3") < command.index("-O0") + assert command.index("-DNDEBUG") < command.index("-g0") -def test_python_sysconfig_profile_flags_do_not_override_wrapper_profile(): +def test_python_sysconfig_profile_flags_do_not_override_wrapper_profile(monkeypatch, tmp_path: Path): compiler = Compiler("GNU", debug=False, execute_commands=False) + monkeypatch.setattr(compiler, "_executable", lambda _language, _tools: "gcc") + object_file = ObjectFile( + source=tmp_path / "binding.c", + object_path=tmp_path / "binding.o", + language="c", + tools=frozenset({"python"}), + ) + + compiler.compile_object(object_file) + + command = compiler.command_log[0] + assert command.count("-O3") == 1 + assert command.count("-DNDEBUG") == 1 + assert "-g" not in command + + +def test_link_keeps_the_declared_object_and_link_argument_order(monkeypatch, tmp_path: Path): + compiler = Compiler("GNU", execute_commands=False) + monkeypatch.setattr(compiler, "_executable", lambda _language, _tools: "gfortran") + native = ObjectFile(tmp_path / "native.f90", tmp_path / "native.o", "fortran") + bridge = ObjectFile(tmp_path / "bridge.f90", tmp_path / "bridge.o", "fortran") + binding = ObjectFile(tmp_path / "binding.c", tmp_path / "binding.o", "c", tools=frozenset({"python"})) + + extension = compiler.link_extension( + module_name="wrapped", + output_dir=tmp_path, + language="fortran", + objects=(native, bridge, binding), + link_args=("-Wl,--as-needed", "-lm"), + ) - assert compiler._without_python_profile_flags(["-g", "-O2", "-DNDEBUG", "-Wall"]) == ["-Wall"] + command = compiler.command_log[0] + assert command.index(str(native.object_path)) < command.index(str(bridge.object_path)) + assert command.index(str(bridge.object_path)) < command.index(str(binding.object_path)) + assert command.index(str(binding.object_path)) < command.index("-Wl,--as-needed") < command.index("-lm") + assert command[command.index("-o") + 1] == str(extension) + + +def test_builtin_toolchains_keep_c_and_fortran_stage_definitions(): + assert vendors == ("GNU", "intel", "PGI", "nvidia", "LLVM") + for toolchain in available_compilers.values(): + for language in ("c", "fortran"): + config = toolchain[language] + assert config["exec"] + assert config["debug_flags"] + assert config["release_flags"] + assert config["general_flags"] + assert toolchain["fortran"]["module_output_flag"] + assert toolchain["c"]["python"]["shared_suffix"] diff --git a/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py b/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py index 2a703a0cf..68e1625c3 100644 --- a/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py +++ b/tests/wrapper/fortran/multiple_files/test_multi_source_builds.py @@ -139,6 +139,7 @@ def _import_extension(module_name: str, build_dir: Path): def _build_sources(sources: tuple[Path, ...], build_dir: Path) -> tuple[object, dict[str, object]]: + build_dir.mkdir(parents=True, exist_ok=True) result = subprocess.run( [ sys.executable, @@ -152,6 +153,7 @@ def _build_sources(sources: tuple[Path, ...], build_dir: Path) -> tuple[object, capture_output=True, text=True, check=True, + cwd=build_dir, ) payload = json.loads(result.stdout) return _import_extension(str(payload["module_name"]), build_dir), payload diff --git a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py index 4f9d85475..f84d0a7f2 100644 --- a/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py +++ b/tests/wrapper_codegen/test_phase1a_wrapper_assembly.py @@ -71,6 +71,24 @@ def swap_args(x: Float64, y: Float64) -> Float64: ... assert "result = SWAP_ARGS(y, x)" in fortran_source +def test_public_generator_reports_each_rendering_operation_in_execution_order(): + plan = _plan("def value(x: Float64) -> Float64: ...", module_name="render_progress") + progress = [] + + WrapperCodeGenerator().generate(plan, progress=lambda label, elapsed: progress.append((label, elapsed))) + + assert [label for label, _ in progress] == [ + "Generate binding source", + "Generate binding source", + "Generate bridge source", + "Generate bridge source", + "Generate binding header", + "Generate binding header", + ] + assert [elapsed is None for _, elapsed in progress] == [True, False, True, False, True, False] + assert all(elapsed >= 0.0 for _, elapsed in progress if elapsed is not None) + + @pytest.mark.parametrize( ("source", "c_fragment", "fortran_fragment"), [ diff --git a/tools/wrapper_plan_staged_walkthrough.py b/tools/wrapper_plan_staged_walkthrough.py index d650d46a1..2a85e8396 100644 --- a/tools/wrapper_plan_staged_walkthrough.py +++ b/tools/wrapper_plan_staged_walkthrough.py @@ -139,7 +139,7 @@ def calculate(x: Float64, y: Float64) -> Float64: ... build_dir.mkdir() compiler = pipeline._new_gnu_compiler() native_object = pipeline._source_compile_object(source, build_dir, object_stem="native") -compiler.compile_module(native_object, output_folder=str(build_dir), language="fortran", verbose=False) +compiler.compile_object(native_object, verbose=False) native_build_plan = pipeline._source_native_build_plan((source,), (native_object,), module_dir=build_dir) build = pipeline._build_rendered_wrapper_extension( artifacts, diff --git a/x2py/cli.py b/x2py/cli.py index e7739978e..ae7219b07 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -1287,12 +1287,12 @@ def _cli_wrapper_c_flags(raw_flags: list[str] | None) -> tuple[str, ...]: def _wrapper_shared_library_alias_path(result, raw_out: str | None) -> Path: if raw_out in (None, ""): - return result.shared_library.with_name(f"{result.module_name}.so") + return Path.cwd() / f"{result.module_name}.so" path = Path(raw_out) target = path if path.suffix else path.with_suffix(".so") if not target.is_absolute() and target.parent == Path("."): - return result.shared_library.with_name(target.name) + return Path.cwd() / target.name return target @@ -1397,12 +1397,17 @@ def _run_stage_reports_with_diagnostics(args: argparse.Namespace, preprocessing: def _run_wrap_build(args: argparse.Namespace, preprocessing: PreprocessingConfig): from x2py.pipeline.build import build_fortran_extension, build_pyi_extension, build_pyi_extension_from_manifest + def record_total_build_time(elapsed: float) -> None: + args._verbose_total_build_time = elapsed + + total_build_time_reporter = record_total_build_time if getattr(args, "verbose", False) else None if _wrap_uses_build_manifest(args): result = build_pyi_extension_from_manifest( args.build_manifest, output_name=_wrapper_output_name(args), makefile=getattr(args, "makefile", False), verbose=1 if getattr(args, "verbose", False) else 0, + _on_total_build_time=total_build_time_reporter, ) return _copy_wrapper_shared_library_alias(args, result) @@ -1424,6 +1429,7 @@ def _run_wrap_build(args: argparse.Namespace, preprocessing: PreprocessingConfig wrapper_compiler_debug=getattr(args, "wrapper_compiler_debug", False), wrapper_fortran_flags=_cli_wrapper_fortran_flags(getattr(args, "wrapper_fortran_flags", None)), wrapper_c_flags=_cli_wrapper_c_flags(getattr(args, "wrapper_c_flags", None)), + _on_total_build_time=total_build_time_reporter, ) return _copy_wrapper_shared_library_alias(args, result) @@ -1442,6 +1448,7 @@ def _run_wrap_build(args: argparse.Namespace, preprocessing: PreprocessingConfig wrapper_compiler_debug=getattr(args, "wrapper_compiler_debug", False), wrapper_fortran_flags=_cli_wrapper_fortran_flags(getattr(args, "wrapper_fortran_flags", None)), wrapper_c_flags=_cli_wrapper_c_flags(getattr(args, "wrapper_c_flags", None)), + _on_total_build_time=total_build_time_reporter, ) return _copy_wrapper_shared_library_alias(args, result) @@ -1676,6 +1683,7 @@ def _print_wrap_build_output(args: argparse.Namespace, result) -> None: payload = result.to_dict() if args.json: print(json.dumps(payload, indent=2)) + _print_verbose_total_build_time(args) return if payload.get("compiled", True): @@ -1691,6 +1699,14 @@ def _print_wrap_build_output(args: argparse.Namespace, result) -> None: print("Generated sources:") for path in generated_sources: print(f" - {path}") + _print_verbose_total_build_time(args) + + +def _print_verbose_total_build_time(args: argparse.Namespace) -> None: + """Print the delayed CLI total after its final artifact summary line.""" + elapsed = getattr(args, "_verbose_total_build_time", None) + if elapsed is not None: + print(f">> Total build time: {elapsed:.3f}s") def print_pyi_output(code: str) -> None: @@ -2017,7 +2033,8 @@ def main() -> int: metavar="DIR", help=( "Directory for --wrap generated sources, objects, and extension module; " - "by default build files go in __x2py__ and the extension is written beside the source" + "by default build files and the ABI-suffixed extension go in ./__x2py__, " + "with a stable .so alias in the current directory" ), ) output_group.add_argument("--verbose", action="store_true", help="Print wrapper compiler commands and build steps") diff --git a/x2py/compiling/README.md b/x2py/compiling/README.md index 281e0bd66..deca750cb 100644 --- a/x2py/compiling/README.md +++ b/x2py/compiling/README.md @@ -8,25 +8,37 @@ linking. | File | Owns | | --- | --- | -| `basic.py` | Compile object model and dependency relationships. | +| `objects.py` | Explicit source-to-object compilation inputs. | | `compilers.py` | Compiler command execution and tool lookup helpers. | -| `default_compilers.py` | Default compiler selection helpers. | -| `runtime_support.py` | Copying and compiling x2py runtime support used by generated wrappers. | +| `compiler_profiles.py` | Built-in vendor compiler profiles and Python-link settings. | +| `runtime_support.py` | Writing runtime support and declaring its object inputs. | -Generated-wrapper compile-object assembly and shared-library orchestration live -in `x2py/pipeline/build.py`, where the canonical rendered wrapper artifacts are -available. The compiling package does not import or regenerate wrapper plans. +Generated-wrapper object assembly and shared-library orchestration live in +`x2py/pipeline/build.py`, where the canonical rendered wrapper artifacts are +available. The compiling package does not import or regenerate wrapper plans, +infer semantic policy, or traverse an implicit dependency graph. ## Pipeline Position ```text -generated wrapper source files - -> compile objects and runtime support - -> compiler commands +native source files + -> native object files +generated Fortran bridge + -> bridge object files +generated C/CPython binding and its runtime support + -> runtime and binding object files +all explicit object files and link inputs -> linked Python extension ``` -Compilation should not decide semantic ownership, Python API shape, or wrapper +The bridge and binding sources are rendered together from one completed wrapper +plan before compilation starts. Their object stages remain separate: the bridge +is compiled after native objects so it can consume native module files; runtime +support is compiled before the binding that includes it; and linking runs only +after every required object exists. Each compiler invocation receives its +source, target, flags, includes, and ordered link inputs explicitly. + +Compilation must not decide semantic ownership, Python API shape, or wrapper readiness. Those decisions happen before generated sources reach this package. ## Tests And Docs diff --git a/x2py/compiling/basic.py b/x2py/compiling/basic.py deleted file mode 100644 index b132ef830..000000000 --- a/x2py/compiling/basic.py +++ /dev/null @@ -1,294 +0,0 @@ -#!/usr/bin/python -""" -Module handling classes for compiler information relevant to a given object -""" - -import sys -from pathlib import Path - -from filelock import FileLock - - -class CompileObj: - """ - Class containing all information necessary for compiling. - - A class which stores all information which may be needed in order to - compile an object. This includes its name, location, and all dependencies - and flags which may need to be passed to the compiler. - - Parameters - ---------- - file_name : str - Name of file to be compiled. - - folder : str - Name of the folder where the file is found. - - flags : str - Any non-default flags passed to the compiler. - - include : iterable of strs - Include directories paths. - - libs : iterable of strs - Required libraries. - - libdir : iterable of strs - Paths to directories containing the required libraries. - - dependencies : iterable of CompileObjs - Objects which must also be compiled in order to compile this module/program. - - extra_compilation_tools : iterable of str - Tools used which require additional compilation flags/include dirs/libs/etc. - - has_target_file : bool, default : True - If set to false then this flag indicates that the file has no target. - Eg an interface for a library. - - prog_target : str, default: None - The name of the executable that should be generated if this file is a - program. If no name is provided then the module name deduced from the file - name is used. - """ - - compilation_in_progress = FileLock(".lock_acquisition.lock") - __slots__ = ( - "_dependencies", - "_extra_compilation_tools", - "_file", - "_flags", - "_folder", - "_has_target_file", - "_include", - "_libdir", - "_libs", - "_link_args", - "_lock_source", - "_lock_target", - "_module_name", - "_module_target", - "_prog_target", - ) - - def __init__( - self, - file_name, - folder, - flags=(), - include=(), - libs=(), - libdir=(), - link_args=(), - dependencies=(), - extra_compilation_tools=(), - has_target_file=True, - prog_target=None, - ): - folder = Path(folder) - self._folder = folder - self._file = folder / file_name - - self._module_name = Path(file_name).stem - rel_mod_name = folder / self._module_name - self._module_target = rel_mod_name.with_suffix(".o") - - if prog_target: - self._prog_target = prog_target - else: - self._prog_target = self._module_name - if sys.platform == "win32": - self._prog_target = self._prog_target + ".exe" - - self._lock_target = FileLock(str(self.module_target.with_suffix(self.module_target.suffix + ".lock"))) - self._lock_source = FileLock(str(self.source.with_suffix(self.source.suffix + ".lock"))) - - self._flags = list(flags) - self._include = {*(Path(i) for i in include)} - if has_target_file: - self._include.add(folder) - self._libs = list(libs) - self._libdir = set(libdir) - self._link_args = tuple(str(arg) for arg in link_args) - self._extra_compilation_tools = set(extra_compilation_tools) - self._dependencies = {getattr(a, "module_target", a): a for a in dependencies} - self._has_target_file = has_target_file - - @property - def source(self): - """Returns the file to be compiled""" - return self._file - - @property - def python_module(self): - """Returns the python name of the file to be compiled""" - return self._module_name - - @property - def module_target(self): - """Returns the .o file to be generated by the compilation step""" - return self._module_target - - @property - def program_target(self): - """Returns the program to be generated by the compilation step""" - return self._prog_target - - @property - def flags(self): - """Returns the additional flags required to compile the file""" - return self._flags - - @property - def include(self): - """ - Get the additional include directories required to compile the file. - - Return a set containing all the directories which must be passed to the - compiler via the include flag `-I`. - """ - return self._include.union([di for d in self._dependencies.values() for di in d.include]) - - @property - def libs(self): - """ - Get the additional libraries required to compile the file. - - Return a list containing all the libraries which must be passed to the - compiler via the library flag `-l`. - """ - return self._libs + [dl for d in self._dependencies.values() for dl in d.libs] - - @property - def libdir(self): - """ - Get the additional library directories required to compile the file. - - Return a set containing all the directories which must be passed to the - compiler via the library directory flag `-L` so that the necessary - libraries can be correctly located. - """ - return self._libdir.union([dld for d in self._dependencies.values() for dld in d.libdir]) - - @property - def link_args(self): - """Return ordered raw linker arguments for the final link command.""" - return self._link_args - - @property - def extra_modules(self): - """Returns the additional objects required to compile the file""" - deps = [] - seen = set() - for d in self._dependencies.values(): - if d.has_target_file: - if d.module_target not in seen: - deps.append(d.module_target) - seen.add(d.module_target) - for extra_module in d.extra_modules: - if extra_module not in seen: - deps.append(extra_module) - seen.add(extra_module) - return tuple(deps) - - @property - def dependencies(self): - """Returns the objects which the file to be compiled uses""" - return self._dependencies.values() - - def add_dependencies(self, *args): - """ - Indicate that the file to be compiled depends on a given other file - - Parameters - ---------- - *args : CompileObj - """ - if not all(isinstance(d, CompileObj) for d in args): - raise TypeError("Dependencies require necessary compile information") - self._dependencies.update({a.module_target: a for a in args}) - - def __enter__(self): - self.compilation_in_progress.acquire() - self.acquire_lock() - - def acquire_lock(self): - """ - Lock the file and its dependencies to prevent race conditions. - - Acquire the file locks for the file being compiled, all dependencies needed - to compile it and the target file which will be generated. - """ - self._lock_source.acquire() - self.acquire_simple_lock() - for d in self.dependencies: - d.acquire_simple_lock() - - def acquire_simple_lock(self): - """ - Lock the file created by this `CompileObj`. - - Acquire the file lock for the file created by this `CompileObj` to prevent - race conditions. This function should be called when the created file is a - dependency, it is therefore not necessary for it to recurse into its own - dependencies. - """ - if self.has_target_file: - self._lock_target.acquire() - - def __exit__(self, _exc_type, value, _traceback): - self.release_lock() - self.compilation_in_progress.release() - - def release_lock(self): - """ - Unlock the file and its dependencies. - - Release the file locks for the file being compiled, all dependencies needed - to compile it and the target file which will be generated. - """ - for d in self.dependencies: - d.release_simple_lock() - self._lock_source.release() - self.release_simple_lock() - - def release_simple_lock(self): - """ - Unlock the file created by this `CompileObj`. - - Release the file lock for the file created by this `CompileObj` to prevent - race conditions. This function should be called when the created file is a - dependency, it is therefore not necessary for it to recurse into its own - dependencies. - """ - if self.has_target_file: - self._lock_target.release() - - @property - def extra_compilation_tools(self): - """ - The name of tools used which require additional compilation information. - - Return a set containing the name of all tools required additional - information to compile the file. This additional informationcan take the - form of flags, include directories, libraries, orr library directories. - Examples of 'extra_compilation_tools' are: openmp, openacc, python. - """ - return self._extra_compilation_tools.union( - [da for d in self._dependencies.values() for da in d.extra_compilation_tools] - ) - - def __eq__(self, other): - return self.module_target == other.module_target - - def __hash__(self): - return hash(self.module_target) - - @property - def has_target_file(self): - """ - Indicates whether the file has a target. - Eg an interface for a library may not have a target - """ - return self._has_target_file diff --git a/x2py/compiling/compiler_profiles.py b/x2py/compiling/compiler_profiles.py new file mode 100644 index 000000000..e191094b9 --- /dev/null +++ b/x2py/compiling/compiler_profiles.py @@ -0,0 +1,270 @@ +"""Built-in compiler profiles for the explicit wrapper build stages.""" + +from __future__ import annotations + +from copy import deepcopy +from pathlib import Path +import sys +import sysconfig + +from numpy import get_include as numpy_include + + +def _words(value: object) -> tuple[str, ...]: + """Return a configuration variable as command-line words.""" + return tuple(str(value or "").split()) + + +def _python_library_candidates(config: dict[str, object]) -> tuple[Path, ...]: + """Find Python library files advertised by the active interpreter.""" + libdir = config.get("LIBDIR") + version = config.get("VERSION") + if not isinstance(libdir, str) or not libdir or not isinstance(version, str) or not version: + return () + directory = Path(libdir) + if not directory.is_dir(): + return () + return tuple(sorted(directory.glob(f"libpython{version}*"))) + + +def _python_library_name(config: dict[str, object]) -> str | None: + """Return the fallback ``-l`` name when no interpreter file is available.""" + library = str(config.get("LDLIBRARY") or config.get("LIBRARY") or "") + if library.startswith("lib"): + return Path(library).stem.removeprefix("lib") + return None + + +def _python_build_settings() -> dict[str, object]: + """Collect the active interpreter's headers, extension suffix, and link input.""" + config = dict(sysconfig.get_config_vars()) + include_dirs = [numpy_include()] + include = config.get("INCLUDEPY") + if isinstance(include, str) and include: + include_dirs.append(include) + + python_settings: dict[str, object] = { + "flags": (*_words(config.get("CFLAGS")), *_words(config.get("CC"))[1:]), + "include": tuple(include_dirs), + "shared_suffix": str(config.get("EXT_SUFFIX") or ".so"), + } + settings: dict[str, object] = {"libs": _words(config.get("LIBM")), "python": python_settings} + candidates = _python_library_candidates(config) + shared_suffixes = (".dylib", ".dll") if sys.platform in {"darwin", "win32"} else (".so",) + shared = tuple(path for path in candidates if any(suffix in path.name for suffix in shared_suffixes)) + static = tuple(path for path in candidates if path.suffix == ".a") + preferred = shared or static + if preferred: + exact = tuple(path for path in preferred if path.suffix in shared_suffixes or path.suffix == ".a") + library = exact[0] if exact else preferred[0] + python_settings["dependencies"] = (str(library),) + python_settings["libdir"] = (str(library.parent),) + return settings + + name = _python_library_name(config) + if name: + python_settings["libs"] = (name,) + libdir = config.get("LIBDIR") + if isinstance(libdir, str) and libdir: + python_settings["libdir"] = (libdir,) + return settings + + +def _language( + executable: str, + mpi_executable: str, + *, + debug_flags: tuple[str, ...], + release_flags: tuple[str, ...], + general_flags: tuple[str, ...], + standard_flags: tuple[str, ...], + module_output_flag: str | None = None, + openmp: dict[str, tuple[str, ...]] | None = None, + openacc: dict[str, tuple[str, ...]] | None = None, +) -> dict[str, object]: + """Create one language entry without mixing it with build orchestration.""" + entry: dict[str, object] = { + "exec": executable, + "mpi_exec": mpi_executable, + "debug_flags": debug_flags, + "release_flags": release_flags, + "general_flags": general_flags, + "standard_flags": standard_flags, + "mpi": {}, + "openmp": openmp or {}, + "openacc": openacc or {}, + } + if module_output_flag is not None: + entry["module_output_flag"] = module_output_flag + return entry + + +_GNU_C = _language( + "gcc", + "mpicc", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops", "-DNDEBUG"), + general_flags=("-fPIC",), + standard_flags=("-std=c99",), + openmp={"flags": ("-fopenmp",), "libs": ("gomp",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) +_GNU_CXX = _language( + "g++", + "mpic++", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops"), + general_flags=("-fPIC",), + standard_flags=("--std=c++20",), + openmp={"flags": ("-fopenmp",), "libs": ("gomp",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) +_GNU_FORTRAN = _language( + "gfortran", + "mpif90", + debug_flags=("-fcheck=bounds", "-g", "-O0"), + release_flags=("-O3", "-funroll-loops", "-DNDEBUG"), + general_flags=("-fPIC", "-cpp"), + standard_flags=("-std=f2003",), + module_output_flag="-J", + openmp={"flags": ("-fopenmp",), "libs": ("gomp",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) + +_INTEL_C = _language( + "icx", + "mpiicx", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops", "-DNDEBUG"), + general_flags=("-fPIC",), + standard_flags=("-std=c99",), + openmp={"flags": ("-qopenmp",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) +_INTEL_CXX = _language( + "icpx", + "mpiicpx", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops"), + general_flags=("-fPIC",), + standard_flags=("--std=c++20",), + openmp={"flags": ("-qopenmp",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) +_INTEL_FORTRAN = _language( + "ifx", + "mpiifx", + debug_flags=("-check", "bounds", "-g", "-O0"), + release_flags=("-O3", "-funroll-loops", "-DNDEBUG"), + general_flags=("-fPIC", "-fpp"), + standard_flags=("-std=f2003",), + module_output_flag="-module", + openmp={"flags": ("-qopenmp", "-nostandard-realloc-lhs"), "libs": ("iomp5",)}, + openacc={"flags": ("-ta=multicore", "-Minfo=accel")}, +) + +_PGI_C = _language( + "pgcc", + "pgcc", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-Munroll", "-DNDEBUG"), + general_flags=("-fPIC",), + standard_flags=("-std=c99",), + openmp={"flags": ("-mp",)}, + openacc={"flags": ("-acc",)}, +) +_PGI_FORTRAN = _language( + "pgfortran", + "pgfortran", + debug_flags=("-Mbounds", "-g", "-O0"), + release_flags=("-O3", "-Munroll", "-DNDEBUG"), + general_flags=("-fPIC", "-cpp"), + standard_flags=("-Mstandard",), + module_output_flag="-module", + openmp={"flags": ("-mp",)}, + openacc={"flags": ("-acc",)}, +) + +_NVIDIA_C = _language( + "nvc", + "mpicc", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-Munroll", "-DNDEBUG"), + general_flags=("-fPIC",), + standard_flags=("-std=c99",), + openmp={"flags": ("-mp",)}, + openacc={"flags": ("-acc",)}, +) +_NVIDIA_CXX = _language( + "nvc++", + "mpic++", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-Munroll"), + general_flags=("-fPIC",), + standard_flags=("--std=c++20",), + openmp={"flags": ("-mp",)}, + openacc={"flags": ("-acc",)}, +) +_NVIDIA_FORTRAN = _language( + "nvfortran", + "mpifort", + debug_flags=("-Mbounds", "-g", "-O0"), + release_flags=("-O3", "-Munroll", "-DNDEBUG"), + general_flags=("-fPIC", "-cpp"), + standard_flags=("-Mstandard",), + module_output_flag="-module", + openmp={"flags": ("-mp",)}, + openacc={"flags": ("-acc",)}, +) + +_CLANG_OPENMP = {"flags": ("-fopenmp",)} +if sys.platform == "darwin": + _CLANG_OPENMP = {"flags": ("-Xpreprocessor", "-fopenmp"), "libs": ("omp",)} +_LLVM_C = _language( + "clang", + "mpicc", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops", "-DNDEBUG"), + general_flags=("-fPIC",), + standard_flags=("-std=c99",), + openmp=_CLANG_OPENMP, + openacc={"flags": ("-fopenacc",)}, +) +_LLVM_CXX = _language( + "clang++", + "mpic++", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-funroll-loops"), + general_flags=("-fPIC",), + standard_flags=("--std=c++20",), + openmp=_CLANG_OPENMP, + openacc={"flags": ("-fopenacc",)}, +) +_LLVM_FORTRAN = _language( + "flang", + "mpifort", + debug_flags=("-g", "-O0"), + release_flags=("-O3", "-DNDEBUG"), + general_flags=("-fPIC", "-cpp"), + standard_flags=("-std=f2003",), + module_output_flag="-J", + openmp=_CLANG_OPENMP, + openacc={"flags": ("-fopenacc",)}, +) + + +def _toolchain(**languages: dict[str, object]) -> dict[str, dict[str, object]]: + """Attach active-Python build settings to independent language entries.""" + python_settings = _python_build_settings() + return {name: {**deepcopy(language), **deepcopy(python_settings)} for name, language in languages.items()} + + +available_compilers = { + "GNU": _toolchain(c=_GNU_C, **{"c++": _GNU_CXX}, fortran=_GNU_FORTRAN), + "intel": _toolchain(c=_INTEL_C, **{"c++": _INTEL_CXX}, fortran=_INTEL_FORTRAN), + "PGI": _toolchain(c=_PGI_C, fortran=_PGI_FORTRAN), + "nvidia": _toolchain(c=_NVIDIA_C, **{"c++": _NVIDIA_CXX}, fortran=_NVIDIA_FORTRAN), + "LLVM": _toolchain(c=_LLVM_C, **{"c++": _LLVM_CXX}, fortran=_LLVM_FORTRAN), +} + +vendors = tuple(available_compilers) diff --git a/x2py/compiling/compilers.py b/x2py/compiling/compilers.py index d55bac56e..d24b01623 100644 --- a/x2py/compiling/compilers.py +++ b/x2py/compiling/compilers.py @@ -1,610 +1,271 @@ -#!/usr/bin/python -""" -Module handling everything related to the compilers used to compile the various generated files -""" +"""Run explicit native object and extension-link compiler commands.""" +from __future__ import annotations + +from collections.abc import Iterable, Mapping import json import os -import pathlib -import platform +from pathlib import Path import shlex import shutil import subprocess -import time import warnings -from .default_compilers import available_compilers, vendors - -if platform.system() == "Darwin": - # Collect version using mac tools to avoid unexpected results on Big Sur - # https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes#Third-Party-Apps - with subprocess.Popen([shutil.which("sw_vers"), "-productVersion"], stdout=subprocess.PIPE) as p: - result, err = p.communicate() - mac_version_tuple = result.decode("utf-8").strip().split(".") - mac_target = ".".join(mac_version_tuple[:2]) - os.environ["MACOSX_DEPLOYMENT_TARGET"] = mac_target - - -def get_condaless_search_path(conda_warnings="basic"): - """ - Get a list of paths excluding the conda paths. - - Get the value of the PATH variable to be set when searching for the compiler - This is the same as the environment PATH variable but without any conda paths. - - Parameters - ---------- - conda_warnings : str, optional - Specify the level of Conda warnings to display (choices: off, basic, verbose), Default is 'basic'. - - Returns - ------- - str - A list of paths excluding the conda paths. - """ - path_sep = ";" if platform.system() == "Windows" else ":" - current_path = os.environ["PATH"] - folders = {f: f.split(os.sep) for f in current_path.split(path_sep)} - conda_folder_names = ( - "conda", - "anaconda", - "miniconda", - "Conda", - "Anaconda", - "Miniconda", - ) - conda_folders = [p for p, f in folders.items() if any(con in f for con in conda_folder_names)] - if conda_folders and conda_warnings in ("basic", "verbose"): - message_warning = "Conda paths are ignored. See https://github.com/x2py/x2py/blob/devel/docs/compiler.md#utilising-x2py-within-anaconda-environment for details" +from .objects import ObjectFile +from .compiler_profiles import available_compilers, vendors + +__all__ = ("Compiler", "get_condaless_search_path") + + +def get_condaless_search_path(conda_warnings: str = "basic") -> str: + """Return ``PATH`` without Conda-managed entries when locating compilers.""" + + path_separator = os.pathsep + entries = tuple(entry for entry in os.environ.get("PATH", "").split(path_separator) if entry) + conda_markers = ("conda", "anaconda", "miniconda") + conda_entries = tuple(entry for entry in entries if any(marker in Path(entry).parts for marker in conda_markers)) + if conda_entries and conda_warnings in {"basic", "verbose"}: + message = "Conda paths are ignored while locating native compilers." if conda_warnings == "verbose": - message_warning = message_warning + "\nConda ignored PATH:\n" - message_warning = message_warning + ":".join(conda_folders) - warnings.warn(UserWarning(message_warning), stacklevel=2) - return path_sep.join(p for p in folders if p not in conda_folders and os.path.exists(p)) + message += "\nIgnored PATH entries:\n" + "\n".join(conda_entries) + warnings.warn(message, stacklevel=2) + return path_separator.join(entry for entry in entries if entry not in conda_entries) -# ------------------------------------------------------------ class Compiler: - """ - Class which handles all compiler options. - - This class uses the compiler vendor or a json file to collect - all compiler configuration parameters. These are then used to - correctly print compiler commands such as shared library - compilation commands or executable creation commands. - - Parameters - ---------- - vendor : str - Name of the family of compilers. - debug : bool - Indicates whether we are compiling in debug mode. - execute_commands : bool - Execute prepared commands immediately. If false, retain them in - ``command_log`` for an external build system. - """ - - __slots__ = ( - "_command_log", - "_compiler_family", - "_compiler_info", - "_debug", - "_execute_commands", - "_language_info", - ) - acceptable_bin_paths = None - - def __init__(self, vendor: str, debug=False, *, execute_commands=True): - if vendor.endswith(".json") and os.path.exists(vendor): - self._compiler_family = pathlib.Path(vendor).stem - with open(vendor, encoding="utf-8") as vendor_file: - self._compiler_info = json.load(vendor_file) - else: - self._compiler_family = vendor - if vendor in vendors: - try: - self._compiler_info = available_compilers[vendor] - except KeyError as e: - raise NotImplementedError("Compiler not available") from e - else: - installed_compiler = ( - pathlib.Path(os.environ.get("X2PY_CONFIG_HOME", pathlib.Path.home() / ".x2py")) / vendor - ) - if installed_compiler.exists(): - with open(installed_compiler / "config.json", encoding="utf-8") as vendor_file: - self._compiler_info = json.load(vendor_file) - else: - raise NotImplementedError(f"Unrecognised compiler vendor : {vendor}") - + """Compile explicit object files and link an explicit extension object list.""" + + def __init__( + self, + vendor: str, + debug: bool = False, + *, + execute_commands: bool = True, + search_path: str | None = None, + ) -> None: + self._toolchain = self._load_toolchain(vendor) self._debug = debug self._execute_commands = execute_commands - self._command_log = [] - self._language_info = None + self._search_path = search_path + self._command_log: list[tuple[str, ...]] = [] @property - def command_log(self): - """Exact expanded compiler commands prepared by this instance.""" - return tuple(tuple(command) for command in self._command_log) - - def _run_or_record_command(self, cmd, verbose): - expanded = [os.path.expandvars(str(part)) for part in cmd] - self._command_log.append(expanded) - if self._execute_commands: - return self.run_command(expanded, verbose) - return expanded - - def get_exec(self, extra_compilation_tools, language=None): - """ - Obtain the path of the executable based on the specified compilation tools. - - The `get_exec` method is responsible for retrieving the path of the executable based on - the specified compilation tools. It is used internally in the X2py module. In particular - the executable depends on whether MPI is used. - - Parameters - ---------- - extra_compilation_tools : str - Specifies the compilation tools to be used. - language : str, optional - The language being compiled. This argument should be provided unless this method - is called from a method of this class after setting self._language_info. - - Returns - ------- - str - The path of the executable corresponding to the specified compilation tools. - - Raises - ------ - X2pyError - If the compiler executable cannot be found. - """ - language_info = self._language_info if language is None else self._compiler_info[language] - # Get executable - exec_cmd = language_info["mpi_exec"] if "mpi" in extra_compilation_tools else language_info["exec"] - - # Clean conda paths out of the PATH variable - current_path = os.environ["PATH"] - os.environ["PATH"] = self.acceptable_bin_paths - - # Find the exact path of the executable - exec_loc = shutil.which(exec_cmd) - - # Reset PATH variable - os.environ["PATH"] = current_path - - if exec_loc is None: - raise FileNotFoundError(f"Could not find compiler ({exec_cmd})") - - return exec_loc - - def _get_flags(self, flags=(), extra_compilation_tools=()): - """ - Collect necessary compile flags. - - Collect necessary compile flags, e.g. those relevant to the - language or compilation mode (debug/release). - - Parameters - ---------- - flags : iterable of str - Any additional flags requested by the user / required by - the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - list[str] - A list containing the flags. - """ - user_flags = list(flags) - flags = [] - - if self._debug: - flags.extend(self._language_info.get("debug_flags", ())) - else: - flags.extend(self._language_info.get("release_flags", ())) - - flags.extend(self._language_info.get("general_flags", ())) - # M_PI is not in the standard - # if 'python' not in extra_compilation_tools: - # # Python sets its own standard - # flags.extend(self._language_info.get('standard_flags',())) - - for a in extra_compilation_tools: - tool_flags = self._language_info.get(a, {}).get("flags", ()) - if a == "python": - tool_flags = self._without_python_profile_flags(tool_flags) - flags.extend(tool_flags) - - flags.extend(user_flags) - - return flags - - @staticmethod - def _without_python_profile_flags(flags): - """Drop Python sysconfig optimization/debug flags in favor of x2py's profile.""" - return [flag for flag in flags if not (flag.startswith("-O") or flag.startswith("-g") or flag == "-DNDEBUG")] - - def _get_property(self, key, properties=(), extra_compilation_tools=()): - """ - Collect necessary compile property. - - Collect necessary compile properties such as include folders - or library directories. - - Parameters - ---------- - key : str - A key describing the property of interest. - properties : iterable of str - Any additional values of the property requested by the - user / required by the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - iterable[str] - An iterable containing the relevant information from the - requested property. - - Examples - -------- - >> self._get_property("libs", ("-lmy_lib",), ()) - dict_keys(['-lmy_lib', '-lm']) - - >> self._get_property("libs", ("-lmy_lib",), ("openmp",)) - dict_keys(['-lmy_lib', '-lm', 'gomp']) - - >> self._get_property("include", ("/home/user/homemade-install-dir/",), ("mpi",)) - dict_keys(['/home/user/homemade-install-dir/']) - """ - # Use a dictionary instead of a set to ensure properties are ordered by insertion - # The keys of the dictionary contain the values for the property of interest. - properties = dict.fromkeys(properties) - - properties.update(dict.fromkeys(self._language_info.get(key, ()))) - - for a in extra_compilation_tools: - properties.update(dict.fromkeys(self._language_info.get(a, {}).get(key, ()))) - - return properties.keys() - - def _get_include(self, include=(), extra_compilation_tools=()): - """ - Collect necessary compile include directories. - - Collect necessary compile include directories. - - Parameters - ---------- - include : iterable of str - Any additional include directories requested by the user - / required by the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - list[str] - A list of the include folders. - """ - return self._get_property("include", include, extra_compilation_tools) - - def _get_libs(self, libs=(), extra_compilation_tools=()): - """ - Collect necessary compile libraries. - - Collect necessary compile libraries. - - Parameters - ---------- - libs : iterable of str - Any additional libraries requested by the user / required - by the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - list[str] - A list of the libraries. - """ - return self._get_property("libs", libs, extra_compilation_tools) - - def _get_libdir(self, libdir=(), extra_compilation_tools=()): - """ - Collect necessary compile library directories. - - Collect necessary compile library directories. - - Parameters - ---------- - libdir : iterable of str - Any additional library directories requested by the user - / required by the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - list[str] - A list of the folders containing libraries. - """ - return self._get_property("libdir", libdir, extra_compilation_tools) - - def _get_dependencies(self, dependencies=(), extra_compilation_tools=()): - """ - Collect necessary dependencies. - - Collect necessary object or static libraries that should be included to compile - this object. - - Parameters - ---------- - dependencies : iterable of str - Any additional dependencies required by the file. - extra_compilation_tools : iterable or str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - list[str] - A list of the necessary dependencies. - """ - return self._get_property("dependencies", dependencies, extra_compilation_tools) - - @staticmethod - def _insert_prefix_to_list(lst, prefix): - """ - Add a prefix into a list. - - Add a prefix into a list. E.g: - >>> lst = [1, 2, 3] - >>> _insert_prefix_to_list(lst, 'num:') - ['num:', 1, 'num:', 2, 'num:', 3] - - Parameters - ---------- - lst : iterable - This sequence is copied to a new list with `prefix` before each element. - prefix : Any - The prefix to be placed before each element of `lst`. - - Returns - ------- - list - The list with the prefix inserted. - """ - lst = [(prefix, i) for i in lst] - return [f for fi in lst for f in fi] - - def _get_compile_components(self, compile_obj, extra_compilation_tools=()): - """ - Provide all components required for compiling. - - Provide all the different components (include directories, libraries, etc) - which are needed in order to compile any file. - - Parameters - ---------- - compile_obj : CompileObj - Object containing all information about the object to be compiled. - extra_compilation_tools : iterable of str - Tools used which require additional compilation flags/include dirs/libs/etc. - - Returns - ------- - exec_cmd : str - The command required to run the executable. - inc_flags : iterable of strs - The include directories required to compile. - libs_flags : iterable of strs - The libraries required to compile. - libdir_flags : iterable of strs - The directories containing libraries required to compile. - m_code : iterable of strs - The objects required to compile. - """ - - # get include - include = self._get_include(compile_obj.include, extra_compilation_tools) - inc_flags = self._insert_prefix_to_list(include, "-I") - - # Get dependencies (.o/.a) - m_code = self._get_dependencies(compile_obj.extra_modules, extra_compilation_tools) - - # Get libraries and library directories - libs = self._get_libs(compile_obj.libs, extra_compilation_tools) - libs_flags = [s if s.startswith("-l") else f"-l{s}" for s in libs] - libdir = self._get_libdir(compile_obj.libdir, extra_compilation_tools) - libdir_flags = self._insert_prefix_to_list(libdir, "-L") - - exec_cmd = self.get_exec(extra_compilation_tools) - - return exec_cmd, inc_flags, libs_flags, libdir_flags, m_code - - def compile_module(self, compile_obj, output_folder, language, verbose): - """ - Compile a module. - - Compile a file containing a module to a .o file. - - Parameters - ---------- - compile_obj : CompileObj - Object containing all information about the object to be compiled. - - output_folder : str - The folder where the result should be saved. - - language : str - Language that we are compiling. - - verbose : int - Indicates the level of verbosity. - """ - if not compile_obj.has_target_file: - return - + def command_log(self) -> tuple[tuple[str, ...], ...]: + """Return exact compiler commands in execution order.""" + + return tuple(self._command_log) + + def compile_object(self, object_file: ObjectFile, *, verbose: bool | int = False) -> None: + """Compile exactly one source file into its declared object path.""" + + object_file.object_path.parent.mkdir(parents=True, exist_ok=True) + language = self._language(object_file.language) + command = [ + self._executable(language, object_file.tools), + *self._flags(language, object_file.tools, object_file.flags), + "-c", + *self._path_flags("-I", self._include_dirs(language, object_file.tools, object_file.include_dirs)), + str(object_file.source), + "-o", + str(object_file.object_path), + ] + if object_file.language == "fortran": + command.extend((str(language["module_output_flag"]), str(object_file.object_path.parent))) + self._run_or_record(command, verbose) + + def link_extension( + self, + *, + module_name: str, + output_dir: str | Path, + language: str, + objects: Iterable[ObjectFile | str | Path], + link_args: Iterable[str] = (), + library_dirs: Iterable[str | Path] = (), + libraries: Iterable[str] = (), + flags: Iterable[str] = (), + tools: Iterable[str] = ("python",), + verbose: bool | int = False, + ) -> Path: + """Link the supplied objects and ordered native link arguments once.""" + + object_items = tuple(objects) + object_paths = tuple(item.object_path if isinstance(item, ObjectFile) else Path(item) for item in object_items) + if not object_paths: + raise ValueError("Extension linking requires at least one object file") + + output_path = Path(output_dir) + output_path.mkdir(parents=True, exist_ok=True) + language_info = self._language(language) + object_files = tuple(item for item in object_items if isinstance(item, ObjectFile)) + selected_tools = {str(tool) for tool in tools} + selected_tools.update(tool for item in object_files for tool in item.tools) + selected_tools.add("python") + selected_library_dirs = self._ordered_paths( + (*library_dirs, *(directory for item in object_files for directory in item.library_dirs)) + ) + selected_libraries = self._ordered_strings( + (*libraries, *(library for item in object_files for library in item.libraries)) + ) + resolved_library_dirs = self._library_dirs(language_info, selected_tools, selected_library_dirs) + extension_path = output_path / f"{module_name}{language_info['python']['shared_suffix']}" if verbose: - print(">> Compiling :: ", compile_obj.module_target) - - self._language_info = self._compiler_info[language] - - extra_compilation_tools = compile_obj.extra_compilation_tools - - # Get flags - flags = self._get_flags(compile_obj.flags, extra_compilation_tools) - flags.append("-c") - - # Get include - include = self._get_include(compile_obj.include, extra_compilation_tools) - inc_flags = self._insert_prefix_to_list(include, "-I") - - # Get executable - exec_cmd = self.get_exec(extra_compilation_tools) - - j_code = (self._language_info["module_output_flag"], output_folder) if language == "fortran" else () - - cmd = [ - exec_cmd, - *flags, - *inc_flags, - compile_obj.source, + print(f">> Create shared library: {extension_path}") + command = [ + self._executable(language_info, selected_tools), + "-shared", + *self._flags(language_info, selected_tools - {"python"}, flags), + *self._path_flags("-L", resolved_library_dirs), + *self._path_flags("-Wl,-rpath", resolved_library_dirs), + *(str(path) for path in object_paths), + *(str(argument) for argument in link_args), + *self._tool_values(language_info, "dependencies", selected_tools), "-o", - compile_obj.module_target, - *j_code, + str(extension_path), + *self._library_flags(self._libraries(language_info, selected_tools, selected_libraries)), ] + self._run_or_record(command, verbose) + return extension_path - with compile_obj: - self._run_or_record_command(cmd, verbose) - - self._language_info = None - - def compile_shared_library(self, compile_obj, output_folder, language, verbose, sharedlib_modname=None): - """ - Compile a module to a shared library. - - Compile a file containing a module with C-API calls to a shared library which can - be called from Python. - - Parameters - ---------- - compile_obj : CompileObj - Object containing all information about the object to be compiled. - - output_folder : str - The folder where the result should be saved. - - language : str - Language that we are compiling. - - verbose : int - Indicates the level of verbosity. - - sharedlib_modname : str, optional - The name of the library that should be generated. If none is provided then it - defaults to matching the name of the file. - - Returns - ------- - str - Generated library name. - """ - self._language_info = self._compiler_info[language] + @staticmethod + def run_command(command: Iterable[str], verbose: bool | int = False) -> tuple[str, ...]: + """Run one argv command and raise a concise error when it fails.""" - # Ensure python options are collected - extra_compilation_tools = set(compile_obj.extra_compilation_tools) + expanded = tuple(os.path.expandvars(str(part)) for part in command) + if verbose: + print(shlex.join(expanded)) + completed = subprocess.run(expanded, capture_output=True, text=True, check=False) + if verbose and completed.stdout: + print(completed.stdout, end="" if completed.stdout.endswith("\n") else "\n") + if completed.returncode: + raise RuntimeError(f"Native compiler command failed:\n{completed.stderr}") + if completed.stderr: + warnings.warn(completed.stderr, stacklevel=2) + return expanded - extra_compilation_tools.remove("python") + def _run_or_record(self, command: Iterable[str], verbose: bool | int) -> tuple[str, ...]: + expanded = tuple(os.path.expandvars(str(part)) for part in command) + self._command_log.append(expanded) + if self._execute_commands: + return self.run_command(expanded, verbose) + return expanded - # get flags - flags = self._get_flags(compile_obj.flags, extra_compilation_tools) + def _load_toolchain(self, vendor: str) -> Mapping[str, Mapping[str, object]]: + configured_path = Path(vendor) + if configured_path.suffix == ".json" and configured_path.is_file(): + return self._read_toolchain(configured_path) + if vendor in vendors: + return available_compilers[vendor] + installed_path = Path(os.environ.get("X2PY_CONFIG_HOME", Path.home() / ".x2py")) / vendor / "config.json" + if installed_path.is_file(): + return self._read_toolchain(installed_path) + raise ValueError(f"Unknown compiler toolchain: {vendor!r}") - extra_compilation_tools.add("python") + @staticmethod + def _read_toolchain(path: Path) -> Mapping[str, Mapping[str, object]]: + with path.open(encoding="utf-8") as stream: + payload = json.load(stream) + if not isinstance(payload, dict): + raise ValueError(f"Compiler configuration must be a JSON object: {path}") + return payload + + def _language(self, language: str) -> Mapping[str, object]: + try: + configuration = self._toolchain[language] + except KeyError: + raise ValueError(f"Toolchain does not support {language!r}") from None + if not isinstance(configuration, Mapping): + raise ValueError(f"Invalid {language!r} toolchain configuration") + return configuration + + def _executable(self, language: Mapping[str, object], tools: Iterable[str]) -> str: + key = "mpi_exec" if "mpi" in tools else "exec" + command = str(language[key]) + executable = shutil.which(command, path=self._search_path) + if executable is None: + raise FileNotFoundError(f"Could not find compiler executable: {command}") + return executable + + def _flags( + self, + language: Mapping[str, object], + tools: Iterable[str], + requested: Iterable[str], + ) -> tuple[str, ...]: + profile = "debug_flags" if self._debug else "release_flags" + values = [*self._strings(language.get(profile, ())), *self._strings(language.get("general_flags", ()))] + for tool in sorted(set(tools)): + flags = self._tool_mapping(language, tool).get("flags", ()) + if tool == "python": + flags = tuple(flag for flag in self._strings(flags) if not self._is_python_profile_flag(flag)) + values.extend(self._strings(flags)) + values.extend(str(flag) for flag in requested) + return tuple(values) + + def _include_dirs( + self, + language: Mapping[str, object], + tools: Iterable[str], + requested: Iterable[str | Path], + ) -> tuple[Path, ...]: + return self._ordered_paths( + (*requested, *self._strings(language.get("include", ())), *self._tool_values(language, "include", tools)) + ) - # Collect compile information - exec_cmd, _, libs_flags, libdir_flags, m_code = self._get_compile_components( - compile_obj, extra_compilation_tools + def _library_dirs( + self, + language: Mapping[str, object], + tools: Iterable[str], + requested: Iterable[str | Path], + ) -> tuple[Path, ...]: + return self._ordered_paths( + (*requested, *self._strings(language.get("libdir", ())), *self._tool_values(language, "libdir", tools)) ) - linker_libdir_flags = ["-Wl,-rpath" if flag == "-L" else flag for flag in libdir_flags] - flags.insert(0, "-shared") + def _libraries( + self, + language: Mapping[str, object], + tools: Iterable[str], + requested: Iterable[str], + ) -> tuple[str, ...]: + return self._ordered_strings( + (*requested, *self._strings(language.get("libs", ())), *self._tool_values(language, "libs", tools)) + ) - # Get name of file - ext_suffix = self._language_info["python"]["shared_suffix"] - sharedlib_modname = sharedlib_modname or compile_obj.python_module - file_out = os.path.join(output_folder, sharedlib_modname + ext_suffix) + def _tool_values(self, language: Mapping[str, object], key: str, tools: Iterable[str]) -> tuple[str, ...]: + return tuple( + value + for tool in sorted(set(tools)) + for value in self._strings(self._tool_mapping(language, tool).get(key, ())) + ) - if verbose: - print(">> Compiling shared library :: ", file_out) - - cmd = [ - exec_cmd, - *flags, - *libdir_flags, - *linker_libdir_flags, - compile_obj.module_target, - *m_code, - *compile_obj.link_args, - "-o", - file_out, - *libs_flags, - ] + @staticmethod + def _tool_mapping(language: Mapping[str, object], tool: str) -> Mapping[str, object]: + value = language.get(tool, {}) + return value if isinstance(value, Mapping) else {} - with compile_obj: - self._run_or_record_command(cmd, verbose) + @staticmethod + def _strings(values: object) -> tuple[str, ...]: + if isinstance(values, str): + return (values,) + return tuple(str(value) for value in values) - self._language_info = None + @staticmethod + def _ordered_paths(paths: Iterable[str | Path]) -> tuple[Path, ...]: + return tuple(dict.fromkeys(Path(path) for path in paths)) - return file_out + @staticmethod + def _ordered_strings(values: Iterable[str]) -> tuple[str, ...]: + return tuple(dict.fromkeys(str(value) for value in values)) @staticmethod - def run_command(cmd, verbose): - """ - Run the provided command and collect the output. - - Run the provided compilation command, collect the output and raise any - necessary errors if the file does not compile. - - Parameters - ---------- - cmd : list of str - The command to run. - verbose : int - Indicates the level of verbosity. - - Returns - ------- - str - The exact command that was run. - - Raises - ------ - RuntimeError - Raises `RuntimeError` if the file does not compile. - """ - cmd = [os.path.expandvars(c) for c in cmd] - if verbose: - print(shlex.join(cmd)) + def _path_flags(flag: str, paths: Iterable[Path]) -> tuple[str, ...]: + return tuple(part for path in paths for part in (flag, str(path))) - start_time = time.perf_counter() - with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) as p: - out, err = p.communicate() - elapsed = time.perf_counter() - start_time + @staticmethod + def _library_flags(libraries: Iterable[str]) -> tuple[str, ...]: + return tuple(library if library.startswith("-l") else f"-l{library}" for library in libraries) - if verbose and out: - print(out) - if verbose: - print(f">> Command completed in {elapsed:.3f}s") - if p.returncode != 0: - err_msg = "Failed to build module" - err_msg += "\n" + err - raise RuntimeError(err_msg) - if err: - warnings.warn(UserWarning(err), stacklevel=2) - - return cmd + @staticmethod + def _is_python_profile_flag(flag: str) -> bool: + return flag.startswith("-O") or flag.startswith("-g") or flag == "-DNDEBUG" diff --git a/x2py/compiling/default_compilers.py b/x2py/compiling/default_compilers.py deleted file mode 100644 index ab0b8d6b8..000000000 --- a/x2py/compiling/default_compilers.py +++ /dev/null @@ -1,409 +0,0 @@ -""" -Module responsible for the creation of the json files containing the default configuration for each available compiler. -This module only needs to be imported once. Once the json files have been generated they can be used directly thus -avoiding the need for a large number of imports -""" - -import glob -import os -import shutil -import subprocess -import sys -import sysconfig - -# pybind11 support is disabled until C++ wrappers are enabled. -# import pybind11 -from numpy import get_include as get_numpy_include - -# ------------------------------------------------------------ -# GNU compilation configurations -# ------------------------------------------------------------ -gfort_info = { - "exec": "gfortran", - "mpi_exec": "mpif90", - "module_output_flag": "-J", - "debug_flags": ["-fcheck=bounds", "-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops", "-DNDEBUG"], - "general_flags": ["-fPIC", "-cpp"], - "standard_flags": ["-std=f2003"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - "libs": ["gomp"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - -# ------------------------------------------------------------ -gcc_info = { - "exec": "gcc", - "mpi_exec": "mpicc", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops", "-DNDEBUG"], - "general_flags": ["-fPIC"], - "standard_flags": ["-std=c99"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - "libs": ["gomp"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - -# ------------------------------------------------------------ -gpp_info = { - "exec": "g++", - "mpi_exec": "mpic++", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops"], - "general_flags": ["-fPIC"], - "standard_flags": ["--std=c++20"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - "libs": ["gomp"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - - -if sys.platform == "darwin": - p = subprocess.run([shutil.which("gcc"), "--version"], check=False, capture_output=True, text=True) - if p.returncode == 0 and "Apple clang" in p.stdout: - p = subprocess.run([shutil.which("brew"), "--prefix"], check=True, capture_output=True) - HOMEBREW_PREFIX = p.stdout.decode().strip() - OMP_PATH = os.path.join(HOMEBREW_PREFIX, "opt/libomp") - - gcc_info["openmp"]["flags"] = ["-Xpreprocessor", "-fopenmp"] - gcc_info["openmp"]["libs"] = ["omp"] - gcc_info["openmp"]["libdir"] = [os.path.join(OMP_PATH, "lib")] - gcc_info["openmp"]["include"] = [os.path.join(OMP_PATH, "include")] - -# ------------------------------------------------------------ -# Intel compilation configurations -# ------------------------------------------------------------ -ifort_info = { - "exec": "ifx", - "mpi_exec": "mpiifx", - "module_output_flag": "-module", - "debug_flags": ["-check", "bounds", "-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops", "-DNDEBUG"], - "general_flags": ["-fPIC", "-fpp"], - "standard_flags": ["-std=f2003"], - "openmp": { - "flags": ["-qopenmp", "-nostandard-realloc-lhs"], - "libs": ["iomp5"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - -# ------------------------------------------------------------ -icc_info = { - "exec": "icx", - "mpi_exec": "mpiicx", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops", "-DNDEBUG"], - "general_flags": ["-fPIC"], - "standard_flags": ["-std=c99"], - "openmp": { - "flags": ["-qopenmp"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - -# ------------------------------------------------------------ -icpp_info = { - "exec": "icpx", - "mpi_exec": "mpiicpx", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops"], - "general_flags": ["-fPIC"], - "standard_flags": ["--std=c++20"], - "openmp": { - "flags": ["-qopenmp"], - }, - "openacc": { - "flags": ["-ta=multicore", "-Minfo=accel"], - }, -} - -# ------------------------------------------------------------ -# PGI compilation configurations -# ------------------------------------------------------------ -pgfortran_info = { - "exec": "pgfortran", - "mpi_exec": "pgfortran", - "module_output_flag": "-module", - "debug_flags": ["-Mbounds", "-g", "-O0"], - "release_flags": ["-O3", "-Munroll", "-DNDEBUG"], - "general_flags": ["-fPIC", "-cpp"], - "standard_flags": ["-Mstandard"], - "openmp": { - "flags": ["-mp"], - }, - "openacc": { - "flags": ["-acc"], - }, -} - -# ------------------------------------------------------------ -pgcc_info = { - "exec": "pgcc", - "mpi_exec": "pgcc", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-Munroll", "-DNDEBUG"], - "general_flags": ["-fPIC"], - "standard_flags": ["-std=c99"], - "openmp": { - "flags": ["-mp"], - }, - "openacc": { - "flags": ["-acc"], - }, -} - -# ------------------------------------------------------------ -# Nvidia compilation configurations -# ------------------------------------------------------------ -nvfort_info = { - "exec": "nvfort", - "mpi_exec": "mpifort", - "module_output_flag": "-module", - "debug_flags": ["-Mbounds", "-g", "-O0"], - "release_flags": ["-O3", "-Munroll", "-DNDEBUG"], - "general_flags": ["-fPIC", "-cpp"], - "standard_flags": ["-Mstandard"], - "openmp": { - "flags": ["-mp"], - }, - "openacc": { - "flags": ["-acc"], - }, -} - -# ------------------------------------------------------------ -nvc_info = { - "exec": "nvc", - "mpi_exec": "mpicc", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-Munroll", "-DNDEBUG"], - "general_flags": ["-fPIC"], - "standard_flags": ["-std=c99"], - "openmp": { - "flags": ["-mp"], - }, - "openacc": { - "flags": ["-acc"], - }, -} - -# ------------------------------------------------------------ -nvcpp_info = { - "exec": "nvc++", - "mpi_exec": "mpic++", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-Munroll"], - "general_flags": ["-fPIC"], - "standard_flags": ["--std=c++20"], - "openmp": { - "flags": ["-mp"], - }, - "openacc": { - "flags": ["-acc"], - }, -} - -# ------------------------------------------------------------ -# Clang compiler configurations -# ------------------------------------------------------------ -flang_info = { - "exec": "flang", - "mpi_exec": "mpifort", - "module_output_flag": "-J", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-DNDEBUG"], - "general_flags": ["-fPIC", "-cpp"], - "standard_flags": ["-std=f2003"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - }, - "openacc": { - "flags": ["-fopenacc"], - }, -} - -# ------------------------------------------------------------ -clang_info = { - "exec": "clang", - "mpi_exec": "mpicc", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops", "-DNDEBUG"], - "general_flags": ["-fPIC"], - "standard_flags": ["-std=c99"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - }, - "openacc": { - "flags": ["-fopenacc"], - }, -} - -# ------------------------------------------------------------ -clangpp_info = { - "exec": "clang++", - "mpi_exec": "mpic++", - "debug_flags": ["-g", "-O0"], - "release_flags": ["-O3", "-funroll-loops"], - "general_flags": ["-fPIC"], - "standard_flags": ["--std=c++20"], - "mpi": {}, - "openmp": { - "flags": ["-fopenmp"], - }, - "openacc": { - "flags": ["-fopenacc"], - }, -} - - -# ------------------------------------------------------------ -def change_to_lib_flag(lib): - """ - Convert a library to a library flag. - - Take a library file and return the associated library - flag by stripping the library suffix. If the file does - not begin with the expected 'lib' prefix then it is returned - unchanged. - - Parameters - ---------- - lib : str - The library file. - - Returns - ------- - str - The library flag. - """ - if lib.startswith("lib"): - end = len(lib) - if lib.endswith(".a"): - end = end - 2 - if lib.endswith(".so"): - end = end - 3 - if lib.endswith(".dylib"): - end = end - 5 - return f"-l{lib[3:end]}" - return lib - - -config_vars = sysconfig.get_config_vars() - -python_info = { - "libs": config_vars.get("LIBM", "").split(), # Strip -l from beginning - "python": { - "flags": config_vars.get("CFLAGS", "").split() + config_vars.get("CC", "").split()[1:], - "include": [get_numpy_include(), *config_vars.get("INCLUDEPY", "").split()], - "shared_suffix": config_vars["EXT_SUFFIX"], - }, -} - -if sys.platform == "win32": - expected_dir = config_vars["prefix"] - version = config_vars["VERSION"] - python_libs = glob.glob(f"{expected_dir}/python{version}.dll") - if python_libs: - python_info["python"]["dependencies"] = list(python_libs) - else: - python_info["python"]["libs"] = [f"python{version}"] - python_info["python"]["libdir"] = config_vars.get("installed_base", "").split() - -else: - # Collect library according to python config file - expected_dir = config_vars["LIBDIR"] - version = config_vars["VERSION"] - python_shared_libs = glob.glob(f"{expected_dir}/libpython{version}*") - - # Collect a list of all possible libraries matching the name in the configs - # which can be found on the system - shared_ending = ".dylib" if sys.platform == "darwin" else ".so" - possible_shared_lib = [library for library in python_shared_libs if shared_ending in library] - possible_static_lib = [library for library in python_shared_libs if ".a" in library] - - # Prefer saving the library as a dependency where possible to avoid - # unnecessary libdir which may lead to the wrong versions being linked - # for other libraries - # Prefer a shared library as it requires less memory - if possible_shared_lib: - if len(possible_shared_lib) > 1: - preferred_lib = [library for library in possible_shared_lib if library.endswith(shared_ending)] - if preferred_lib: - possible_shared_lib = preferred_lib - - python_info["python"]["dependencies"] = [possible_shared_lib[0]] - python_info["python"]["libdir"] = [os.path.dirname(possible_shared_lib[0])] - elif possible_static_lib: - if len(possible_static_lib) > 1: - preferred_lib = [library for library in possible_static_lib if library.endswith(".a")] - if preferred_lib: - possible_static_lib = preferred_lib - python_info["python"]["dependencies"] = [possible_static_lib[0]] - else: - # If the proposed library does not exist use different config flags - # to specify the library - linker_flags = [ - change_to_lib_flag(flag) - for flag in config_vars.get("LDSHARED", "").split() + config_vars.get("LIBRARY", "").split()[1:] - ] - python_info["python"]["libs"] = [flag[2:] for flag in linker_flags if flag.startswith("-l")] - python_info["python"]["libdir"] = ( - [flag[2:] for flag in linker_flags if flag.startswith("-L")] - + config_vars.get("LIBPL", "").split() - + config_vars.get("LIBDIR", "").split() - ) - -# ------------------------------------------------------------ -gcc_info.update(python_info) -gpp_info.update(python_info) -gfort_info.update(python_info) -icc_info.update(python_info) -icpp_info.update(python_info) -ifort_info.update(python_info) -pgcc_info.update(python_info) -pgfortran_info.update(python_info) -nvc_info.update(python_info) -nvcpp_info.update(python_info) -nvfort_info.update(python_info) -clang_info.update(python_info) -clangpp_info.update(python_info) -flang_info.update(python_info) - -available_compilers = { - "GNU": {"c": gcc_info, "c++": gpp_info, "fortran": gfort_info}, - "intel": {"c": icc_info, "c++": icpp_info, "fortran": ifort_info}, - "PGI": {"c": pgcc_info, "fortran": pgfortran_info}, - "nvidia": {"c": nvc_info, "c++": nvcpp_info, "fortran": nvfort_info}, - "LLVM": {"c": clang_info, "c++": clangpp_info, "fortran": flang_info}, -} - -# for config in available_compilers.values(): -# cpp_config = config.get("c++", None) -# if cpp_config: -# cpp_config.setdefault("python", {}).setdefault("include", []).append( -# pybind11.get_include() -# ) - -vendors = ("GNU", "intel", "PGI", "nvidia", "LLVM") diff --git a/x2py/compiling/objects.py b/x2py/compiling/objects.py new file mode 100644 index 000000000..b284dc484 --- /dev/null +++ b/x2py/compiling/objects.py @@ -0,0 +1,37 @@ +"""Explicit object-file inputs for native wrapper builds.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +__all__ = ("ObjectFile",) + + +@dataclass(frozen=True) +class ObjectFile: + """Describe one source file and the object file it must produce. + + Build orchestration owns the order in which objects are compiled and linked. + This value only carries the complete inputs for one compiler invocation. + """ + + source: Path + object_path: Path + language: str + flags: tuple[str, ...] = () + include_dirs: tuple[Path, ...] = () + library_dirs: tuple[Path, ...] = () + libraries: tuple[str, ...] = () + tools: frozenset[str] = frozenset() + + def __post_init__(self) -> None: + if self.language not in {"c", "fortran"}: + raise ValueError(f"Unsupported compilation language: {self.language!r}") + object.__setattr__(self, "source", Path(self.source)) + object.__setattr__(self, "object_path", Path(self.object_path)) + object.__setattr__(self, "flags", tuple(str(flag) for flag in self.flags)) + object.__setattr__(self, "include_dirs", tuple(Path(path) for path in self.include_dirs)) + object.__setattr__(self, "library_dirs", tuple(Path(path) for path in self.library_dirs)) + object.__setattr__(self, "libraries", tuple(str(library) for library in self.libraries)) + object.__setattr__(self, "tools", frozenset(str(tool) for tool in self.tools)) diff --git a/x2py/compiling/runtime_support.py b/x2py/compiling/runtime_support.py index c8a5cd83d..041394562 100644 --- a/x2py/compiling/runtime_support.py +++ b/x2py/compiling/runtime_support.py @@ -8,7 +8,7 @@ import x2py.stdlib as stdlib_folder -from .basic import CompileObj +from .objects import ObjectFile _RUNTIME_IMPORT = "x2py_runtime" @@ -26,32 +26,28 @@ def _numpy_version_header() -> str: return header -def install_runtime_support(imports, *, x2py_dirpath, compiler, wrapper_obj, language, verbose): - """Copy, register, and compile runtime support imported by one wrapper.""" +def install_runtime_support(imports, *, x2py_dirpath, verbose: bool | int = False) -> tuple[ObjectFile, ...]: + """Write runtime support and return its explicit compilation inputs.""" if not any(name == _RUNTIME_IMPORT or name.startswith(f"{_RUNTIME_IMPORT}/") for name in imports): - return + return () destination = Path(x2py_dirpath) / _RUNTIME_IMPORT + if verbose: + print(f">> Write runtime support: {destination}") with FileLock(str(destination.with_suffix(".lock"))): shutil.rmtree(destination, ignore_errors=True) - if verbose: - print(f">> Copying {_RUNTIME_SOURCE} to {destination}") shutil.copytree(_RUNTIME_SOURCE, destination) (destination / "numpy_version.h").write_text( _numpy_version_header(), encoding="utf-8", ) - runtime_obj = CompileObj( - "python_runtime.c", - destination, - include=(destination,), - extra_compilation_tools=("python",), - ) - wrapper_obj.add_dependencies(runtime_obj) - compiler.compile_module( - compile_obj=runtime_obj, - output_folder=destination, - language=language, - verbose=verbose, + return ( + ObjectFile( + source=destination / "python_runtime.c", + object_path=destination / "python_runtime.o", + language="c", + include_dirs=(destination,), + tools=frozenset({"python"}), + ), ) diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 56e0172f2..8d13a84f6 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Iterable +from collections.abc import Callable, Iterable from dataclasses import dataclass, field, replace import json import os @@ -10,9 +10,7 @@ import shlex import time -from filelock import FileLock - -from x2py.compiling.basic import CompileObj +from x2py.compiling.objects import ObjectFile from x2py.compiling.compilers import Compiler, get_condaless_search_path from x2py.compiling.runtime_support import install_runtime_support from x2py.parsers.fortran.parser import parse_fortran_project @@ -73,10 +71,35 @@ } -def _print_verbose_timing(verbose: bool | int, label: str, elapsed: float) -> None: - """Print one elapsed build-stage timing when verbose output is enabled.""" +def _print_verbose_timing(verbose: bool | int, elapsed: float) -> None: + """Print the elapsed time for the immediately preceding build operation.""" if verbose: - print(f">> Timing :: {label}: {elapsed:.3f}s") + print(f">> Timing: {elapsed:.3f}s") + + +def _print_verbose_total_build_time(verbose: bool | int, elapsed: float) -> None: + """Print the completed end-to-end direct-build duration.""" + if verbose: + print(f">> Total build time: {elapsed:.3f}s") + + +def _report_total_build_time( + verbose: bool | int, + elapsed: float, + *, + on_total_build_time: Callable[[float], None] | None, +) -> None: + """Print or defer the final duration for one successful direct build.""" + if on_total_build_time is not None: + on_total_build_time(elapsed) + return + _print_verbose_total_build_time(verbose, elapsed) + + +def _print_verbose_step(verbose: bool | int, label: str) -> None: + """Print one readable build step before it can report a native error.""" + if verbose: + print(f">> {label}") @dataclass(frozen=True) @@ -246,19 +269,23 @@ def _compiler_flags(flags: Iterable[str] | None) -> tuple[str, ...]: def _new_gnu_compiler(*, execute_commands: bool = True, debug: bool = False) -> Compiler: - Compiler.acceptable_bin_paths = get_condaless_search_path("verbose") - return Compiler("GNU", debug=debug, execute_commands=execute_commands) + return Compiler( + "GNU", + debug=debug, + execute_commands=execute_commands, + search_path=get_condaless_search_path("verbose"), + ) def _expected_generated_files( *, - source_objects: tuple[CompileObj, ...], + source_objects: tuple[ObjectFile, ...], output_dir: Path, module_name: str, shared_library: Path, ) -> tuple[Path, ...]: candidates = [ - *(source_obj.module_target for source_obj in source_objects), + *(source_obj.object_path for source_obj in source_objects), output_dir / f"bind_c_{module_name}.mod", output_dir / f"bind_c_{module_name}_wrapper.mod", output_dir / f"bind_c_{module_name}_wrapper.f90", @@ -284,11 +311,14 @@ def _rendered_artifact_output_path(output_dir: Path, path: Path) -> Path: def _write_rendered_wrapper_sources( rendered: RenderedGeneratedWrapperArtifacts, output_dir: Path, + *, + verbose: bool | int = False, ) -> tuple[Path, ...]: """Write rendered wrapper-plan sources into one build directory.""" written = [] for source in rendered.sources: path = _rendered_artifact_output_path(output_dir, source.path) + _print_verbose_step(verbose, f"{_rendered_source_write_label(rendered, source.path)}: {path}") path.parent.mkdir(parents=True, exist_ok=True) path.write_text(source.text, encoding="utf-8") written.append(path) @@ -302,6 +332,17 @@ def _rendered_source_payloads( return {Path(source.path): source for source in rendered.sources} +def _rendered_source_write_label(rendered: RenderedGeneratedWrapperArtifacts, source_path: Path) -> str: + """Return the verbose write label for one generated artifact.""" + if source_path in rendered.artifacts.bridge_sources: + return "Write bridge source" + if source_path in rendered.artifacts.binding_sources: + return "Write binding source" + if source_path in rendered.artifacts.header_files: + return "Write binding header" + return "Write wrapper artifact" + + def _rendered_wrapper_compile_source_paths( rendered: RenderedGeneratedWrapperArtifacts, ) -> tuple[Path, ...]: @@ -333,69 +374,89 @@ def _rendered_wrapper_runtime_imports(runtime_support_keys: Iterable[str]) -> tu return tuple(imports) -def _rendered_wrapper_compile_obj( +def _rendered_wrapper_object_file( source_path: Path, output_dir: Path, *, - dependencies: tuple[CompileObj, ...], - link_args: tuple[str, ...], flags: tuple[str, ...], include_dirs: tuple[Path, ...], - library_dirs: tuple[Path, ...], language: str, -) -> CompileObj: - """Return one compile object for a rendered wrapper-plan source.""" - return CompileObj( - source_path.name, - _rendered_artifact_output_path(output_dir, source_path).parent, +) -> ObjectFile: + """Return one explicit object-file input for a rendered wrapper source.""" + source = _rendered_artifact_output_path(output_dir, source_path) + return ObjectFile( + source=source, + object_path=source.with_suffix(".o"), + language=language, flags=flags, - include=include_dirs, - libdir=library_dirs, - link_args=link_args, - dependencies=dependencies, - extra_compilation_tools=("python",) if language == "c" else (), + include_dirs=include_dirs, + tools=frozenset({"python"}) if language == "c" else frozenset(), ) -def _rendered_wrapper_compile_objects( +def _rendered_wrapper_object_stages( rendered: RenderedGeneratedWrapperArtifacts, output_dir: Path, *, - native_dependencies: tuple[CompileObj, ...], - native_link_args: tuple[str, ...], wrapper_fortran_flags: tuple[str, ...], wrapper_c_flags: tuple[str, ...], native_module_dirs: tuple[Path, ...], - native_library_dirs: tuple[Path, ...], -) -> tuple[tuple[CompileObj, str, Path], ...]: - """Return compile objects for rendered wrapper-plan sources.""" - compiled: list[CompileObj] = [] - result = [] +) -> tuple[tuple[ObjectFile, ...], tuple[ObjectFile, ...]]: + """Return bridge and binding objects in their required compile order.""" source_paths = _rendered_wrapper_compile_source_paths(rendered) - final_source = source_paths[-1] - for source_path in source_paths: - language = _rendered_wrapper_source_language(source_path) - flags = wrapper_c_flags if language == "c" else wrapper_fortran_flags - obj = _rendered_wrapper_compile_obj( + bridge_source_paths = source_paths[: len(rendered.artifacts.bridge_sources)] + binding_source_paths = source_paths[len(bridge_source_paths) :] + bridge_objects = tuple( + _rendered_wrapper_object_file( source_path, output_dir, - dependencies=(*native_dependencies, *compiled), - link_args=native_link_args if source_path == final_source else (), - flags=flags, - include_dirs=native_module_dirs if language == "fortran" else (), - library_dirs=native_library_dirs if source_path == final_source else (), - language=language, + flags=wrapper_fortran_flags, + include_dirs=native_module_dirs, + language=_rendered_wrapper_source_language(source_path), ) - compiled.append(obj) - result.append((obj, language, source_path)) - return tuple(result) + for source_path in bridge_source_paths + ) + binding_objects = tuple( + _rendered_wrapper_object_file( + source_path, + output_dir, + flags=wrapper_c_flags, + include_dirs=(), + language=_rendered_wrapper_source_language(source_path), + ) + for source_path in binding_source_paths + ) + return bridge_objects, binding_objects -def _rendered_wrapper_link_language(compile_items: tuple[tuple[CompileObj, str, Path], ...]) -> str: +def _rendered_wrapper_link_language( + bridge_objects: tuple[ObjectFile, ...], + binding_objects: tuple[ObjectFile, ...], +) -> str: """Return the linker language for rendered wrapper-plan sources.""" - if any(language == "fortran" for _obj, language, _path in compile_items): + if bridge_objects: return "fortran" - return compile_items[-1][1] + if not binding_objects: + raise ValueError("Rendered wrapper artifacts must include at least one binding source") + return binding_objects[-1].language + + +def _compile_object_stage( + compiler: Compiler, + object_files: Iterable[ObjectFile], + *, + label: str, + verbose: bool | int, +) -> None: + """Compile one named object group and expose that boundary in verbose logs.""" + objects = tuple(object_files) + if not objects: + return + for object_file in objects: + _print_verbose_step(verbose, f"{label}: {object_file.source} -> {object_file.object_path}") + started = time.perf_counter() + compiler.compile_object(object_file, verbose=verbose) + _print_verbose_timing(verbose, time.perf_counter() - started) def _build_rendered_wrapper_extension( @@ -405,7 +466,7 @@ def _build_rendered_wrapper_extension( shared_library_output_dir: str | Path | None = None, sources: Iterable[str | Path] = (), native_build_plan: NativeBuildPlan | None = None, - native_dependencies: Iterable[CompileObj] = (), + native_dependencies: Iterable[ObjectFile] = (), native_link_args: Iterable[str] = (), wrapper_fortran_flags: Iterable[str] | None = None, wrapper_c_flags: Iterable[str] | None = None, @@ -418,17 +479,13 @@ def _build_rendered_wrapper_extension( output_path.mkdir(parents=True, exist_ok=True) shared_output_path = Path(shared_library_output_dir) if shared_library_output_dir is not None else output_path shared_output_path.mkdir(parents=True, exist_ok=True) - printing_started = time.perf_counter() - _write_rendered_wrapper_sources(rendered, output_path) - _print_verbose_timing(verbose, "Wrapper printing", time.perf_counter() - printing_started) + _write_rendered_wrapper_sources(rendered, output_path, verbose=verbose) compiler = compiler or _new_gnu_compiler() resolved_native_build_plan = native_build_plan or NativeBuildPlan() - compile_items = _rendered_wrapper_compile_objects( + bridge_objects, binding_objects = _rendered_wrapper_object_stages( rendered, output_path, - native_dependencies=tuple(native_dependencies), - native_link_args=tuple(native_link_args), wrapper_fortran_flags=_compiler_flags(wrapper_fortran_flags), wrapper_c_flags=_compiler_flags(wrapper_c_flags), native_module_dirs=_unique_paths( @@ -437,38 +494,44 @@ def _build_rendered_wrapper_extension( *resolved_native_build_plan.include_dirs, ) ), - native_library_dirs=resolved_native_build_plan.library_dirs, ) - compilation_started = time.perf_counter() runtime_imports = _rendered_wrapper_runtime_imports(rendered.artifacts.runtime_support_keys) - for compile_obj, language, source_path in compile_items: - imports = runtime_imports if source_path in rendered.artifacts.binding_sources else () - install_runtime_support( - imports, - x2py_dirpath=str(output_path), - compiler=compiler, - wrapper_obj=compile_obj, - language=language, - verbose=verbose, - ) - compiler.compile_module( - compile_obj=compile_obj, - output_folder=str(output_path), - language=language, - verbose=verbose, - ) + runtime_objects = install_runtime_support( + runtime_imports, + x2py_dirpath=str(output_path), + verbose=verbose, + ) + _compile_object_stage( + compiler, + bridge_objects, + label="Compile bridge source", + verbose=verbose, + ) + _compile_object_stage( + compiler, + runtime_objects, + label="Compile runtime source", + verbose=verbose, + ) + _compile_object_stage( + compiler, + binding_objects, + label="Compile binding source", + verbose=verbose, + ) - final_obj = compile_items[-1][0] - shared_library = Path( - compiler.compile_shared_library( - final_obj, - output_folder=str(shared_output_path), - sharedlib_modname=rendered.artifacts.module_name, - language=_rendered_wrapper_link_language(compile_items), - verbose=verbose, - ) + linking_started = time.perf_counter() + shared_library = compiler.link_extension( + module_name=rendered.artifacts.module_name, + output_dir=shared_output_path, + language=_rendered_wrapper_link_language(bridge_objects, binding_objects), + objects=(*tuple(native_dependencies), *bridge_objects, *runtime_objects, *binding_objects), + link_args=tuple(native_link_args), + library_dirs=resolved_native_build_plan.library_dirs, + flags=_compiler_flags(wrapper_c_flags), + verbose=verbose, ) - _print_verbose_timing(verbose, "Wrapper compilation", time.perf_counter() - compilation_started) + _print_verbose_timing(verbose, time.perf_counter() - linking_started) generated_sources = tuple( path for path in rendered.artifacts.generated_files @@ -497,7 +560,7 @@ def _attach_build_makefile( result: WrapperBuildResult, *, compiler: Compiler, - source_objects: tuple[CompileObj, ...], + source_objects: tuple[ObjectFile, ...], extra_dependencies: tuple[Path, ...] = (), build_manifest: Path | None = None, ) -> WrapperBuildResult: @@ -519,10 +582,14 @@ def _attach_build_makefile( ) -def _render_wrapper_plan(module: SemanticModule) -> RenderedGeneratedWrapperArtifacts: +def _render_wrapper_plan( + module: SemanticModule, + *, + progress: Callable[[str, float | None], None] | None = None, +) -> RenderedGeneratedWrapperArtifacts: """Render one policy-completed module through the canonical generator.""" plan = WrapperPlanner().build(module) - return WrapperCodeGenerator().generate(plan) + return WrapperCodeGenerator().generate(plan, progress=progress) def _require_wrapper_plan_support(module: SemanticModule) -> None: @@ -542,12 +609,19 @@ def _generated_wrapper_plan_artifacts( verbose: bool | int = False, ) -> RenderedGeneratedWrapperArtifacts: """Complete policy and generate the one production wrapper representation.""" - creation_started = time.perf_counter() + _print_verbose_step(verbose, "Complete wrapper policies") + policy_started = time.perf_counter() complete_semantic_policies(module, strict_wrapper_names=strict_wrapper_names) _require_wrapper_plan_support(module) - rendered = _render_wrapper_plan(module) - _print_verbose_timing(verbose, "Wrapper creation", time.perf_counter() - creation_started) - return rendered + _print_verbose_timing(verbose, time.perf_counter() - policy_started) + + def render_progress(label: str, elapsed: float | None) -> None: + if elapsed is None: + _print_verbose_step(verbose, label) + return + _print_verbose_timing(verbose, elapsed) + + return _render_wrapper_plan(module, progress=render_progress) def _source_compile_object( @@ -557,20 +631,15 @@ def _source_compile_object( object_stem: str, flags: Iterable[str] = (), include_dirs: Iterable[Path] = (), -) -> CompileObj: - compile_obj = CompileObj( - file_name=source_path.name, - folder=str(source_path.parent), +) -> ObjectFile: + target = output_dir / f"{object_stem}.o" + return ObjectFile( + source=source_path, + object_path=target, + language="fortran", flags=tuple(flags), - include=tuple(include_dirs), - has_target_file=True, + include_dirs=(*tuple(include_dirs), output_dir), ) - target = output_dir / f"{object_stem}.o" - if target != compile_obj.module_target: - compile_obj._module_target = target - compile_obj._lock_target = FileLock(str(target.with_suffix(target.suffix + ".lock"))) - compile_obj._include.add(output_dir) - return compile_obj def _source_paths(sources: str | Path | Iterable[str | Path]) -> tuple[Path, ...]: @@ -583,6 +652,16 @@ def _source_paths(sources: str | Path | Iterable[str | Path]) -> tuple[Path, ... return paths +def _wrapper_output_paths(output_dir: str | Path | None) -> tuple[Path, Path]: + """Return build and extension directories owned by one wrapper invocation.""" + if output_dir is not None: + path = Path(output_dir) + return path, path + invocation_dir = Path.cwd() + build_dir = invocation_dir / _DEFAULT_BUILD_DIR_NAME + return build_dir, build_dir + + def _pyi_entry_path(contract: str | Path) -> Path: if not isinstance(contract, str | Path): raise TypeError(".pyi wrapper build accepts exactly one entry contract path") @@ -948,16 +1027,16 @@ def _unique_paths(paths: Iterable[Path]) -> tuple[Path, ...]: def _source_native_build_plan( source_paths: tuple[Path, ...], - source_objects: tuple[CompileObj, ...], + source_objects: tuple[ObjectFile, ...], *, module_dir: Path, ) -> NativeBuildPlan: - produced_objects = tuple(Path(source_object.module_target) for source_object in source_objects) + produced_objects = tuple(source_object.object_path for source_object in source_objects) return NativeBuildPlan( compilation_units=tuple( NativeCompilationUnit( source=source_path, - object_path=source_object.module_target, + object_path=source_object.object_path, language="fortran", module_dir=module_dir, include_dirs=(module_dir,), @@ -975,7 +1054,7 @@ def _source_native_build_plan( def _pyi_native_build_plan( *, source_paths: tuple[Path, ...], - source_objects: tuple[CompileObj, ...], + source_objects: tuple[ObjectFile, ...], artifact_paths: tuple[Path, ...], libraries: tuple[str, ...], explicit_link_items: tuple[NativeLinkItem, ...], @@ -985,7 +1064,7 @@ def _pyi_native_build_plan( include_dirs: tuple[Path, ...], module_dir: Path | None, ) -> NativeBuildPlan: - produced_objects = tuple(Path(source_object.module_target) for source_object in source_objects) + produced_objects = tuple(source_object.object_path for source_object in source_objects) source_link_items = tuple(NativeLinkItem("object", object_path) for object_path in produced_objects) prebuilt_artifacts = tuple( NativePrebuiltArtifact(path=path, kind=_native_artifact_kind(path)) for path in artifact_paths @@ -1007,7 +1086,7 @@ def _pyi_native_build_plan( compilation_units=tuple( NativeCompilationUnit( source=source_path, - object_path=source_object.module_target, + object_path=source_object.object_path, language="fortran", module_dir=module_dir, include_dirs=include_dirs, @@ -1162,7 +1241,7 @@ def _pyi_native_source_objects( *, output_path: Path, include_dirs: tuple[Path, ...], -) -> tuple[CompileObj, ...]: +) -> tuple[ObjectFile, ...]: return tuple( _source_compile_object( source_path, @@ -1543,7 +1622,7 @@ def _write_build_makefile( *, path: Path, commands: tuple[tuple[str, ...], ...], - source_objects: tuple[CompileObj, ...], + source_objects: tuple[ObjectFile, ...], working_directory: Path, extra_dependencies: Iterable[Path] = (), ) -> Path: @@ -1554,7 +1633,7 @@ def _write_build_makefile( raise RuntimeError("cannot generate Makefile without a shared-library link command") user_outputs = tuple( - _absolute_command_path(source_object.module_target, working_directory) for source_object in source_objects + _absolute_command_path(source_object.object_path, working_directory) for source_object in source_objects ) compile_outputs = tuple( _absolute_command_path(_command_output(command), working_directory) for command in compile_commands @@ -1684,17 +1763,18 @@ def build_fortran_extension( wrapper_compiler_debug: bool = False, wrapper_fortran_flags: Iterable[str] | None = None, wrapper_c_flags: Iterable[str] | None = None, + _on_total_build_time: Callable[[float], None] | None = None, ) -> WrapperBuildResult: """Build one extension, or generate its Makefile, from ordered sources.""" if makefile and verbose: raise ValueError("makefile generation and verbose direct compilation are separate modes") + build_started = time.perf_counter() source_paths = _source_paths(sources) primary_source = source_paths[0] - output_path = Path(output_dir) if output_dir is not None else primary_source.parent / _DEFAULT_BUILD_DIR_NAME - shared_library_output_path = Path(output_dir) if output_dir is not None else primary_source.parent + output_path, shared_library_output_path = _wrapper_output_paths(output_dir) output_path.mkdir(parents=True, exist_ok=True) preprocessing = preprocessing or _default_preprocessing_config() @@ -1747,13 +1827,12 @@ def build_fortran_extension( source_objects, module_dir=output_path, ) - for source_obj in source_objects: - compiler.compile_module( - source_obj, - output_folder=str(output_path), - language="fortran", - verbose=verbose, - ) + _compile_object_stage( + compiler, + source_objects, + label="Compile native source", + verbose=verbose, + ) result = _build_rendered_wrapper_extension( rendered_wrapper_plan, @@ -1774,6 +1853,11 @@ def build_fortran_extension( compiler=compiler, source_objects=source_objects, ) + _report_total_build_time( + verbose, + time.perf_counter() - build_started, + on_total_build_time=_on_total_build_time, + ) return result @@ -1796,12 +1880,14 @@ def build_pyi_extension( wrapper_compiler_debug: bool = False, wrapper_fortran_flags: Iterable[str] | None = None, wrapper_c_flags: Iterable[str] | None = None, + _on_total_build_time: Callable[[float], None] | None = None, ) -> WrapperBuildResult: """Build one extension from one entry `.pyi` and native link inputs.""" if makefile and verbose: raise ValueError("makefile generation and verbose direct compilation are separate modes") + build_started = time.perf_counter() entry = _pyi_entry_path(contract) bundle = _pyi_contract_bundle(entry) native_inputs = _pyi_native_build_inputs( @@ -1815,9 +1901,7 @@ def build_pyi_extension( native_include_dirs=native_include_dirs, ) - primary_contract = bundle.entry - output_path = Path(output_dir) if output_dir is not None else primary_contract.parent / _DEFAULT_BUILD_DIR_NAME - shared_library_output_path = Path(output_dir) if output_dir is not None else primary_contract.parent + output_path, shared_library_output_path = _wrapper_output_paths(output_dir) output_path.mkdir(parents=True, exist_ok=True) wrapper_fortran_flags = _compiler_flags(wrapper_fortran_flags) wrapper_c_flags = _compiler_flags(wrapper_c_flags) @@ -1853,13 +1937,12 @@ def build_pyi_extension( ) _validate_native_link_paths(native_build_plan) compiler = _new_gnu_compiler(execute_commands=not makefile, debug=wrapper_compiler_debug) - for source_obj in native_source_objects: - compiler.compile_module( - source_obj, - output_folder=str(output_path), - language="fortran", - verbose=verbose, - ) + _compile_object_stage( + compiler, + native_source_objects, + label="Compile native source", + verbose=verbose, + ) native_array_build_requirements = native_array_handle_build_requirements(module) result = _build_rendered_wrapper_extension( @@ -1900,6 +1983,11 @@ def build_pyi_extension( extra_dependencies=dependencies, build_manifest=build_manifest, ) + _report_total_build_time( + verbose, + time.perf_counter() - build_started, + on_total_build_time=_on_total_build_time, + ) return result @@ -1909,9 +1997,11 @@ def build_pyi_extension_from_manifest( output_name: str | None = None, makefile: bool = False, verbose: bool | int = False, + _on_total_build_time: Callable[[float], None] | None = None, ) -> WrapperBuildResult: """Replay a saved semantic `.pyi` wrapper build manifest.""" + build_started = time.perf_counter() manifest_path, payload = _load_build_manifest(manifest) base = manifest_path.parent native_section = _manifest_section(payload, "native_build_plan") @@ -1950,12 +2040,18 @@ def build_pyi_extension_from_manifest( wrapper_fortran_flags=_manifest_string_list(compiler_section, "wrapper_fortran_flags"), wrapper_c_flags=_manifest_string_list(compiler_section, "wrapper_c_flags"), complete_native_link_items=_manifest_link_items(native_section, base=base), + _on_total_build_time=lambda _elapsed: None, ) recorded_contracts = tuple( _resolve_manifest_path(path, base=base) for path in _manifest_string_list(payload, "contract_paths") ) if result.sources != recorded_contracts: raise ValueError("Current .pyi import graph does not match the wrapper build manifest contract_paths") + _report_total_build_time( + verbose, + time.perf_counter() - build_started, + on_total_build_time=_on_total_build_time, + ) return result diff --git a/x2py/stdlib/x2py_runtime/CMakeLists.txt b/x2py/stdlib/x2py_runtime/CMakeLists.txt deleted file mode 100644 index cb4717bcd..000000000 --- a/x2py/stdlib/x2py_runtime/CMakeLists.txt +++ /dev/null @@ -1,11 +0,0 @@ -add_library(x2py_runtime OBJECT python_runtime.c) - -target_include_directories(x2py_runtime - PUBLIC - ${CMAKE_CURRENT_SOURCE_DIR} -) - -target_link_libraries(x2py_runtime - PUBLIC - Python::NumPy -) diff --git a/x2py/stdlib/x2py_runtime/meson.build b/x2py/stdlib/x2py_runtime/meson.build deleted file mode 100644 index 111714181..000000000 --- a/x2py/stdlib/x2py_runtime/meson.build +++ /dev/null @@ -1,8 +0,0 @@ -py_dep = py.dependency() -numpy_dep = dependency('numpy') - -x2py_runtime_incdir = include_directories('.') - -x2py_runtime_dep = declare_dependency(sources: 'python_runtime.c', - include_directories : x2py_runtime_incdir, - dependencies: [py_dep, numpy_dep]) diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index dc1805671..a851e25e7 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -609,6 +609,10 @@ def _require_derived_lifecycle_supported(action: LifecycleActionPlan) -> None: def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: """Return a complete C module and header from one shared plan.""" + return self.binding_module(plan), self.binding_header(plan) + + def binding_module(self, plan: ModulePlan) -> CModule: + """Lower the binding implementation from one completed wrapper plan.""" self._class_python_names = { surface.type_identity: surface.python_names[0] for namespace in plan.namespaces @@ -618,7 +622,7 @@ def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: functions = tuple(function for namespace in plan.namespaces for function in self.visit(namespace)) needs_runtime = self.requires_runtime_support(plan) needs_free = self._module_needs_allocator(plan) - c_module = CModule( + return CModule( name=f"{plan.binding.owner_path}_wrapper", defines=self._module_defines(needs_runtime), includes=self._module_includes(plan, needs_runtime, needs_free), @@ -637,12 +641,14 @@ def _visit_ModulePlan(self, plan: ModulePlan) -> tuple[CModule, CHeader]: self._module_init(plan, needs_runtime), ), ) - c_header = CHeader( + + def binding_header(self, plan: ModulePlan) -> CHeader: + """Lower the binding header from one completed wrapper plan.""" + return CHeader( guard=f"{plan.binding.owner_path.upper()}_WRAPPER_H", includes=(CInclude("Python.h"),), prototypes=tuple(self._binding_prototype(function) for function in self._functions(plan)), ) - return c_module, c_header def _visit_NamespacePlan(self, plan: NamespacePlan) -> tuple[CFunction, ...]: """Return binding functions directly owned by one Python namespace.""" diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index 669f5f637..d483afc8c 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -3,7 +3,9 @@ from __future__ import annotations from collections import Counter +from collections.abc import Callable from pathlib import Path +import time from x2py.pipeline.wrapper_artifacts import ( GeneratedSourceFile, @@ -115,19 +117,47 @@ def __init__( self._c_printer = c_printer or CSourcePrinter() self._fortran_printer = fortran_printer or FortranSourcePrinter() - def generate(self, plan: ModulePlan) -> RenderedGeneratedWrapperArtifacts: + def generate( + self, + plan: ModulePlan, + *, + progress: Callable[[str, float | None], None] | None = None, + ) -> RenderedGeneratedWrapperArtifacts: """Consume exactly one editable plan and return rendered artifacts.""" plan.freeze() self._validate_plan(plan) self._c_generator.require_supported(plan) self._fortran_generator.require_supported(plan) - c_module, c_header = self._c_generator.visit(plan) + + if progress is not None: + progress("Generate binding source", None) + started = time.perf_counter() + c_module = self._c_generator.binding_module(plan) + c_source = self._c_printer.doprint(c_module) + if progress is not None: + progress("Generate binding source", time.perf_counter() - started) + + if progress is not None: + progress("Generate bridge source", None) + started = time.perf_counter() fortran_module = self._fortran_generator.visit(plan) + fortran_source = self._fortran_printer.doprint(fortran_module) + if progress is not None: + progress("Generate bridge source", time.perf_counter() - started) + + if progress is not None: + progress("Generate binding header", None) + started = time.perf_counter() + c_header = self._c_generator.binding_header(plan) + c_header_source = self._c_printer.doprint(c_header) + if progress is not None: + progress("Generate binding header", time.perf_counter() - started) + return self._rendered_artifacts( plan.owner_path, - self._c_printer.doprint(c_module), - self._c_printer.doprint(c_header), - self._fortran_printer.doprint(fortran_module), + c_source, + c_header_source, + fortran_source, runtime_support_keys=(("python_runtime",) if self._c_generator.requires_runtime_support(plan) else ()), required_headers=plan.required_headers, ) From d8bcf47e9b08c85582a34238bfe722513ab1ce9c Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 20:31:01 +0100 Subject: [PATCH 24/30] improve naming --- THIRD_PARTY_NOTICES.md | 7 +- docs/developer/development-workflow.md | 4 + docs/developer/repository-structure.md | 1 + docs/developer/source-map.md | 2 +- .../wrapper-plan-migration-checklist.md | 3 +- docs/user/guide/fortran-wrapper.md | 5 + tests/README.md | 2 + tests/architecture/test_test_suite_layout.py | 2 + tests/naming/test_policy.py | 44 +++ tests/semantics/policy/test_wrapper_policy.py | 2 +- tests/utilities/test_strings.py | 30 ++ .../callbacks/test_scalar_callbacks.py | 4 +- .../test_calls_and_policy_metadata.py | 17 +- .../printers/test_classes_and_methods.py | 2 +- .../test_phase1a_wrapper_assembly.py | 18 +- .../test_phase1b_scalar_input_conversion.py | 38 +- .../test_phase2b_hidden_scalar_outputs.py | 2 +- .../test_phase2d_native_runtime_envelope.py | 4 +- .../test_phase2e_scalar_boundaries.py | 28 +- .../test_phase2f_multiple_scalar_results.py | 2 +- ...st_phase3_scalar_presence_and_writeback.py | 22 +- .../test_phase5a_string_inputs.py | 12 +- .../test_phase5c_fixed_string_writeback.py | 34 +- .../test_phase5d_string_addresses.py | 18 +- .../test_phase6a_array_buffers.py | 12 +- .../test_phase6b_dense_array_shapes.py | 31 +- .../test_phase6c_strided_arrays.py | 4 +- .../test_phase6d_array_output_identity.py | 6 +- ...ase6f_optional_assumed_character_arrays.py | 12 +- .../test_phase6g_raw_array_addresses.py | 12 +- .../test_phase7_native_array_handles.py | 6 +- .../test_phase8_derived_types.py | 2 +- x2py/naming/policy.py | 336 ++++++++---------- x2py/utilities/metaclasses.py | 39 -- x2py/utilities/strings.py | 101 ++---- x2py/wrapper_codegen/c/binding.py | 4 +- 36 files changed, 436 insertions(+), 432 deletions(-) create mode 100644 tests/utilities/test_strings.py delete mode 100644 x2py/utilities/metaclasses.py diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index c6a3692fb..5e7f53b06 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -1,9 +1,8 @@ # Third-Party Notices -The native NumPy/Python runtime in `x2py/stdlib/x2py_runtime/` and small -naming utilities in `x2py/utilities/strings.py`, `x2py/utilities/metaclasses.py`, -and `x2py/naming/policy.py` contain code adapted from the Pyccel project. The -current compilation package, semantic printer, and wrapper source printers are +The native NumPy/Python runtime in `x2py/stdlib/x2py_runtime/` contains code +adapted from the Pyccel project. The Python utilities, naming policy, +compilation package, semantic printer, and wrapper source printers are independent implementations and are not covered by this attribution. Upstream project: diff --git a/docs/developer/development-workflow.md b/docs/developer/development-workflow.md index 381ccdc84..a52c21ff7 100644 --- a/docs/developer/development-workflow.md +++ b/docs/developer/development-workflow.md @@ -308,6 +308,10 @@ X2PY_C_DOCS_END --> in semantic IR instead of the printed `.pyi`. - Use `SourceName("...")` only when a source identifier cannot be used as the Python target. Do not infer source identifiers from normalized Python names. +- Binding locals derived from a Python-visible argument must use the reserved + `bound_` namespace. Generated binding sources include Python, standard-library, + optional descriptor, NumPy, and runtime headers, so their imported identifier + sets are not a stable public-name vocabulary. - Omit `Polymorphic` only for the passed-object dummy of a type-bound procedure, where the binding itself restores that native fact. Ordinary `class(T)` arguments must retain it. diff --git a/docs/developer/repository-structure.md b/docs/developer/repository-structure.md index 2cc68cd38..201de0d74 100644 --- a/docs/developer/repository-structure.md +++ b/docs/developer/repository-structure.md @@ -49,6 +49,7 @@ through `x2py/__init__.py`. | `tests/pipeline/` | Preprocessing and semantic `.pyi` build-orchestration tests. | | `tests/semantics/` | Semantic conversion, completed policy, and readiness tests. | | `tests/wrapper_codegen/` | Typed planning, direct bridge/binding generation, and source-printer tests. | +| `tests/utilities/` | Shared Python utility tests. | | `tests/wrapper/fortran/` | Runtime wrapper tests that compile, import, call, and check failure paths. | | `tests/docs/` | Documentation example and structure checks. | | `tests/tools/` | Repository tooling tests. | diff --git a/docs/developer/source-map.md b/docs/developer/source-map.md index b7958a4b8..c3131c9eb 100644 --- a/docs/developer/source-map.md +++ b/docs/developer/source-map.md @@ -73,7 +73,7 @@ X2PY_C_DOCS_END --> | `x2py/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | | `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | | `x2py/stdlib/` | Native runtime support files copied into generated wrapper builds | `x2py_runtime/` | wrapper runtime tests | -| `x2py/utilities/` | Small shared Python utilities | `metaclasses.py`, `strings.py` | tests that exercise callers | +| `x2py/utilities/` | Small shared Python utilities | `strings.py`, `visitor.py` | tests that exercise callers | | `x2py/types/` | Semantic-to-Python ecosystem type mappings | `numpy.py` | `tests/types/test_numpy.py` | | `x2py/parsers/` | Public namespace for language and semantic `.pyi` frontends | child parser packages | `tests/parsing/`, parser references, semantic `.pyi` reference | | `x2py/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | -| `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and runtime support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `runtime_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | -| `x2py/stdlib/` | Native runtime support files copied into generated wrapper builds | `x2py_runtime/` | wrapper runtime tests | +| `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and native support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `native_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | +| `x2py/binding_support/` | Bundled native binding support copied into generated wrapper builds | support implementation and header | wrapper build tests | | `x2py/utilities/` | Small shared Python utilities | `strings.py`, `visitor.py` | tests that exercise callers | Verbose wrapper builds should print the exact compiler command lines they run, not only the source or target being compiled. The printed command should be shell-quoted so users can copy it to reproduce object compilation, generated -wrapper compilation, runtime support compilation, and final shared-library +wrapper compilation, native binding support compilation, and final shared-library linking. | Editable semantic contracts | `x2py/parsers/pyi/parser.py`, `x2py/pipeline/pyi.py`, `x2py/semantics/pyi2ir.py`, `x2py/wrapper_codegen/printers/pyi_printer.py` | `docs/user/reference/semantic-pyi-format.md` | | Readiness | `x2py/semantics/readiness.py` | `docs/user/reference/diagnostic-codes.md` | | Wrapper policy and lowering | `x2py/semantics/policy_completion.py`, `x2py/semantics/ownership.py`, `x2py/wrapper_codegen/planner.py`, `x2py/wrapper_codegen/generator.py` | `docs/user/guide/fortran-wrapper.md`, ownership docs | -| Native build | `x2py/pipeline/build.py`, `x2py/compiling/compilers.py`, `x2py/compiling/runtime_support.py` | compiling package README and build-system docs | +| Native build | `x2py/pipeline/build.py`, `x2py/compiling/compilers.py`, `x2py/compiling/native_support.py` | compiling package README and build-system docs | | Ownership, lifetime, ABI, or projection policy is unsafe | `x2py/semantics/ownership.py`, readiness, or `ir2ast` | | Generated code cannot represent a supported contract | bridge or binding generator with focused tests | | Compiler/linker invocation is wrong | `x2py/compiling/` or `x2py/pipeline/build.py` | -| Python runtime behavior is wrong | generated binding, runtime support, or ownership policy | +| Python binding behavior is wrong | generated binding, native support, or ownership policy | diff --git a/docs/maintainer/roadmap/documentation-content-checklist.md b/docs/maintainer/roadmap/documentation-content-checklist.md index 2c194a205..658df8079 100644 --- a/docs/maintainer/roadmap/documentation-content-checklist.md +++ b/docs/maintainer/roadmap/documentation-content-checklist.md @@ -97,7 +97,7 @@ X2PY_C_DOCS_END --> verification paths, fixture regeneration, documentation examples, wrapper runtime tests, and static-analysis gates. - [ ] `docs/developer/build-system.md`: document native compile model, - generated Makefiles, build manifests, runtime support files, compiler probes, + generated Makefiles, build manifests, native support files, compiler probes, and future packaging boundaries. - [ ] `docs/developer/coding-standards.md`: document Python style, documentation front matter, no-compatibility-layer rule, parser/codegen @@ -127,7 +127,7 @@ X2PY_C_DOCS_END --> preprocessing boundaries, model facts, diagnostics, and fixture strategy. - [ ] `docs/maintainer/design/semantic-analysis.md`: document source-to-IR lowering, `.pyi`-to-IR loading, policy completion, readiness blockers, and invariants. -- [ ] `docs/maintainer/design/runtime-model.md`: document runtime support files, generated +- [ ] `docs/maintainer/design/runtime-model.md`: document native support files, generated wrappers, native state, callbacks, threading, and finalization. - [ ] `docs/maintainer/design/error-propagation-model.md`: document diagnostic categories, Python exception projection, native failure handling, cleanup, and user-facing @@ -149,7 +149,7 @@ X2PY_C_DOCS_END --> - [ ] `docs/maintainer/internal-architecture/type-system.md`: document scalar kinds, arrays, characters, derived types, pointers, allocatables, callbacks, and unsupported storage forms. -- [ ] `docs/maintainer/internal-architecture/runtime-layer.md`: document runtime support +- [ ] `docs/maintainer/internal-architecture/runtime-layer.md`: document native support installation, extension initialization, callbacks, cleanup, and shared native state. - [ ] `docs/maintainer/internal-architecture/ownership-tracking.md`: document ownership diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 97bb7c7b2..26e02b1da 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -357,7 +357,7 @@ class WrapperCodeGenerator: The generator constructs `RenderedGeneratedWrapperArtifacts` directly from the printed source plus artifact metadata. It does not duplicate native build plans, -compiler selection, link ordering, runtime-support installation, or compilation +compiler selection, link ordering, native-support installation, or compilation policy; those remain in existing build/link orchestration. ## Direct Lowering Methods diff --git a/docs/old_docs/developper_guide.md b/docs/old_docs/developper_guide.md index a8c5cfa0e..e019def0b 100644 --- a/docs/old_docs/developper_guide.md +++ b/docs/old_docs/developper_guide.md @@ -75,7 +75,7 @@ Use these documentation roles consistently: | [fortran_parser.md](fortran_parser.md) | Maintainer inventory for the Fortran frontend | | [semantics.md](semantics.md) | Accepted semantic IR and datatype contract | | [pyi_format.md](pyi_format.md) | User-visible semantic `.pyi` syntax and roadmap | -| [wrapper_design_notes.md](wrapper_design_notes.md) | Clearly deferred wrapper policy, not current runtime support | +| [wrapper_design_notes.md](wrapper_design_notes.md) | Clearly deferred wrapper policy, not current native binding support | When adding a user example: @@ -210,7 +210,7 @@ implementation files. | Fortran wrapper orchestration | `x2py/wrapping.py` | `tests/wrapper/fortran/native_build/test_build_modes.py`, `tests/wrapper/fortran/multi_source/test_multi_source_builds.py` | | Semantic IR to codegen AST | `x2py/semantics/ir2ast.py` | `tests/semantics/test_ir2ast.py`, `tests/wrapper/` | | Fortran-to-C bridge and CPython binding | `x2py/codegen/bridges/fortran_to_c.py`, `x2py/codegen/bindings/c_to_python.py` | `tests/wrapper/` subject suites | -| Native compilation and runtime support | `x2py/compiling/`, `x2py/stdlib/x2py_runtime/` | `tests/wrapper/fortran/native_build/test_runtime_abi.py`, `tests/wrapper/fortran/native_build/test_build_modes.py` | +| Native compilation and binding support | `x2py/compiling/`, `x2py/binding_support/` | `tests/wrapper/fortran/native_build/test_runtime_abi.py`, `tests/wrapper/fortran/native_build/test_build_modes.py` | | Public API exports | `x2py/__init__.py` | `tests/parser/test_parser_public_entrypoints.py`, `tests/parser/c/test_c_public_api_skeleton.py` | | Executable Markdown examples | `README.md`, `docs/*.md` | `tests/tools/test_documentation_examples.py` | @@ -729,7 +729,7 @@ The main ownership boundaries are: reference handling, and CPython wrapper construction; - `x2py/codegen/printers/{fcode,ccode,cpythoncode}.py`: source rendering only; - `x2py/compiling/`: compiler commands and shared-library linking; and -- `x2py/stdlib/x2py_runtime/`: native runtime support copied into each build. +- `x2py/binding_support/`: native binding support copied into each build. Do not move semantic ownership or projection policy into printers. Do not infer source dependencies: multi-source builds compile in caller order, and the first diff --git a/docs/old_docs/fortran_wrapper.md b/docs/old_docs/fortran_wrapper.md index 5a8d1cc1d..6f26b603d 100644 --- a/docs/old_docs/fortran_wrapper.md +++ b/docs/old_docs/fortran_wrapper.md @@ -107,7 +107,7 @@ ordered Fortran source files -> merged public wrapper module and collision-safe Python names -> codegen AST -> Fortran bind(C) bridge - -> C/CPython binding and x2py runtime support + -> C/CPython binding and native binding support -> compile user sources and generated sources -> link one Python extension module ``` @@ -115,7 +115,7 @@ ordered Fortran source files The Fortran bridge converts non-interoperable Fortran contracts into a stable C ABI. The generated C layer validates Python and NumPy objects, manages Python references and wrapper-owned temporaries, calls the bridge, and projects native -results onto the documented Python API. The runtime support supplies shared +results onto the documented Python API. The native binding support supplies shared array, error, allocation, and ownership helpers. Typical generated artifacts are: @@ -124,7 +124,7 @@ Typical generated artifacts are: | --- | --- | | `bind_c__wrapper.f90` | Fortran-to-C ABI bridge | | `_wrapper.c` and `.h` | CPython extension binding | -| `x2py_runtime/` | Shared native runtime support | +| `binding_support/` | Shared native binding support | | user and generated `.o`/`.mod` files | Native build intermediates | | `..so` | Importable extension on Linux | @@ -1320,7 +1320,7 @@ python3 -m x2py mesh.f90 solver.f90 --makefile --out-dir build --json make -f build/Makefile.x2py -j4 X2PY_FFLAGS=-O3 X2PY_CFLAGS=-O3 ``` -The Makefile covers user sources, generated wrappers, runtime support, and the +The Makefile covers user sources, generated wrappers, native binding support, and the shared-library link. It records resolved compilers and exposes `FC`, `CC`, `X2PY_LD`, `X2PY_FFLAGS`, `X2PY_CFLAGS`, and `X2PY_LDFLAGS`. User Fortran sources are conservatively chained in supplied order; independent generated C diff --git a/docs/old_docs/semantics.md b/docs/old_docs/semantics.md index ae9c6a046..693546a5d 100644 --- a/docs/old_docs/semantics.md +++ b/docs/old_docs/semantics.md @@ -849,7 +849,7 @@ only when a real array storage contract is known. ## Design Proposal: Self-Contained C Semantic `.pyi` Runtime Contract -> **Status: design only, not implemented runtime support.** x2py currently +> **Status: design only, not implemented native binding support.** x2py currently > parses C, converts the supported subset to semantic IR, emits and loads > semantic `.pyi`, and reports readiness. It does not currently generate, > lower, compile, or execute C wrappers. Every runtime behavior, wrapper error, diff --git a/docs/old_docs/tutorial.md b/docs/old_docs/tutorial.md index 75d4b4e44..14b9b0e6b 100644 --- a/docs/old_docs/tutorial.md +++ b/docs/old_docs/tutorial.md @@ -52,7 +52,7 @@ ordered Fortran sources -> semantic IR -> codegen AST -> generated Fortran bind(C) bridge - -> generated C/CPython binding and runtime support + -> generated C/CPython binding and native binding support -> compiled and linked Python extension ``` @@ -279,7 +279,7 @@ The build lowers semantic IR through two native layers: 2. A generated C/CPython binding validates Python objects, manages ownership and references, invokes the bridge, and creates Python or NumPy results. -The x2py runtime support is compiled with those generated sources. The final +The native binding support is compiled with those generated sources. The final link combines user objects, the Fortran bridge, the CPython binding, and the runtime into one extension module. Generated sources are build artifacts; the public behavior is the documented semantic and wrapper contract. @@ -676,7 +676,7 @@ Use x2py for the behavior implemented and tested today: - generated and compiled CPython extensions from one or more ordered Fortran source files; -- generated Fortran `bind(C)` bridges, C/CPython bindings, and runtime support +- generated Fortran `bind(C)` bridges, C/CPython bindings, and native binding support for the contracts in the [Fortran wrapper guide](fortran_wrapper.md); - wrapper-relevant Fortran and C source-fact extraction; - compiler-preprocessed CLI workflows; diff --git a/docs/old_docs/wrapper_design_notes.md b/docs/old_docs/wrapper_design_notes.md index 9cfdffbf5..de0228768 100644 --- a/docs/old_docs/wrapper_design_notes.md +++ b/docs/old_docs/wrapper_design_notes.md @@ -62,8 +62,8 @@ functions, variables, structs, enums, typedefs, constants, arrays, pointers, callbacks, and the metadata needed for readiness decisions. Generated CPython extension builds copy their bundled C/Python support sources -into an `x2py_runtime/` directory inside the build output. The generated C -extension includes `x2py_runtime/python_runtime.h`. These files are an +into a `binding_support/` directory inside the build output. The generated C +extension includes `binding_support/x2py_binding.h`. These files are an implementation detail of the generated extension, but their names are intentionally x2py-specific so they do not look like user source or a generic C wrapper. @@ -91,7 +91,7 @@ attributes. Verbose wrapper builds should print the exact compiler command lines they run, not only the source or target being compiled. The printed command should be shell-quoted so users can copy it to reproduce object compilation, generated -wrapper compilation, runtime support compilation, and final shared-library +wrapper compilation, native binding support compilation, and final shared-library linking. Normal C parsing uses a real compiler preprocessor first. Macro expansion, diff --git a/docs/user/examples/recipes/generate-editable-makefile.md b/docs/user/examples/recipes/generate-editable-makefile.md index b26436828..f5e90e786 100644 --- a/docs/user/examples/recipes/generate-editable-makefile.md +++ b/docs/user/examples/recipes/generate-editable-makefile.md @@ -22,7 +22,7 @@ python3 -m x2py tests/data/fortran/wrapper/fruntime_abi_f90.f90 \ --json ``` -This writes generated wrapper sources, runtime support, dependency files, and +This writes generated wrapper sources, native binding support, dependency files, and `build/fruntime_abi/Makefile.x2py`. For a semantic `.pyi` contract with native implementation sources, use the same diff --git a/docs/user/getting-started/beginner-workflow.md b/docs/user/getting-started/beginner-workflow.md index a128d1ae5..f42a46bd2 100644 --- a/docs/user/getting-started/beginner-workflow.md +++ b/docs/user/getting-started/beginner-workflow.md @@ -38,7 +38,7 @@ under version control. Do not commit `build/`. @@ -109,7 +109,7 @@ You normally do not need to open generated files. When debugging, expect | Artifact | Purpose | | --- | --- | -| `x2py_runtime/` | shared runtime support sources | +| `binding_support/` | shared native binding support sources | | `.o` and `.mod` files | native intermediates | | `.` | importable extension | diff --git a/docs/user/getting-started/verification.md b/docs/user/getting-started/verification.md index f78cdae47..cdb3c6704 100644 --- a/docs/user/getting-started/verification.md +++ b/docs/user/getting-started/verification.md @@ -77,10 +77,10 @@ python3 -m x2py scale.f90 \ The command must create: - an importable `scale` extension under `build/verify`; and -- generated native bridge, object, runtime-support, and extension files. +- generated native bridge, object, native-support, and extension files. Import the extension from that build directory: diff --git a/docs/user/guide/data-types.md b/docs/user/guide/data-types.md index a29b191e4..c53e436ec 100644 --- a/docs/user/guide/data-types.md +++ b/docs/user/guide/data-types.md @@ -238,7 +238,7 @@ finalization behavior. ## Unsupported Widths And Forms The semantic format can represent names such as `Float128` and `Complex256`, -but representation in a `.pyi` file is not a runtime support claim. Current +but representation in a `.pyi` file is not a native binding support claim. Current Fortran wrapper generation blocks: - real storage wider than 64 bits; diff --git a/docs/user/guide/fortran-wrapper.md b/docs/user/guide/fortran-wrapper.md index 30b75d42d..e57f91806 100644 --- a/docs/user/guide/fortran-wrapper.md +++ b/docs/user/guide/fortran-wrapper.md @@ -142,7 +142,7 @@ ordered Fortran source files -> post-IR policy completion -> ordered wrapper plan preserving native module namespaces and ABI slots -> direct Fortran bind(C) bridge lowering - -> direct C/CPython binding lowering and x2py runtime support + -> direct C/CPython binding lowering and native binding support -> compile user sources and generated sources -> link one Python extension module ``` @@ -162,7 +162,7 @@ bridge generators dispatch them directly into emitted source. The Fortran bridge converts non-interoperable Fortran contracts into a stable C ABI. The generated C layer validates Python and NumPy objects, manages Python references and wrapper-owned temporaries, calls the bridge, and projects native -results onto the documented Python API. The runtime support supplies shared +results onto the documented Python API. The native binding support supplies shared array, error, allocation, and ownership helpers. X2PY_C_DOCS_END --> @@ -170,7 +170,7 @@ Typical generated artifacts are: | Artifact | Purpose | | --- | --- | -| `x2py_runtime/` | Shared native runtime support | +| `binding_support/` | Shared native binding support | | user and generated `.o`/`.mod` files | Native build intermediates | | `..so` | Importable extension on Linux | @@ -289,7 +289,7 @@ compiler and linker command. It first announces binding, bridge, and header source-text generation on separate lines without paths, because those files do not exist yet. Each line is printed immediately before its separate lowering and printing operation, followed by `Timing: ...` for that operation. It then announces each written artifact with its output path -(`Write bridge source: ...` and `Write runtime support: ...`), each native, bridge, runtime, and binding compilation with its +(`Write bridge source: ...` and `Write native support: ...`), each native, bridge, native-support, and binding compilation with its source and object path (`Compile bridge source: source -> object`), and the final extension path before linking (`Create shared library: ...`). The exact shell-escaped command follows each compilation or link announcement, so it can @@ -1877,7 +1877,7 @@ make -f build/Makefile.x2py -j4 X2PY_FFLAGS=-O3 X2PY_CFLAGS=-O3 ``` X2PY_C_DOCS_END --> diff --git a/pyproject.toml b/pyproject.toml index a9414c999..133d5605f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,7 +32,7 @@ where = ["."] include = ["x2py*"] [tool.setuptools.package-data] -"x2py.stdlib" = ["x2py_runtime/*"] +"x2py.binding_support" = ["*.c", "*.h"] [project.scripts] x2py = "x2py.cli:main" diff --git a/tests/docs/test_structure.py b/tests/docs/test_structure.py index 62a0d1794..f819da1cc 100644 --- a/tests/docs/test_structure.py +++ b/tests/docs/test_structure.py @@ -248,9 +248,9 @@ "x2py/wrapper_codegen/printers/source_printers.py", "x2py/compiling/objects.py", "x2py/compiling/compilers.py", - "x2py/compiling/runtime_support.py", + "x2py/compiling/native_support.py", "x2py/naming/policy.py", - "x2py/stdlib/", + "x2py/binding_support/", ] SOURCE_NAVIGATION_PUBLIC_DOCS = [ "README.md", diff --git a/tests/pipeline/test_rendered_wrapper_artifact_build.py b/tests/pipeline/test_rendered_wrapper_artifact_build.py index b4d2ddcfe..ab73292ba 100644 --- a/tests/pipeline/test_rendered_wrapper_artifact_build.py +++ b/tests/pipeline/test_rendered_wrapper_artifact_build.py @@ -131,27 +131,27 @@ def scale(x: Float64) -> Float64: ... bridge_source = build_dir / "bind_c_plan_scalar_build_wrapper.f90" binding_source = build_dir / "plan_scalar_build_wrapper.c" header = build_dir / "plan_scalar_build_wrapper.h" - runtime_source = build_dir / "x2py_runtime" / "python_runtime.c" - runtime_object = build_dir / "x2py_runtime" / "python_runtime.o" + native_support_source = build_dir / "binding_support" / "x2py_binding.c" + native_support_object = build_dir / "binding_support" / "x2py_binding.o" assert bridge_source.read_text(encoding="utf-8") == rendered.sources[0].text assert binding_source.read_text(encoding="utf-8") == rendered.sources[1].text assert header.read_text(encoding="utf-8") == rendered.sources[2].text - assert runtime_source.exists() - assert runtime_object.exists() + assert native_support_source.exists() + assert native_support_object.exists() assert [object_file.language for object_file, _verbose in compiler.compiled] == ["fortran", "c", "c"] bridge_obj = compiler.compiled[0][0] - runtime_obj = compiler.compiled[1][0] + native_support_obj = compiler.compiled[1][0] binding_obj = compiler.compiled[2][0] assert native_dir in bridge_obj.include_dirs - assert runtime_obj.tools == frozenset({"python"}) + assert native_support_obj.tools == frozenset({"python"}) assert binding_obj.tools == frozenset({"python"}) assert compiler.linked == ( "plan_scalar_build", tmp_path / "extension", "fortran", - (native_obj, bridge_obj, runtime_obj, binding_obj), + (native_obj, bridge_obj, native_support_obj, binding_obj), ("-lm",), (native_dir,), (), @@ -171,8 +171,8 @@ def scale(x: Float64) -> Float64: ... assert result.generated_sources == (bridge_source, binding_source, header) assert bridge_obj.object_path in result.generated_files assert binding_obj.object_path in result.generated_files - assert runtime_source in result.generated_files - assert runtime_object in result.generated_files + assert native_support_source in result.generated_files + assert native_support_object in result.generated_files step_lines = [ line.removeprefix(">> ") for line in capsys.readouterr().out.splitlines() @@ -182,26 +182,26 @@ def scale(x: Float64) -> Float64: ... f"Write bridge source: {bridge_source}", f"Write binding source: {binding_source}", f"Write binding header: {header}", - f"Write runtime support: {build_dir / 'x2py_runtime'}", + f"Write native support: {build_dir / 'binding_support'}", f"Compile bridge source: {bridge_source} -> {bridge_obj.object_path}", - f"Compile runtime source: {runtime_source} -> {runtime_obj.object_path}", + f"Compile native support source: {native_support_source} -> {native_support_obj.object_path}", f"Compile binding source: {binding_source} -> {binding_obj.object_path}", f"Create shared library: {result.shared_library}", ] -def test_build_rendered_wrapper_extension_rejects_unknown_runtime_support_key(tmp_path: Path): +def test_build_rendered_wrapper_extension_rejects_unknown_native_support_key(tmp_path: Path): rendered = RenderedGeneratedWrapperArtifacts( artifacts=GeneratedWrapperArtifacts( module_name="bad_runtime", binding_sources=(Path("bad_runtime_wrapper.c"),), - runtime_support_keys=("unknown_runtime",), + native_support_keys=("unknown_native_support",), ), sources=(GeneratedSourceFile(Path("bad_runtime_wrapper.c"), "PyObject *unused;\n"),), extension_init_name="PyInit_bad_runtime", ) - with pytest.raises(ValueError, match="Unsupported wrapper runtime support key"): + with pytest.raises(ValueError, match="Unsupported wrapper native support key"): _build_rendered_wrapper_extension( rendered, output_dir=tmp_path, diff --git a/tests/wrapper/fortran/_support.py b/tests/wrapper/fortran/_support.py index dda8cf6cf..e84f5c0a5 100644 --- a/tests/wrapper/fortran/_support.py +++ b/tests/wrapper/fortran/_support.py @@ -93,7 +93,7 @@ def _build_and_import(source_template: Path, workdir: Path, expected_generated_s assert shared_library.parent == workdir assert {Path(path).name for path in payload["generated_sources"]} == expected_generated_sources generated_files = [Path(path) for path in payload["generated_files"]] - assert any(path.name == "python_runtime.c" and path.parent.name == "x2py_runtime" for path in generated_files) + assert any(path.name == "x2py_binding.c" and path.parent.name == "binding_support" for path in generated_files) sys.modules.pop(module_name, None) sys.path.insert(0, str(workdir)) diff --git a/tests/wrapper/fortran/build_from_source/test_build_modes.py b/tests/wrapper/fortran/build_from_source/test_build_modes.py index c423b4aef..5779717ae 100644 --- a/tests/wrapper/fortran/build_from_source/test_build_modes.py +++ b/tests/wrapper/fortran/build_from_source/test_build_modes.py @@ -254,7 +254,7 @@ def test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper(tmp assert result.compiled is True assert result.build_makefile is None assert any( - path.name == "python_runtime.c" and path.parent.name == "x2py_runtime" for path in result.generated_files + path.name == "x2py_binding.c" and path.parent.name == "binding_support" for path in result.generated_files ) sys.modules.pop(result.module_name, None) diff --git a/tests/wrapper/fortran/scalars/test_verified_baseline.py b/tests/wrapper/fortran/scalars/test_verified_baseline.py index 0aa15e05b..21ee78d9b 100644 --- a/tests/wrapper/fortran/scalars/test_verified_baseline.py +++ b/tests/wrapper/fortran/scalars/test_verified_baseline.py @@ -95,7 +95,7 @@ def test_fmath_scalar_sources_use_canonical_wrapper_plan( assert {path.name for path in wrapper_result.generated_sources} == expected_generated_sources assert any(path.name == f"{source.stem}_wrapper.h" for path in wrapper_result.generated_files) assert any( - path.name == "python_runtime.c" and path.parent.name == "x2py_runtime" + path.name == "x2py_binding.c" and path.parent.name == "binding_support" for path in wrapper_result.generated_files ) assert wrapper_result.compiled is True diff --git a/tests/wrapper_codegen/test_phase0b_contracts.py b/tests/wrapper_codegen/test_phase0b_contracts.py index f863732cb..bc99b6367 100644 --- a/tests/wrapper_codegen/test_phase0b_contracts.py +++ b/tests/wrapper_codegen/test_phase0b_contracts.py @@ -192,7 +192,7 @@ def test_generated_wrapper_artifacts_keep_compile_and_link_ownership_out_of_the_ bridge_sources=(Path("bind_c_demo.f90"),), binding_sources=(Path("demo.c"),), header_files=(Path("demo.h"),), - runtime_support_keys=("python_runtime",), + native_support_keys=("binding_support",), ) assert artifacts.source_files == (Path("bind_c_demo.f90"), Path("demo.c")) @@ -203,6 +203,6 @@ def test_generated_wrapper_artifacts_keep_compile_and_link_ownership_out_of_the_ "bridge_sources", "binding_sources", "header_files", - "runtime_support_keys", + "native_support_keys", "required_headers", } diff --git a/tests/wrapper_codegen/test_phase4_scalar_module_variables.py b/tests/wrapper_codegen/test_phase4_scalar_module_variables.py index 782baea74..c272a154f 100644 --- a/tests/wrapper_codegen/test_phase4_scalar_module_variables.py +++ b/tests/wrapper_codegen/test_phase4_scalar_module_variables.py @@ -144,7 +144,7 @@ def test_module_setter_assignment_mismatch_fails_before_backend_preflight_or_low fortran_generator.require_supported.assert_not_called() c_generator.visit.assert_not_called() fortran_generator.visit.assert_not_called() - c_generator.requires_runtime_support.assert_not_called() + c_generator.requires_native_support.assert_not_called() c_printer.doprint.assert_not_called() fortran_printer.doprint.assert_not_called() diff --git a/x2py/README.md b/x2py/README.md index 37394fe18..dd9f56416 100644 --- a/x2py/README.md +++ b/x2py/README.md @@ -17,7 +17,7 @@ jumping directly into generated-code internals. | `parsers/` | Parser namespace containing the `c`, `fortran`, and semantic `.pyi` frontends. | | `semantics/` | Language-neutral semantic IR, policy completion, readiness, and `.pyi` conversion. | | `wrapper_codegen/` | Canonical wrapper plans, direct native bridge/binding generation, and source printers. | -| `compiling/` | Native compiler objects, wrapper compilation, runtime support installation, and linking. | +| `compiling/` | Native compiler objects, wrapper compilation, native support installation, and linking. | | `utilities/` | Small domain-neutral helpers, including class visitor dispatch. | The package root contains the public entrypoint modules plus the shared diff --git a/x2py/binding_support/__init__.py b/x2py/binding_support/__init__.py new file mode 100644 index 000000000..3494e0ac6 --- /dev/null +++ b/x2py/binding_support/__init__.py @@ -0,0 +1 @@ +"""Bundled C sources used by generated CPython bindings.""" diff --git a/x2py/stdlib/x2py_runtime/python_runtime.c b/x2py/binding_support/x2py_binding.c similarity index 99% rename from x2py/stdlib/x2py_runtime/python_runtime.c rename to x2py/binding_support/x2py_binding.c index 530dd2bd7..fa528523c 100644 --- a/x2py/stdlib/x2py_runtime/python_runtime.c +++ b/x2py/binding_support/x2py_binding.c @@ -1,4 +1,4 @@ -#include "python_runtime.h" +#include "x2py_binding.h" diff --git a/x2py/stdlib/x2py_runtime/python_runtime.h b/x2py/binding_support/x2py_binding.h similarity index 99% rename from x2py/stdlib/x2py_runtime/python_runtime.h rename to x2py/binding_support/x2py_binding.h index bb2a33ba1..0e66ad463 100644 --- a/x2py/stdlib/x2py_runtime/python_runtime.h +++ b/x2py/binding_support/x2py_binding.h @@ -6,8 +6,8 @@ * - Functions which test the type of PythonObjects */ -#ifndef X2PY_PYTHON_RUNTIME_H -# define X2PY_PYTHON_RUNTIME_H +#ifndef X2PY_BINDING_H +# define X2PY_BINDING_H # define PY_SSIZE_T_CLEAN # include "Python.h" diff --git a/x2py/compiling/README.md b/x2py/compiling/README.md index deca750cb..8cc3bf5f6 100644 --- a/x2py/compiling/README.md +++ b/x2py/compiling/README.md @@ -1,7 +1,7 @@ # Compiling Package This package owns native compiler command construction, compile objects, -generated wrapper compilation, runtime support installation, and shared-library +generated wrapper compilation, native support installation, and shared-library linking. ## Entry Points @@ -11,7 +11,7 @@ linking. | `objects.py` | Explicit source-to-object compilation inputs. | | `compilers.py` | Compiler command execution and tool lookup helpers. | | `compiler_profiles.py` | Built-in vendor compiler profiles and Python-link settings. | -| `runtime_support.py` | Writing runtime support and declaring its object inputs. | +| `native_support.py` | Writing native binding support and declaring its object inputs. | Generated-wrapper object assembly and shared-library orchestration live in `x2py/pipeline/build.py`, where the canonical rendered wrapper artifacts are @@ -25,7 +25,7 @@ native source files -> native object files generated Fortran bridge -> bridge object files -generated C/CPython binding and its runtime support +generated C/CPython binding and its native support -> runtime and binding object files all explicit object files and link inputs -> linked Python extension diff --git a/x2py/compiling/runtime_support.py b/x2py/compiling/native_support.py similarity index 53% rename from x2py/compiling/runtime_support.py rename to x2py/compiling/native_support.py index 041394562..c83738b15 100644 --- a/x2py/compiling/runtime_support.py +++ b/x2py/compiling/native_support.py @@ -1,4 +1,4 @@ -"""Install the bundled native runtime used by generated CPython wrappers.""" +"""Install bundled native binding support for generated CPython wrappers.""" from pathlib import Path import shutil @@ -6,17 +6,17 @@ from filelock import FileLock import numpy as np -import x2py.stdlib as stdlib_folder +import x2py.binding_support as binding_support_folder from .objects import ObjectFile -_RUNTIME_IMPORT = "x2py_runtime" -_RUNTIME_SOURCE = Path(stdlib_folder.__file__).parent / _RUNTIME_IMPORT +_NATIVE_SUPPORT_IMPORT = "binding_support" +_NATIVE_SUPPORT_SOURCE = Path(binding_support_folder.__file__).parent def _numpy_version_header() -> str: - """Return NumPy API version guards for the bundled native runtime.""" + """Return NumPy API version guards for the bundled native support.""" maximum_supported = [1, 19] current = [int(value) for value in np.version.version.split(".")[:2]] major, minor = min(maximum_supported, current) @@ -26,17 +26,17 @@ def _numpy_version_header() -> str: return header -def install_runtime_support(imports, *, x2py_dirpath, verbose: bool | int = False) -> tuple[ObjectFile, ...]: - """Write runtime support and return its explicit compilation inputs.""" - if not any(name == _RUNTIME_IMPORT or name.startswith(f"{_RUNTIME_IMPORT}/") for name in imports): +def install_native_support(imports, *, x2py_dirpath, verbose: bool | int = False) -> tuple[ObjectFile, ...]: + """Write native binding support and return its explicit compilation inputs.""" + if not any(name == _NATIVE_SUPPORT_IMPORT or name.startswith(f"{_NATIVE_SUPPORT_IMPORT}/") for name in imports): return () - destination = Path(x2py_dirpath) / _RUNTIME_IMPORT + destination = Path(x2py_dirpath) / _NATIVE_SUPPORT_IMPORT if verbose: - print(f">> Write runtime support: {destination}") + print(f">> Write native support: {destination}") with FileLock(str(destination.with_suffix(".lock"))): shutil.rmtree(destination, ignore_errors=True) - shutil.copytree(_RUNTIME_SOURCE, destination) + shutil.copytree(_NATIVE_SUPPORT_SOURCE, destination) (destination / "numpy_version.h").write_text( _numpy_version_header(), @@ -44,8 +44,8 @@ def install_runtime_support(imports, *, x2py_dirpath, verbose: bool | int = Fals ) return ( ObjectFile( - source=destination / "python_runtime.c", - object_path=destination / "python_runtime.o", + source=destination / "x2py_binding.c", + object_path=destination / "x2py_binding.o", language="c", include_dirs=(destination,), tools=frozenset({"python"}), diff --git a/x2py/parsers/fortran/README.md b/x2py/parsers/fortran/README.md index 1f465c714..33baf9f07 100644 --- a/x2py/parsers/fortran/README.md +++ b/x2py/parsers/fortran/README.md @@ -28,6 +28,6 @@ callers may also use the stable parser functions and models exported from the - Fixture suite: `tests/parsing/fortran/test_fortran_fixture_suite.py` - Semantic handoff tests: `tests/semantics/conversion/fortran/` -Parser support alone does not establish wrapper runtime support. Wrapper +Parser support alone does not establish native binding support. Wrapper features need semantic lowering, readiness policy, codegen, compilation, and runtime tests. diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 8d13a84f6..5ae83888a 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -12,7 +12,7 @@ from x2py.compiling.objects import ObjectFile from x2py.compiling.compilers import Compiler, get_condaless_search_path -from x2py.compiling.runtime_support import install_runtime_support +from x2py.compiling.native_support import install_native_support from x2py.parsers.fortran.parser import parse_fortran_project from x2py.probes.fortran_types import evaluate_fortran_type_facts, evaluate_fortran_type_requirements from x2py.pipeline.preprocessing import PreprocessingConfig, preprocess_source @@ -66,8 +66,8 @@ ".for": "fortran", ".ftn": "fortran", } -_RENDERED_WRAPPER_RUNTIME_IMPORTS = { - "python_runtime": ("x2py_runtime/python_runtime",), +_RENDERED_WRAPPER_NATIVE_SUPPORT_IMPORTS = { + "binding_support": ("binding_support/x2py_binding",), } @@ -295,9 +295,9 @@ def _expected_generated_files( output_dir / f"{module_name}_wrapper.o", shared_library, ] - runtime_support_dir = output_dir / "x2py_runtime" - if runtime_support_dir.is_dir(): - candidates.extend(sorted(path for path in runtime_support_dir.rglob("*") if path.is_file())) + native_support_dir = output_dir / "binding_support" + if native_support_dir.is_dir(): + candidates.extend(sorted(path for path in native_support_dir.rglob("*") if path.is_file())) return tuple(path for path in candidates if path.exists()) @@ -363,14 +363,14 @@ def _rendered_wrapper_source_language(path: Path) -> str: raise ValueError(f"Unsupported rendered wrapper source suffix: {path}") from None -def _rendered_wrapper_runtime_imports(runtime_support_keys: Iterable[str]) -> tuple[str, ...]: - """Return runtime-support import keys consumed by the existing installer.""" +def _rendered_wrapper_native_support_imports(native_support_keys: Iterable[str]) -> tuple[str, ...]: + """Return native-support import keys consumed by the support installer.""" imports: list[str] = [] - for key in runtime_support_keys: + for key in native_support_keys: try: - imports.extend(_RENDERED_WRAPPER_RUNTIME_IMPORTS[key]) + imports.extend(_RENDERED_WRAPPER_NATIVE_SUPPORT_IMPORTS[key]) except KeyError: - raise ValueError(f"Unsupported wrapper runtime support key: {key!r}") from None + raise ValueError(f"Unsupported wrapper native support key: {key!r}") from None return tuple(imports) @@ -495,9 +495,9 @@ def _build_rendered_wrapper_extension( ) ), ) - runtime_imports = _rendered_wrapper_runtime_imports(rendered.artifacts.runtime_support_keys) - runtime_objects = install_runtime_support( - runtime_imports, + native_support_imports = _rendered_wrapper_native_support_imports(rendered.artifacts.native_support_keys) + native_support_objects = install_native_support( + native_support_imports, x2py_dirpath=str(output_path), verbose=verbose, ) @@ -509,8 +509,8 @@ def _build_rendered_wrapper_extension( ) _compile_object_stage( compiler, - runtime_objects, - label="Compile runtime source", + native_support_objects, + label="Compile native support source", verbose=verbose, ) _compile_object_stage( @@ -525,7 +525,7 @@ def _build_rendered_wrapper_extension( module_name=rendered.artifacts.module_name, output_dir=shared_output_path, language=_rendered_wrapper_link_language(bridge_objects, binding_objects), - objects=(*tuple(native_dependencies), *bridge_objects, *runtime_objects, *binding_objects), + objects=(*tuple(native_dependencies), *bridge_objects, *native_support_objects, *binding_objects), link_args=tuple(native_link_args), library_dirs=resolved_native_build_plan.library_dirs, flags=_compiler_flags(wrapper_c_flags), diff --git a/x2py/pipeline/wrapper_artifacts.py b/x2py/pipeline/wrapper_artifacts.py index 874840977..2e6fa7547 100644 --- a/x2py/pipeline/wrapper_artifacts.py +++ b/x2py/pipeline/wrapper_artifacts.py @@ -30,7 +30,7 @@ class GeneratedWrapperArtifacts(StageRecord): bridge_sources: tuple[Path, ...] = () binding_sources: tuple[Path, ...] = () header_files: tuple[Path, ...] = () - runtime_support_keys: tuple[str, ...] = () + native_support_keys: tuple[str, ...] = () required_headers: tuple[str, ...] = () @property diff --git a/x2py/stdlib/__init__.py b/x2py/stdlib/__init__.py deleted file mode 100644 index fbff30a09..000000000 --- a/x2py/stdlib/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Runtime support files used by generated extension wrappers.""" diff --git a/x2py/wrapper_codegen/c/binding.py b/x2py/wrapper_codegen/c/binding.py index 811951cd4..fe3281cdb 100644 --- a/x2py/wrapper_codegen/c/binding.py +++ b/x2py/wrapper_codegen/c/binding.py @@ -620,12 +620,12 @@ def binding_module(self, plan: ModulePlan) -> CModule: if surface.python_names } functions = tuple(function for namespace in plan.namespaces for function in self.visit(namespace)) - needs_runtime = self.requires_runtime_support(plan) + needs_native_support = self.requires_native_support(plan) needs_free = self._module_needs_allocator(plan) return CModule( name=f"{plan.binding.owner_path}_wrapper", - defines=self._module_defines(needs_runtime), - includes=self._module_includes(plan, needs_runtime, needs_free), + defines=self._module_defines(needs_native_support), + includes=self._module_includes(plan, needs_native_support, needs_free), declarations=self._module_declarations(plan), functions=( *self._module_allocator_functions(needs_free), @@ -638,7 +638,7 @@ def binding_module(self, plan: ModulePlan) -> CModule: *self._derived_handle_operation_functions(plan), *self._native_array_operation_functions(plan), *functions, - self._module_init(plan, needs_runtime), + self._module_init(plan, needs_native_support), ), ) @@ -657,8 +657,8 @@ def _visit_NamespacePlan(self, plan: NamespacePlan) -> tuple[CFunction, ...]: *(function for variable in plan.variables for function in self.visit(variable)), ) - def requires_runtime_support(self, plan: ModulePlan) -> bool: - """Return whether module lowering consumes NumPy/runtime helpers.""" + def requires_native_support(self, plan: ModulePlan) -> bool: + """Return whether module lowering consumes bundled native helpers.""" return ( bool(tuple(self._variables(plan))) or any(function.arguments or function.results for function in self._functions(plan)) @@ -695,16 +695,16 @@ def _function_needs_allocator(self, function: FunctionPlan) -> bool: ) ) - def _module_defines(self, needs_runtime: bool) -> tuple[CMacroDefinition, ...]: + def _module_defines(self, needs_native_support: bool) -> tuple[CMacroDefinition, ...]: """Return compile-time definitions selected by assembled module needs.""" - if not needs_runtime: + if not needs_native_support: return () return (CMacroDefinition("PY_ARRAY_UNIQUE_SYMBOL", "CWRAPPER_ARRAY_API"),) def _module_includes( self, plan: ModulePlan, - needs_runtime: bool, + needs_native_support: bool, needs_free: bool, ) -> tuple[CInclude, ...]: """Return dependency-closed includes for one assembled C module.""" @@ -721,7 +721,7 @@ def _module_includes( else () ), *(CInclude(header) for header in plan.required_headers), - *self._module_runtime_includes(needs_runtime), + *self._module_native_support_includes(needs_native_support), CInclude(f"{plan.binding.owner_path}_wrapper.h", system=False), ) @@ -786,14 +786,14 @@ def _module_uses_derived_origin_ops(self, plan: ModulePlan) -> bool: """Return whether runtime-selected module origins need typed operations.""" return any(variable.derived is not None for variable in self._variables(plan)) - def _module_runtime_includes(self, required: bool) -> tuple[CInclude, ...]: - """Return NumPy/runtime includes when generated nodes consume them.""" + def _module_native_support_includes(self, required: bool) -> tuple[CInclude, ...]: + """Return bundled native-support includes consumed by generated nodes.""" if not required: return () return ( - CInclude("x2py_runtime/numpy_version.h", system=False), + CInclude("binding_support/numpy_version.h", system=False), CInclude("numpy/arrayobject.h"), - CInclude("x2py_runtime/python_runtime.h", system=False), + CInclude("binding_support/x2py_binding.h", system=False), ) # Immediate callback runtime. @@ -10288,14 +10288,14 @@ def _module_def(self, module: ModulePlan, namespace: NamespacePlan) -> CModuleDe f"{owner}_{symbol}_methods", ) - def _module_init(self, plan: ModulePlan, needs_runtime: bool) -> CFunction: + def _module_init(self, plan: ModulePlan, needs_native_support: bool) -> CFunction: module_name = plan.binding.owner_path root_namespace = self._namespace(plan, ()) return CFunction( f"PyInit_{module_name}", "PyMODINIT_FUNC", body=( - *((CExpressionStatement(CodeExpression("import_array()")),) if needs_runtime else ()), + *((CExpressionStatement(CodeExpression("import_array()")),) if needs_native_support else ()), CDeclaration( "mod", "PyObject *", diff --git a/x2py/wrapper_codegen/generator.py b/x2py/wrapper_codegen/generator.py index d483afc8c..9469ba922 100644 --- a/x2py/wrapper_codegen/generator.py +++ b/x2py/wrapper_codegen/generator.py @@ -158,7 +158,7 @@ def generate( c_source, c_header_source, fortran_source, - runtime_support_keys=(("python_runtime",) if self._c_generator.requires_runtime_support(plan) else ()), + native_support_keys=(("binding_support",) if self._c_generator.requires_native_support(plan) else ()), required_headers=plan.required_headers, ) @@ -4082,7 +4082,7 @@ def _rendered_artifacts( c_source: str, c_header: str, fortran_source: str, - runtime_support_keys: tuple[str, ...], + native_support_keys: tuple[str, ...], required_headers: tuple[str, ...], ) -> RenderedGeneratedWrapperArtifacts: artifacts = GeneratedWrapperArtifacts( @@ -4090,7 +4090,7 @@ def _rendered_artifacts( bridge_sources=(Path(f"bind_c_{module_name}_wrapper.f90"),), binding_sources=(Path(f"{module_name}_wrapper.c"),), header_files=(Path(f"{module_name}_wrapper.h"),), - runtime_support_keys=runtime_support_keys, + native_support_keys=native_support_keys, required_headers=required_headers, ) return RenderedGeneratedWrapperArtifacts( From 80c6f00cfab686a83cfbcd5284c28ce25dbfd987 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 22:02:48 +0100 Subject: [PATCH 27/30] add helpers in binding support and remove pyccel notice --- THIRD_PARTY_NOTICES.md | 33 - .../maintainer/design/wrapper-design-notes.md | 8 + docs/old_docs/wrapper_design_notes.md | 8 + .../test_native_binding_support.py | 36 + .../test_phase1a_wrapper_assembly.py | 6 +- .../test_phase1b_scalar_input_conversion.py | 30 +- .../test_phase2a_scalar_results.py | 18 +- .../test_phase2b_hidden_scalar_outputs.py | 2 +- .../test_phase2e_scalar_boundaries.py | 4 +- .../test_phase2f_multiple_scalar_results.py | 4 +- ...st_phase3_scalar_presence_and_writeback.py | 2 +- .../test_phase5c_fixed_string_writeback.py | 2 +- .../test_phase6e_array_results.py | 4 +- ...ase6f_optional_assumed_character_arrays.py | 4 +- .../test_phase8_derived_types.py | 2 +- x2py/binding_support/x2py_binding.c | 706 +++--------------- x2py/binding_support/x2py_binding.h | 246 +----- x2py/wrapper_codegen/c/binding.py | 137 ++-- x2py/wrapper_codegen/nodes.py | 6 +- .../wrapper_codegen/primitive_scalar_types.py | 54 +- 20 files changed, 351 insertions(+), 961 deletions(-) delete mode 100644 THIRD_PARTY_NOTICES.md create mode 100644 tests/wrapper_codegen/test_native_binding_support.py diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md deleted file mode 100644 index 9310c4e5b..000000000 --- a/THIRD_PARTY_NOTICES.md +++ /dev/null @@ -1,33 +0,0 @@ -# Third-Party Notices - -The native CPython binding support in `x2py/binding_support/` contains code -adapted from the Pyccel project. The Python utilities, naming policy, -compilation package, semantic printer, and wrapper source printers are -independent implementations and are not covered by this attribution. - -Upstream project: - -The surviving-file comparison was refreshed against Pyccel commit -`f3361939fdd736474e510d90d502e7bee7157e12`. - -Pyccel is licensed under the MIT License: - -Copyright (c) 2017-2020, Pyccel Developers. - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/docs/maintainer/design/wrapper-design-notes.md b/docs/maintainer/design/wrapper-design-notes.md index cc22abb9f..7e8ae9b7b 100644 --- a/docs/maintainer/design/wrapper-design-notes.md +++ b/docs/maintainer/design/wrapper-design-notes.md @@ -82,6 +82,14 @@ extension includes `binding_support/x2py_binding.h`. These files are an implementation detail of the generated extension, but their names are intentionally x2py-specific so they do not look like user source or a generic C wrapper. + +The support header exposes a deliberately small `x2py_*` mechanical API: +scalar type matching, scalar unpacking, scalar creation as a Python or NumPy +object, and release of a bridge-owned allocation. The generated binding passes +the completed NumPy type, layout, ownership, and mutation decisions into those +operations. Native support must not infer a layout, accept a different dtype, +or choose ownership behavior from a value at runtime; those are completed +wrapper-plan decisions. X2PY_C_DOCS_END --> | `x2py/parsers/` | Public namespace for language and semantic `.pyi` frontends | child parser packages | `tests/parsing/`, parser references, semantic `.pyi` reference | | `x2py/parsers/fortran/` | Fortran lexer, recursive parser, models, type resolver, and parser CLI helpers | `parser.py`, `lexer.py`, `models.py`, `type_resolver.py`, `cli.py` | `tests/parser/`, `tests/parser/fortran/`, `docs/developer/fortran-parser-reference.md` | | `x2py/compiling/` | Native compile objects, compiler command execution, shared-library linking, and native support installation; wrapper build orchestration lives in `x2py/pipeline/build.py` | `objects.py`, `compilers.py`, `compiler_profiles.py`, `native_support.py` | `tests/wrapper/fortran/build_from_source/test_build_modes.py`, `tests/wrapper/fortran/build_from_source/test_runtime_abi.py` | -| `x2py/binding_support/` | Bundled native binding support copied into generated wrapper builds | support implementation and header | wrapper build tests | +| `x2py/binding_support/` | Bundled header-only native binding support copied into generated wrapper builds | support header | wrapper build tests | | `x2py/utilities/` | Small shared Python utilities | `strings.py`, `visitor.py` | tests that exercise callers | Verbose wrapper builds should print the exact compiler command lines they run, not only the source or target being compiled. The printed command should be shell-quoted so users can copy it to reproduce object compilation, generated -wrapper compilation, native binding support compilation, and final shared-library -linking. +wrapper compilation (including its header-only native binding support), and +final shared-library linking. diff --git a/docs/user/guide/packaging.md b/docs/user/guide/packaging.md index 084bf31dc..033a89aef 100644 --- a/docs/user/guide/packaging.md +++ b/docs/user/guide/packaging.md @@ -68,7 +68,7 @@ outside project-specific build scripts. ## Generated Artifacts An output directory can contain native object and module files, generated -wrapper sources, native binding support, build metadata, and the importable extension. +wrapper sources, header-only native binding support, build metadata, and the importable extension. These files are build products. Do not edit them as the source of the public API; change the native source or an intentional semantic `.pyi` contract. diff --git a/pyproject.toml b/pyproject.toml index 133d5605f..594cda9dc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,7 +32,7 @@ where = ["."] include = ["x2py*"] [tool.setuptools.package-data] -"x2py.binding_support" = ["*.c", "*.h"] +"x2py.binding_support" = ["*.h"] [project.scripts] x2py = "x2py.cli:main" diff --git a/tests/pipeline/test_rendered_wrapper_artifact_build.py b/tests/pipeline/test_rendered_wrapper_artifact_build.py index ab73292ba..a46e181cd 100644 --- a/tests/pipeline/test_rendered_wrapper_artifact_build.py +++ b/tests/pipeline/test_rendered_wrapper_artifact_build.py @@ -131,27 +131,23 @@ def scale(x: Float64) -> Float64: ... bridge_source = build_dir / "bind_c_plan_scalar_build_wrapper.f90" binding_source = build_dir / "plan_scalar_build_wrapper.c" header = build_dir / "plan_scalar_build_wrapper.h" - native_support_source = build_dir / "binding_support" / "x2py_binding.c" - native_support_object = build_dir / "binding_support" / "x2py_binding.o" + native_support_header = build_dir / "binding_support" / "x2py_binding.h" assert bridge_source.read_text(encoding="utf-8") == rendered.sources[0].text assert binding_source.read_text(encoding="utf-8") == rendered.sources[1].text assert header.read_text(encoding="utf-8") == rendered.sources[2].text - assert native_support_source.exists() - assert native_support_object.exists() - assert [object_file.language for object_file, _verbose in compiler.compiled] == ["fortran", "c", "c"] + assert native_support_header.exists() + assert [object_file.language for object_file, _verbose in compiler.compiled] == ["fortran", "c"] bridge_obj = compiler.compiled[0][0] - native_support_obj = compiler.compiled[1][0] - binding_obj = compiler.compiled[2][0] + binding_obj = compiler.compiled[1][0] assert native_dir in bridge_obj.include_dirs - assert native_support_obj.tools == frozenset({"python"}) assert binding_obj.tools == frozenset({"python"}) assert compiler.linked == ( "plan_scalar_build", tmp_path / "extension", "fortran", - (native_obj, bridge_obj, native_support_obj, binding_obj), + (native_obj, bridge_obj, binding_obj), ("-lm",), (native_dir,), (), @@ -171,8 +167,7 @@ def scale(x: Float64) -> Float64: ... assert result.generated_sources == (bridge_source, binding_source, header) assert bridge_obj.object_path in result.generated_files assert binding_obj.object_path in result.generated_files - assert native_support_source in result.generated_files - assert native_support_object in result.generated_files + assert native_support_header in result.generated_files step_lines = [ line.removeprefix(">> ") for line in capsys.readouterr().out.splitlines() @@ -184,7 +179,6 @@ def scale(x: Float64) -> Float64: ... f"Write binding header: {header}", f"Write native support: {build_dir / 'binding_support'}", f"Compile bridge source: {bridge_source} -> {bridge_obj.object_path}", - f"Compile native support source: {native_support_source} -> {native_support_obj.object_path}", f"Compile binding source: {binding_source} -> {binding_obj.object_path}", f"Create shared library: {result.shared_library}", ] diff --git a/tests/wrapper/fortran/_support.py b/tests/wrapper/fortran/_support.py index e84f5c0a5..2467d88be 100644 --- a/tests/wrapper/fortran/_support.py +++ b/tests/wrapper/fortran/_support.py @@ -93,7 +93,7 @@ def _build_and_import(source_template: Path, workdir: Path, expected_generated_s assert shared_library.parent == workdir assert {Path(path).name for path in payload["generated_sources"]} == expected_generated_sources generated_files = [Path(path) for path in payload["generated_files"]] - assert any(path.name == "x2py_binding.c" and path.parent.name == "binding_support" for path in generated_files) + assert any(path.name == "x2py_binding.h" and path.parent.name == "binding_support" for path in generated_files) sys.modules.pop(module_name, None) sys.path.insert(0, str(workdir)) diff --git a/tests/wrapper/fortran/build_from_source/test_build_modes.py b/tests/wrapper/fortran/build_from_source/test_build_modes.py index 5779717ae..a030f0057 100644 --- a/tests/wrapper/fortran/build_from_source/test_build_modes.py +++ b/tests/wrapper/fortran/build_from_source/test_build_modes.py @@ -254,7 +254,7 @@ def test_internal_preprocessing_mode_still_builds_importable_runtime_wrapper(tmp assert result.compiled is True assert result.build_makefile is None assert any( - path.name == "x2py_binding.c" and path.parent.name == "binding_support" for path in result.generated_files + path.name == "x2py_binding.h" and path.parent.name == "binding_support" for path in result.generated_files ) sys.modules.pop(result.module_name, None) diff --git a/tests/wrapper/fortran/scalars/test_verified_baseline.py b/tests/wrapper/fortran/scalars/test_verified_baseline.py index 21ee78d9b..74c9078a1 100644 --- a/tests/wrapper/fortran/scalars/test_verified_baseline.py +++ b/tests/wrapper/fortran/scalars/test_verified_baseline.py @@ -95,7 +95,7 @@ def test_fmath_scalar_sources_use_canonical_wrapper_plan( assert {path.name for path in wrapper_result.generated_sources} == expected_generated_sources assert any(path.name == f"{source.stem}_wrapper.h" for path in wrapper_result.generated_files) assert any( - path.name == "x2py_binding.c" and path.parent.name == "binding_support" + path.name == "x2py_binding.h" and path.parent.name == "binding_support" for path in wrapper_result.generated_files ) assert wrapper_result.compiled is True diff --git a/tests/wrapper_codegen/test_native_binding_support.py b/tests/wrapper_codegen/test_native_binding_support.py index 03879e326..7a7fcc314 100644 --- a/tests/wrapper_codegen/test_native_binding_support.py +++ b/tests/wrapper_codegen/test_native_binding_support.py @@ -8,9 +8,9 @@ SUPPORT_SOURCE = ROOT / "x2py" / "binding_support" / "x2py_binding.c" -def test_native_binding_support_exposes_only_the_small_x2py_api(): +def test_native_binding_support_is_header_only_and_exposes_the_small_x2py_api(): header = SUPPORT_HEADER.read_text(encoding="utf-8") - source = SUPPORT_SOURCE.read_text(encoding="utf-8") + assert not SUPPORT_SOURCE.exists() expected_api = ( "x2py_scalar_matches", @@ -21,7 +21,7 @@ def test_native_binding_support_exposes_only_the_small_x2py_api(): ) for name in expected_api: assert name in header - assert name in source + assert header.count("static inline") == len(expected_api) removed_compatibility_names = ( "PyInt32_to_Int32", @@ -33,4 +33,3 @@ def test_native_binding_support_exposes_only_the_small_x2py_api(): ) for name in removed_compatibility_names: assert name not in header - assert name not in source diff --git a/x2py/binding_support/__init__.py b/x2py/binding_support/__init__.py index 3494e0ac6..a8ea9724e 100644 --- a/x2py/binding_support/__init__.py +++ b/x2py/binding_support/__init__.py @@ -1 +1 @@ -"""Bundled C sources used by generated CPython bindings.""" +"""Bundled header-only C support used by generated CPython bindings.""" diff --git a/x2py/binding_support/x2py_binding.c b/x2py/binding_support/x2py_binding.c deleted file mode 100644 index 09c23b6aa..000000000 --- a/x2py/binding_support/x2py_binding.c +++ /dev/null @@ -1,147 +0,0 @@ -#include "x2py_binding.h" - -#include - -bool x2py_scalar_matches(PyObject *value, int numpy_type) -{ - switch (numpy_type) { - case NPY_BOOL: - return PyBool_Check(value) || PyArray_IsScalar(value, Bool); - case NPY_INT8: - return PyArray_IsScalar(value, Int8); - case NPY_INT16: - return PyArray_IsScalar(value, Int16); - case NPY_INT32: - return PyArray_IsScalar(value, Int); - case NPY_INT64: - return PyArray_IsScalar(value, Int64); - case NPY_FLOAT32: - return PyArray_IsScalar(value, Float); - case NPY_FLOAT64: - return PyArray_IsScalar(value, Double); - case NPY_COMPLEX64: - return PyArray_IsScalar(value, CFloat); - case NPY_COMPLEX128: - return PyArray_IsScalar(value, CDouble); - default: - return false; - } -} - -int x2py_scalar_unpack(PyObject *value, int numpy_type, void *destination) -{ - if (destination == NULL) { - PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar destination"); - return -1; - } - - if (numpy_type == NPY_BOOL) { - int truth = PyObject_IsTrue(value); - if (truth < 0) { - return -1; - } - *(bool *)destination = truth != 0; - return 0; - } - if (x2py_scalar_matches(value, numpy_type)) { - PyArray_ScalarAsCtype(value, destination); - return PyErr_Occurred() == NULL ? 0 : -1; - } - - switch (numpy_type) { - case NPY_INT8: - *(int8_t *)destination = (int8_t)PyLong_AsLong(value); - break; - case NPY_INT16: - *(int16_t *)destination = (int16_t)PyLong_AsLong(value); - break; - case NPY_INT32: - *(int32_t *)destination = (int32_t)PyLong_AsLong(value); - break; - case NPY_INT64: - *(int64_t *)destination = (int64_t)PyLong_AsLongLong(value); - break; - case NPY_FLOAT32: - *(float *)destination = (float)PyFloat_AsDouble(value); - break; - case NPY_FLOAT64: - *(double *)destination = PyFloat_AsDouble(value); - break; - case NPY_COMPLEX64: { - float real = (float)PyComplex_RealAsDouble(value); - float imaginary = (float)PyComplex_ImagAsDouble(value); - *(float complex *)destination = real + imaginary * I; - break; - } - case NPY_COMPLEX128: { - double real = PyComplex_RealAsDouble(value); - double imaginary = PyComplex_ImagAsDouble(value); - *(double complex *)destination = real + imaginary * I; - break; - } - default: - PyErr_Format(PyExc_TypeError, "unsupported x2py scalar type %d", numpy_type); - return -1; - } - return PyErr_Occurred() == NULL ? 0 : -1; -} - -PyObject *x2py_scalar_to_python(int numpy_type, const void *value) -{ - if (value == NULL) { - PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar value"); - return NULL; - } - - switch (numpy_type) { - case NPY_BOOL: - return PyBool_FromLong(*(const bool *)value); - case NPY_INT8: - return PyLong_FromLong(*(const int8_t *)value); - case NPY_INT16: - return PyLong_FromLong(*(const int16_t *)value); - case NPY_INT32: - return PyLong_FromLong(*(const int32_t *)value); - case NPY_INT64: - return PyLong_FromLongLong(*(const int64_t *)value); - case NPY_FLOAT32: - return PyFloat_FromDouble(*(const float *)value); - case NPY_FLOAT64: - return PyFloat_FromDouble(*(const double *)value); - case NPY_COMPLEX64: { - float complex number = *(const float complex *)value; - return PyComplex_FromDoubles(crealf(number), cimagf(number)); - } - case NPY_COMPLEX128: { - double complex number = *(const double complex *)value; - return PyComplex_FromDoubles(creal(number), cimag(number)); - } - default: - PyErr_Format(PyExc_TypeError, "unsupported x2py scalar type %d", numpy_type); - return NULL; - } -} - -PyObject *x2py_scalar_to_numpy(int numpy_type, const void *value) -{ - if (value == NULL) { - PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar value"); - return NULL; - } - - PyArray_Descr *descriptor = PyArray_DescrFromType(numpy_type); - if (descriptor == NULL) { - return NULL; - } - return PyArray_Scalar((void *)value, descriptor, NULL); -} - -void x2py_release_owned_memory(PyObject *capsule) -{ - void *memory = PyCapsule_GetPointer(capsule, NULL); - if (memory == NULL) { - PyErr_Clear(); - return; - } - free(memory); -} diff --git a/x2py/binding_support/x2py_binding.h b/x2py/binding_support/x2py_binding.h index eecec56d1..4a812d486 100644 --- a/x2py/binding_support/x2py_binding.h +++ b/x2py/binding_support/x2py_binding.h @@ -9,6 +9,7 @@ #include #include #include +#include #include "numpy_version.h" @@ -21,18 +22,152 @@ #include /* Return whether value is exactly the NumPy scalar required by numpy_type. */ -bool x2py_scalar_matches(PyObject *value, int numpy_type); +static inline bool x2py_scalar_matches(PyObject *value, int numpy_type) +{ + switch (numpy_type) { + case NPY_BOOL: + return PyBool_Check(value) || PyArray_IsScalar(value, Bool); + case NPY_INT8: + return PyArray_IsScalar(value, Int8); + case NPY_INT16: + return PyArray_IsScalar(value, Int16); + case NPY_INT32: + return PyArray_IsScalar(value, Int); + case NPY_INT64: + return PyArray_IsScalar(value, Int64); + case NPY_FLOAT32: + return PyArray_IsScalar(value, Float); + case NPY_FLOAT64: + return PyArray_IsScalar(value, Double); + case NPY_COMPLEX64: + return PyArray_IsScalar(value, CFloat); + case NPY_COMPLEX128: + return PyArray_IsScalar(value, CDouble); + default: + return false; + } +} /* Copy one Python scalar into caller-owned native storage after boundary checks. */ -int x2py_scalar_unpack(PyObject *value, int numpy_type, void *destination); +static inline int x2py_scalar_unpack(PyObject *value, int numpy_type, void *destination) +{ + if (destination == NULL) { + PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar destination"); + return -1; + } + + if (numpy_type == NPY_BOOL) { + int truth = PyObject_IsTrue(value); + if (truth < 0) { + return -1; + } + *(bool *)destination = truth != 0; + return 0; + } + if (x2py_scalar_matches(value, numpy_type)) { + PyArray_ScalarAsCtype(value, destination); + return PyErr_Occurred() == NULL ? 0 : -1; + } + + switch (numpy_type) { + case NPY_INT8: + *(int8_t *)destination = (int8_t)PyLong_AsLong(value); + break; + case NPY_INT16: + *(int16_t *)destination = (int16_t)PyLong_AsLong(value); + break; + case NPY_INT32: + *(int32_t *)destination = (int32_t)PyLong_AsLong(value); + break; + case NPY_INT64: + *(int64_t *)destination = (int64_t)PyLong_AsLongLong(value); + break; + case NPY_FLOAT32: + *(float *)destination = (float)PyFloat_AsDouble(value); + break; + case NPY_FLOAT64: + *(double *)destination = PyFloat_AsDouble(value); + break; + case NPY_COMPLEX64: { + float real = (float)PyComplex_RealAsDouble(value); + float imaginary = (float)PyComplex_ImagAsDouble(value); + *(float complex *)destination = real + imaginary * I; + break; + } + case NPY_COMPLEX128: { + double real = PyComplex_RealAsDouble(value); + double imaginary = PyComplex_ImagAsDouble(value); + *(double complex *)destination = real + imaginary * I; + break; + } + default: + PyErr_Format(PyExc_TypeError, "unsupported x2py scalar type %d", numpy_type); + return -1; + } + return PyErr_Occurred() == NULL ? 0 : -1; +} /* Create a normal Python scalar from native storage. */ -PyObject *x2py_scalar_to_python(int numpy_type, const void *value); +static inline PyObject *x2py_scalar_to_python(int numpy_type, const void *value) +{ + if (value == NULL) { + PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar value"); + return NULL; + } + + switch (numpy_type) { + case NPY_BOOL: + return PyBool_FromLong(*(const bool *)value); + case NPY_INT8: + return PyLong_FromLong(*(const int8_t *)value); + case NPY_INT16: + return PyLong_FromLong(*(const int16_t *)value); + case NPY_INT32: + return PyLong_FromLong(*(const int32_t *)value); + case NPY_INT64: + return PyLong_FromLongLong(*(const int64_t *)value); + case NPY_FLOAT32: + return PyFloat_FromDouble(*(const float *)value); + case NPY_FLOAT64: + return PyFloat_FromDouble(*(const double *)value); + case NPY_COMPLEX64: { + float complex number = *(const float complex *)value; + return PyComplex_FromDoubles(crealf(number), cimagf(number)); + } + case NPY_COMPLEX128: { + double complex number = *(const double complex *)value; + return PyComplex_FromDoubles(creal(number), cimag(number)); + } + default: + PyErr_Format(PyExc_TypeError, "unsupported x2py scalar type %d", numpy_type); + return NULL; + } +} /* Create a NumPy scalar from native storage. */ -PyObject *x2py_scalar_to_numpy(int numpy_type, const void *value); +static inline PyObject *x2py_scalar_to_numpy(int numpy_type, const void *value) +{ + if (value == NULL) { + PyErr_SetString(PyExc_RuntimeError, "x2py generated a null scalar value"); + return NULL; + } + + PyArray_Descr *descriptor = PyArray_DescrFromType(numpy_type); + if (descriptor == NULL) { + return NULL; + } + return PyArray_Scalar((void *)value, descriptor, NULL); +} /* Release a bridge-owned allocation transferred through a NumPy base capsule. */ -void x2py_release_owned_memory(PyObject *capsule); +static inline void x2py_release_owned_memory(PyObject *capsule) +{ + void *memory = PyCapsule_GetPointer(capsule, NULL); + if (memory == NULL) { + PyErr_Clear(); + return; + } + free(memory); +} #endif diff --git a/x2py/compiling/README.md b/x2py/compiling/README.md index 8cc3bf5f6..56579c958 100644 --- a/x2py/compiling/README.md +++ b/x2py/compiling/README.md @@ -11,7 +11,7 @@ linking. | `objects.py` | Explicit source-to-object compilation inputs. | | `compilers.py` | Compiler command execution and tool lookup helpers. | | `compiler_profiles.py` | Built-in vendor compiler profiles and Python-link settings. | -| `native_support.py` | Writing native binding support and declaring its object inputs. | +| `native_support.py` | Writing the header-only native binding support. | Generated-wrapper object assembly and shared-library orchestration live in `x2py/pipeline/build.py`, where the canonical rendered wrapper artifacts are @@ -25,17 +25,17 @@ native source files -> native object files generated Fortran bridge -> bridge object files -generated C/CPython binding and its native support - -> runtime and binding object files +generated C/CPython binding and its header-only native support + -> binding object files all explicit object files and link inputs -> linked Python extension ``` The bridge and binding sources are rendered together from one completed wrapper plan before compilation starts. Their object stages remain separate: the bridge -is compiled after native objects so it can consume native module files; runtime -support is compiled before the binding that includes it; and linking runs only -after every required object exists. Each compiler invocation receives its +is compiled after native objects so it can consume native module files; the +header-only native support is compiled with the binding that includes it; and +linking runs only after every required object exists. Each compiler invocation receives its source, target, flags, includes, and ordered link inputs explicitly. Compilation must not decide semantic ownership, Python API shape, or wrapper diff --git a/x2py/compiling/native_support.py b/x2py/compiling/native_support.py index c83738b15..ca4661b31 100644 --- a/x2py/compiling/native_support.py +++ b/x2py/compiling/native_support.py @@ -8,9 +8,6 @@ import x2py.binding_support as binding_support_folder -from .objects import ObjectFile - - _NATIVE_SUPPORT_IMPORT = "binding_support" _NATIVE_SUPPORT_SOURCE = Path(binding_support_folder.__file__).parent @@ -26,10 +23,10 @@ def _numpy_version_header() -> str: return header -def install_native_support(imports, *, x2py_dirpath, verbose: bool | int = False) -> tuple[ObjectFile, ...]: - """Write native binding support and return its explicit compilation inputs.""" +def install_native_support(imports, *, x2py_dirpath, verbose: bool | int = False) -> None: + """Write header-only native binding support when a generated binding imports it.""" if not any(name == _NATIVE_SUPPORT_IMPORT or name.startswith(f"{_NATIVE_SUPPORT_IMPORT}/") for name in imports): - return () + return destination = Path(x2py_dirpath) / _NATIVE_SUPPORT_IMPORT if verbose: @@ -42,12 +39,3 @@ def install_native_support(imports, *, x2py_dirpath, verbose: bool | int = False _numpy_version_header(), encoding="utf-8", ) - return ( - ObjectFile( - source=destination / "x2py_binding.c", - object_path=destination / "x2py_binding.o", - language="c", - include_dirs=(destination,), - tools=frozenset({"python"}), - ), - ) diff --git a/x2py/pipeline/build.py b/x2py/pipeline/build.py index 5ae83888a..1ffd24081 100644 --- a/x2py/pipeline/build.py +++ b/x2py/pipeline/build.py @@ -496,7 +496,7 @@ def _build_rendered_wrapper_extension( ), ) native_support_imports = _rendered_wrapper_native_support_imports(rendered.artifacts.native_support_keys) - native_support_objects = install_native_support( + install_native_support( native_support_imports, x2py_dirpath=str(output_path), verbose=verbose, @@ -507,12 +507,6 @@ def _build_rendered_wrapper_extension( label="Compile bridge source", verbose=verbose, ) - _compile_object_stage( - compiler, - native_support_objects, - label="Compile native support source", - verbose=verbose, - ) _compile_object_stage( compiler, binding_objects, @@ -525,7 +519,7 @@ def _build_rendered_wrapper_extension( module_name=rendered.artifacts.module_name, output_dir=shared_output_path, language=_rendered_wrapper_link_language(bridge_objects, binding_objects), - objects=(*tuple(native_dependencies), *bridge_objects, *native_support_objects, *binding_objects), + objects=(*tuple(native_dependencies), *bridge_objects, *binding_objects), link_args=tuple(native_link_args), library_dirs=resolved_native_build_plan.library_dirs, flags=_compiler_flags(wrapper_c_flags), @@ -1642,7 +1636,7 @@ def _write_build_makefile( lines = [ "# Generated by x2py. Edit variables or override them on the make command line.", "# User Fortran sources are conservatively chained in supplied order.", - "# Independent generated C/runtime objects may be built in parallel with make -j.", + "# Generated bridge and C binding objects may be built in parallel with make -j.", f"FC := {_make_shell_literal(shlex.quote(_compiler_executable(commands, language='fortran', shared=False)))}", f"CC := {_make_shell_literal(shlex.quote(_compiler_executable(commands, language='c', shared=False)))}", f"X2PY_LD := {_make_shell_literal(shlex.quote(_compiler_executable(commands, language=None, shared=True)))}", From 0d76eb0315c93ee4bb53f4680d0213f39734472b Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 22:42:22 +0100 Subject: [PATCH 29/30] add more tests for module variables of scalar derived type kind --- .../roadmap/wrapper-plan-migration-checklist.md | 2 +- docs/user/guide/wrapping-derived-types.md | 7 +++++++ .../fscalar_derived_actual_dummy_matrix_f90.f90 | 8 ++++++-- tests/wrapper/fortran/derived_types/README.md | 3 +++ .../phase8_left_types.pyi | 3 +++ .../phase8_right_types.pyi | 3 +++ .../test_scalar_derived_actual_dummy_matrix.py | 10 ++++++++++ 7 files changed, 33 insertions(+), 3 deletions(-) diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index 26e02b1da..e0a27f472 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -721,7 +721,7 @@ already covered by the new generator. | `tests/wrapper/fortran/derived_types/test_inheritance.py::*` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; classes/methods/properties/overloads | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_phase8_derived_plan.py::*` | reduced passing legacy/source artifacts compared with direct typed-plan generation; plain non-target module objects intentionally use the safer member-proxy correction described in Phase 8 | scalar derived arguments/results; optional and by-value inputs; projected identity; owned/borrowed lifecycle; plain/`Aliased` module objects; scalar/string/array/nested/native-handle fields; production routing | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_phase9_bound_constructors.py::*` | reduced direct-plan bound-constructor runtime and artifact proof | explicit bound construction; shared method plan; allocation and owner commit | `wrapper-plan` | -| `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py::*` | complete source/generated-contract and direct-plan proof over the canonical scalar-derived matrix fixture; replaces the former isolated descriptor rejection unit; final Phase 8H cross-suite verification remains a separate closure gate | all five actual declarations from module and wrapper origins; all six dummy forms; exact action/error selection; holder, scoped-address, allocation and pointer transactions; mixed multi-argument acquisition and reverse cleanup | `wrapper-plan` | +| `tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py::*` | complete source/generated-contract and direct-plan proof over the canonical scalar-derived matrix fixture; replaces the former isolated descriptor rejection unit; final Phase 8H cross-suite verification remains a separate closure gate | all five actual declarations from module and wrapper origins; all six dummy forms; exact action/error selection; holder, scoped-address, allocation and pointer transactions; mixed multi-argument acquisition and reverse cleanup; distinct module-origin callbacks for qualified types from separate Fortran modules | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_and_derived_pointer_handles_track_native_association[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_module_native_array_handles_use_canonical_plan` | canonical reduced module-only contract | borrowed pointer/allocatable handles; descriptor calls; strided extraction; ordinary array actuals; operation permissions | `wrapper-plan` | | `tests/wrapper/fortran/derived_types/test_pointers.py::test_pointer_array_handles_block_on_unsupported_result_owner_policy[*]` | source/generated-.pyi parity or parametrized route | derived types/object lifetimes; native handles/descriptors | `wrapper-plan` | diff --git a/docs/user/guide/wrapping-derived-types.md b/docs/user/guide/wrapping-derived-types.md index acc337669..feaf4088c 100644 --- a/docs/user/guide/wrapping-derived-types.md +++ b/docs/user/guide/wrapping-derived-types.md @@ -193,6 +193,13 @@ in reverse order. It does not generate `2**N` call variants. Repeated read-only use of the same object shares one acquisition; ambiguous writable aliasing is rejected before any module state moves. +This also applies when arguments are module variables from different Fortran +modules and have different qualified derived types. Each module variable owns a +separate bridge operation table and scoped callback; the binding validates the +table's qualified native type before the callback is invoked. There is no +shared type-specific callback slot, so one module variable cannot overwrite +another argument's transport. + If a later acquisition or the native call reports a normal ABI error, cleanup continues for every acquired origin and Python raises only after the Fortran frames have returned. Concurrent or recursive use of the same active module diff --git a/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 b/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 index 79ab1663d..60c5f544d 100644 --- a/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 +++ b/tests/data/fortran/wrapper/fscalar_derived_actual_dummy_matrix_f90.f90 @@ -3,9 +3,11 @@ module phase8_left_types implicit none type :: item - integer(c_int32_t) :: value = 0_c_int32_t + integer(c_int32_t) :: value = 3_c_int32_t end type item + type(item) :: state + contains function make_item(initial) result(value) @@ -21,9 +23,11 @@ module phase8_right_types implicit none type :: item - integer(c_int32_t) :: value = 0_c_int32_t + integer(c_int32_t) :: value = 7_c_int32_t end type item + type(item) :: state + contains function make_item(initial) result(value) diff --git a/tests/wrapper/fortran/derived_types/README.md b/tests/wrapper/fortran/derived_types/README.md index 86496ae4f..ef0e1851b 100644 --- a/tests/wrapper/fortran/derived_types/README.md +++ b/tests/wrapper/fortran/derived_types/README.md @@ -34,3 +34,6 @@ states, holder and module transactions, multi-argument acquisition and rollback, duplicate-origin validation, qualified types with the same short name, ordinary/`sequence` typed-value calls, injected cleanup failures, secondary-compiler ABI smoke coverage, and exact unsupported-cell diagnostics. +It also proves that one call can receive module variables from separate +Fortran modules whose derived types share a short name but have distinct native +identities. diff --git a/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_left_types.pyi b/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_left_types.pyi index 1ac33cb3f..e19770744 100644 --- a/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_left_types.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_left_types.pyi @@ -5,4 +5,7 @@ class item: value: Int32 +state: item + + def make_item(initial: Int32) -> item: ... diff --git a/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_right_types.pyi b/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_right_types.pyi index 1ac33cb3f..e19770744 100644 --- a/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_right_types.pyi +++ b/tests/wrapper/fortran/derived_types/contracts/fscalar_derived_actual_dummy_matrix_phase8/phase8_right_types.pyi @@ -5,4 +5,7 @@ class item: value: Int32 +state: item + + def make_item(initial: Int32) -> item: ... diff --git a/tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py b/tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py index 8b61d8177..f3f7a0c5c 100644 --- a/tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py +++ b/tests/wrapper/fortran/derived_types/test_scalar_derived_actual_dummy_matrix.py @@ -357,6 +357,16 @@ def test_qualified_same_short_name_types_keep_exact_native_identity(scalar_matri module.read_qualified(*values) +def test_module_origins_from_separate_modules_keep_type_specific_callbacks(scalar_matrix): + module = scalar_matrix.module + left = scalar_matrix.left_module.state + right = scalar_matrix.right_module.state + + assert module.read_qualified(left, right) == 307 + with pytest.raises(TypeError, match=r"left_item.*left"): + module.read_qualified(right, left) + + def test_duplicate_origins_share_reads_and_reject_writes_before_native_call(scalar_matrix): module = scalar_matrix.module module.reset_state() From 10a210453eaa0705e6c001bb76beff3990aa67b4 Mon Sep 17 00:00:00 2001 From: said Date: Fri, 17 Jul 2026 23:14:59 +0100 Subject: [PATCH 30/30] fix errors --- docs/maintainer/roadmap/wrapper-plan-migration-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md index e0a27f472..0d8288e47 100644 --- a/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md +++ b/docs/maintainer/roadmap/wrapper-plan-migration-checklist.md @@ -593,7 +593,7 @@ summary, the exhaustive matrix, and the test tree disagree. | Status | Collected nodes | | --- | ---: | -| `wrapper-plan` | 348 | +| `wrapper-plan` | 349 | | `dual-route` | 0 | | `legacy` | 0 | | `not-applicable` | 75 |