diff --git a/docs/README.md b/docs/README.md index 5e55fe076..bd0e4ebd3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,6 +10,8 @@ examples. Contribution and pull-request requirements remain in long-term architecture and runtime model. - [Wrapper `.pyi` format](semantics/pyi_format.md): editable semantic interface syntax and conversion behavior. +- [C to semantic IR mapping](semantics/c2ir_mapping.md): implemented C + semantic conversion subset and blocker policy. - [Self-contained C semantic `.pyi` specification](semantics/c_pyi_self_contained_specification.md): staged C wrapper interface design. @@ -29,8 +31,8 @@ examples. Contribution and pull-request requirements remain in ### C -- [C parser reference](c_parser/c_parser_reference.md): implemented parse-only - frontend behavior and testing workflow. +- [C parser reference](c_parser/c_parser_reference.md): implemented parser + behavior, semantic handoff, and testing workflow. - [C parser architecture plan](c_parser/c_parser_architecture.md): design and integration decisions. - [C parser CLI workflow plan](c_parser/c_parser_cli_workflow.md): command diff --git a/docs/c_parser/c_parser_architecture.md b/docs/c_parser/c_parser_architecture.md index 2d476d8e4..89dde341a 100644 --- a/docs/c_parser/c_parser_architecture.md +++ b/docs/c_parser/c_parser_architecture.md @@ -12,7 +12,7 @@ The x2py CLI also has a shared C/Fortran preprocessing option surface and can run an exact compiler/preprocessor executable for C compiler mode. Compiler and preprocessed C inputs preserve `#line`/GCC linemarker source locations for parsed declarations and diagnostics. A separate compiler-derived standard-type -probe supplies target ABI facts needed by later C semantic conversion. +probe supplies target ABI facts consumed by C semantic conversion. This document records the target architecture for the C parser frontend in x2py. The initial skeleton has grown into a partial parser, and the remaining @@ -91,12 +91,18 @@ Implemented now: preserving compiler/runner/source provenance for semantic conversion. It carries target-relevant include, macro, undefine, and compiler-argument flags and records the requested project standard as provenance. -- `--language c --semantics`, `--language c --pyi`, and C wrap-readiness are - rejected until semantic conversion exists. +- `semantics.c2ir` converts the supported C parser subset into semantic IR, + including scalar functions, pointer storage contracts, declared arrays, + structs/opaque structs, enum and numeric macro constants, local typedef + chains, target standard-type probe facts, and C-specific readiness blockers. +- `--language c --semantics`, `--language c --wrap-readiness`, and starter + exact-contract `--language c --pyi` output are enabled for the supported C + semantic subset. - Focused partial CLI/API, declaration/function, diagnostic color, project include/index, raw lexer/directive, project golden, error golden, preprocessed - linemarker remapping, and JSON schema tests are active. Remaining - parser-suite skips are limited to the pinned corpus roadmap. + linemarker remapping, JSON schema, and cJSON partial-parse regression tests + are active. A separately pinned/provenanced corpus remains deferred work, + not a skipped test. - `tests/data/c/` contains general fixtures modeled after the Fortran general fixture themes, additional C-specific API shapes, fatal diagnostic inputs, and real-world cJSON/jsmn/tinyexpr/linmath/NanoSVG/stb inputs whose partial @@ -110,7 +116,7 @@ Deferred: - compiler attributes and alignment specifiers - broader compiler-family validation for preprocessing; parsed declarations already retain preprocessed origin and mapped source identity -- C semantic readiness, semantic IR conversion, and `.pyi` output +- broader C callback/ownership policy beyond exact starter `.pyi` stubs Documentation rule: any future C parser implementation change must update all affected docs under `docs/c_parser/` in the same change. This applies to model, @@ -328,8 +334,8 @@ class CParser: These entrypoints are exposed from both `c_parser` and `x2py.__init__`, using the same top-level file/project invocation pattern already provided for -Fortran. C semantic conversion remains unavailable despite the parse API -export. +Fortran. C semantic conversion is exposed separately through +`semantics.c2ir` and top-level `x2py` compatibility helpers. ## Core Model Families @@ -723,9 +729,9 @@ Python APIs. ## `.pyi` Integration -Generated `.pyi` stubs for C should come after parser models and semantic IR -conversion are stable. Readiness, if added for C, should follow the semantic -layer pattern already used by the project. +Generated `.pyi` stubs for C are emitted from semantic IR for the supported +exact-contract subset. Readiness follows the semantic-layer pattern already +used by the project. Likely stub patterns: diff --git a/docs/c_parser/c_parser_cli_workflow.md b/docs/c_parser/c_parser_cli_workflow.md index 293c8d211..f8778bef9 100644 --- a/docs/c_parser/c_parser_cli_workflow.md +++ b/docs/c_parser/c_parser_cli_workflow.md @@ -13,8 +13,9 @@ represent parenthesized pointer/array precedence through concrete parameters expose both declared and C-adjusted effective type facts. The CLI also exposes shared C/Fortran preprocessing flags and can run exact compiler/preprocessor executables for compiler mode. Target-specific standard -header type facts for later C semantics are available through the separate -`python -m x2py.c_type_probe` command. +header type facts for C semantics are available through the separate +`python -m x2py.c_type_probe` command. C semantic IR, readiness, and starter +exact-contract `.pyi` output are available through the shared `x2py` CLI. This document records the implemented C parse command shape, output schema, and diagnostic contract, plus deferred CLI behavior. @@ -28,6 +29,9 @@ python -m x2py path/to/api.h --language c --parse python -m x2py path/to/api.h --language c --parse --json python -m x2py path/to/api.h --language c --parse --out report.json python -m x2py path/to/api.h --language c --parse --preprocess compiler --compiler clang-18 -I include -D API_EXPORT= --std c11 +python -m x2py path/to/api.h --language c --semantics +python -m x2py path/to/api.h --language c --wrap-readiness +python -m x2py path/to/api.h --language c --pyi ``` The C parser accepts explicit `.c`, `.h`, and direct `.i` files, plus @@ -86,17 +90,15 @@ as `int i; int i;` are also merged. Duplicate definitions and incompatible top-level redeclarations are reported as diagnostics. Local declarations inside function bodies are not parsed. -Unsupported C stages: +Unsupported C display controls: ```bash -python -m x2py path/to/api.h --language c --semantics -python -m x2py path/to/api.h --language c --pyi -python -m x2py path/to/api.h --language c --wrap-readiness +python -m x2py path/to/api.h --language c --parse --show-vars +python -m x2py path/to/api.h --language c --parse --print-limit 20 ``` -These commands return clear argparse errors until C semantic IR conversion and -`.pyi` generation are implemented. Fortran-only parse display flags such as -`--show-vars` and `--print-limit` are rejected in C mode. +Fortran-only parse display flags such as `--show-vars` and `--print-limit` +return clear argparse errors in C mode until C-specific display controls exist. ## Current CLI Baseline @@ -402,9 +404,9 @@ JSON output for a file without raw directives: } ``` -The parser should not claim C files are wrappable. If C readiness is added -later, it should follow the semantics-owned readiness boundary used elsewhere -in x2py, not become parser JSON. +The parser should not claim C files are wrappable. C readiness follows the +semantics-owned readiness boundary used elsewhere in x2py and does not become +parser JSON. For raw directives, the same JSON shape is used, but `includes`, `macros`, `raw_directives`, `macro_dependencies`, and `diagnostics` may contain populated @@ -597,10 +599,9 @@ The active CLI/parser tests cover the current partial subset: by focused C tests. - `--show-vars` and `--print-limit` are rejected in C mode until C-specific display controls exist. -- `--semantics` with `--language c` is rejected until C semantic conversion is - implemented. -- `--pyi` with `--language c` is rejected until C `.pyi` emission is - implemented. +- `--semantics`, `--wrap-readiness`, and `--pyi` with `--language c` use + `semantics.c2ir`, the semantic readiness checker, and the shared `.pyi` + emitter for the supported exact-contract subset. ## Integration Order @@ -648,8 +649,10 @@ Completed order: including recursively mapped preprocessed provenance. 21. Added `_Atomic(type)` type-specifier parsing on the shared declarator path and executable parser-developer walkthrough tests. -22. Added compiler-derived standard-header ABI probing for future C semantic - mapping without hard-coded host type aliases. +22. Added compiler-derived standard-header ABI probing for C semantic mapping + without hard-coded host type aliases. +23. Added C semantic IR, readiness, and starter exact-contract `.pyi` output + through `x2py --language c`. Next implementation work should continue with fixture-driven compiler extension policy and broader project conflict policy diff --git a/docs/c_parser/c_parser_reference.md b/docs/c_parser/c_parser_reference.md index eb844b606..b84165eca 100644 --- a/docs/c_parser/c_parser_reference.md +++ b/docs/c_parser/c_parser_reference.md @@ -1,11 +1,11 @@ # C Parser Reference -Status: current reference for the parse-only C frontend. The `c_parser` +Status: current reference for the partial C frontend. The `c_parser` package, typed parser models, explicit C CLI parse path, raw directive metadata, compiler-assisted preprocessing, source-location remapping, project -indexes, parser goldens, and C standard-type probe are implemented. C semantic -readiness, semantic IR conversion, and `.pyi` generation remain future work and -are intentionally rejected by the CLI. +indexes, parser goldens, C standard-type probe, first semantic IR conversion +subset, semantic readiness path, and starter exact-contract C `.pyi` +generation are implemented. ## Purpose @@ -52,8 +52,11 @@ Implemented: the `c_parser` package entrypoints - `CParseError` with compiler-style diagnostic formatting - explicit `x2py --language c --parse` output +- explicit `x2py --language c --semantics` and + `x2py --language c --wrap-readiness` output +- starter exact-contract `x2py --language c --pyi` output for the supported C + semantic subset - C JSON partial output and `--out` behavior -- rejection of C `--semantics`, `--pyi`, and `--wrap-readiness` - raw lexer records with comment stripping, line-continuation folding, and lightweight token source locations - top-level source splitting that tracks braces, parentheses, brackets, and @@ -123,6 +126,10 @@ Implemented: `tests/data/c/errors/parser/`, and partial-parser regression inputs under `tests/data/c/json/`, `tests/data/c/tinyexpr/`, `tests/data/c/linmath/`, `tests/data/c/nanosvg/`, and top-level C inputs from `tests/data/c/stb/` +- `semantics.c2ir` conversion for the first identity subset: scalar + functions, const/mutable pointer storage contracts, declared arrays, + structs/opaque structs, enums, numeric macro constants, local typedef + chains, standard-type probe facts, and explicit semantic readiness blockers Still deferred: @@ -130,7 +137,8 @@ Still deferred: - broad compiler-extension declarators - broader typedef/tag conflict policy beyond the implemented basic project resolution -- semantic readiness, semantic IR conversion, and `.pyi` generation +- richer C ownership/callback projection policy beyond exact starter `.pyi` + stubs ## Supported C Subset @@ -292,9 +300,8 @@ target-relevant flags from the matching entry to the probe explicitly. For cross targets, provide a runner, for example `--runner=qemu-aarch64 --runner=-L --runner=/opt/aarch64-sysroot`. -The eventual C semantic converter should accept this report as target context. -The parser model remains source-faithful and does not embed host ABI -assumptions. +The C semantic converter accepts this report as target context. The parser +model remains source-faithful and does not embed host ABI assumptions. ## Public API @@ -603,7 +610,7 @@ Test families should mirror the Fortran parser: - typedef tests - macro/constant tests - include/project tests -- semantic readiness tests once C semantic conversion exists +- C semantic readiness tests - CLI tests - semantic conversion tests - `.pyi` generation/parser tests @@ -611,8 +618,8 @@ Test families should mirror the Fortran parser: - error fixture/golden tests - corpus parse-only tests -The C test area contains active partial-parser/raw-metadata tests plus narrowly -scoped roadmap skips under `tests/parser/c/`. The active tests cover +The C test area contains active partial-parser/raw-metadata tests, including +parse-only cJSON regression coverage under `tests/parser/c/`. The active tests cover public entrypoints, empty model serialization, CLI discovery, JSON/output-file behavior, unsupported C stages, comment stripping, line-continuation folding, top-level splitting, include collection, simple macro collection, macro-shaped @@ -625,8 +632,8 @@ prototypes/definitions, function-definition start/end locations, JSON golden serialization, fatal diagnostic goldens, and project-level callback typedef resolution. The `json` regression inputs intentionally retain recoverable diagnostics from unsupported constructs; they -do not claim complete library parsing. Remaining parser-suite skips cover the -pinned/provenanced corpus target. Golden comparison tests rewrite their baselines when +do not claim complete library parsing. A separately pinned/provenanced corpus +target remains deferred without disabling parser tests. Golden comparison tests rewrite their baselines when `C_PARSER_UPDATE_GOLDENS=1` is set. Future implementation branches should activate only the tests for the capability they implement. diff --git a/docs/semantics/c2ir_mapping.md b/docs/semantics/c2ir_mapping.md new file mode 100644 index 000000000..41ae34b80 --- /dev/null +++ b/docs/semantics/c2ir_mapping.md @@ -0,0 +1,64 @@ +# C To Semantic IR Mapping + +Status: first C semantic conversion subset implemented in `semantics/c2ir.py`. +The converter consumes `c_parser` models and emits the same language-neutral +semantic IR used by Fortran and edited `.pyi` files. + +## Supported Identity Subset + +- C translation unit -> one `SemanticModule` named from the source file stem. +- C function -> `SemanticFunction`, preserving native name and parameter order. +- C parameter -> `SemanticArgument`. +- `void` return -> `None`. +- `_Bool` -> `Bool`. +- `char` -> `Int8` with `c_char_policy` metadata; `signed char` -> `Int8`; + `unsigned char` -> `UInt8`. +- `short`, `int`, `long`, and `long long` map to fixed signed integer names + using the current Linux-oriented defaults: `Int16`, `Int32`, `Int64`, + `Int64`. +- Unsigned integer spellings map to `UInt16`, `UInt32`, `UInt64`, and + `UInt64`; fixed-width typedef spellings such as `uint32_t` map to the + matching `UInt*` fallback. +- `float` -> `Float32`; `double` -> `Float64`. +- `float _Complex` -> `Complex64`; `double _Complex` -> `Complex128`. +- Local typedef chains are resolved when their parser model definitions are + available. +- `size_t` maps to `SizeT` without a target probe; supplied + `x2py.c_type_probe` facts override standard typedefs with width-specific + `Int*`, `UInt*`, or `Float*` semantic names. +- Opaque standard-type probe facts such as `FILE` create named opaque semantic + classes when referenced by converted declarations. +- Object-like numeric macros and enum constants become `Final`-style semantic + variables through the `Constant` constraint. +- Struct definitions become `SemanticClass` entries. Incomplete structs become + opaque classes and may be used through direct `Ptr(...)` identity contracts. +- Declared C arrays, including adjusted array parameters, become semantic array + storage contracts with C order for rank greater than one. +- Pointers become explicit `SemanticStorageContract` pointer/reference + metadata. `const` on the pointee makes the storage read-only, and `restrict` + is preserved as aliasing metadata. + +## Conservative Blockers + +The converter does not silently invent wrapper policy. It attaches +`readiness_blockers` metadata that the semantic readiness checker reports: + +- unresolved typedef or unknown type references; +- macro-dependent declarations in raw C input; +- variadic functions; +- function pointer/callback signatures without edited `.pyi` `Callable` + policy; +- mutable numeric or `void *` pointer parameters without ownership, + scalar-reference, or array policy; +- arrays with unknown extents; +- incomplete structs used by value; +- unions used in semantic signatures; +- `long double`, `volatile`, `_Atomic`, bitfields, and unsupported declarator + compositions. + +The current C semantic path supports `--language c --semantics`, +`--language c --wrap-readiness`, and starter exact-contract +`--language c --pyi` output for this supported subset. Generated stubs remain +conservative: ambiguous ownership, callback, ABI-extension, and Pythonic +projection policy stays out of the generated `.pyi` until supplied by the +semantic model or an edited interface. diff --git a/docs/semantics/c_pyi_self_contained_specification.md b/docs/semantics/c_pyi_self_contained_specification.md index 3d00795f0..637de0487 100644 --- a/docs/semantics/c_pyi_self_contained_specification.md +++ b/docs/semantics/c_pyi_self_contained_specification.md @@ -196,8 +196,8 @@ constraint is written. not part of the canonical public array annotation unless they produce an actual storage constraint. In particular, Fortran dummy bounds are established by native association rather than supplied as Python array - metadata. This does not add C semantic conversion support; C conversion - remains deferred. + metadata. The implemented C conversion subset is described in + [C to semantic IR mapping](c2ir_mapping.md). Stride-aware dimensions use a slice step marker: diff --git a/docs/semantics/pyi_format.md b/docs/semantics/pyi_format.md index f8e830050..1330d7e0d 100644 --- a/docs/semantics/pyi_format.md +++ b/docs/semantics/pyi_format.md @@ -5,9 +5,8 @@ language-neutral: Fortran and future C inputs use the same type, storage, pointer, array, layout and metadata notation. Source language differences are represented by contracts and metadata, not by separate syntax families. -This document describes the behavior implemented for the current Fortran path -and the shared notation it establishes for later C semantic conversion. C -semantic conversion and C `.pyi` generation remain deferred. +This document describes the behavior implemented for the current Fortran and C +semantic conversion paths. ## Canonical Type And Storage Contract @@ -39,6 +38,10 @@ entries express range or stride contracts (`Float64[1:n]`, `Float64[::Strided]`, `Float64[:, 0:n:m]`). `Strided` means the runtime stride is part of the accepted storage contract. +Generic semantic constraints are not represented as type subscriptions. +Constants use `Final[T]`; other constraints and non-dimensional array metadata +use `Annotated[T[...], Constraint, ...]`. + `Annotated[...]` carries non-dimensional metadata: - `ORDER_F` for a Fortran-oriented multidimensional contract. @@ -411,20 +414,21 @@ visible Python values The projection mechanism is language-neutral. It can later adapt exact Fortran or C contracts through the same notation and runtime concepts, but this milestone does not implement automatic Pythonic generation, current -exact-reference adaptation, coercion/contract execution or C semantic -conversion/output. +exact-reference adaptation, coercion/contract execution or C wrapper lowering. +The C frontend can generate starter exact-contract `.pyi` output for the +implemented semantic subset. ## Deferred C Work -The shared model is capable of representing future C functions, variables, +The shared model represents the current C semantic conversion subset for +functions, variables, fields, constants, scalar references, pointers, arrays with known contracts, -origin metadata, mutability and ownership facts. This task does not implement: +origin metadata, mutability and ownership facts. The C frontend can generate +starter exact-contract stubs from that model. Remaining C work includes: -- `semantics/c2ir.py`; -- C semantic conversion; -- C `.pyi` generation; - C wrapper lowering; -- C ownership, callback or pointer policy inference. +- C ownership, callback or pointer policy inference beyond facts already + present in exact contracts. Future C conversion should use the same notation: by-value scalars as bare types, unrefined pointers as `Ptr(T)` or `Ptr(Const(T))`, and array notation diff --git a/docs/x2py_checklist.md b/docs/x2py_checklist.md index f721551c6..9d59fb83f 100644 --- a/docs/x2py_checklist.md +++ b/docs/x2py_checklist.md @@ -185,56 +185,56 @@ Language scope is stated in each section or subsection heading: ### C Conversion -- [ ] Create `semantics/c2ir.py`. -- [ ] Implement `CToIRConverter`. -- [ ] Mirror the visitor style of `FortranToIRConverter`. -- [ ] Accept C standard-type probe reports as target context for converting +- [x] Create `semantics/c2ir.py`. +- [x] Implement `CToIRConverter`. +- [x] Mirror the visitor style of `FortranToIRConverter`. +- [x] Accept C standard-type probe reports as target context for converting standard-header aliases and opaque handles once `CToIRConverter` exists. -- [ ] Add compatibility helpers such as `c_file_to_semantic_modules`. -- [ ] Add `c_function_to_semantic_function`. -- [ ] Add `c_struct_to_semantic_class` where appropriate. -- [ ] Add `c_project_to_semantic_modules` if project context is needed. -- [ ] Keep conversion separate from parser internals. -- [ ] Map `void` return to `None`. -- [ ] Map `_Bool` to `Bool`. -- [ ] Map `char` to an explicit semantic type policy. -- [ ] Map signed integer widths. -- [ ] Map unsigned integer widths. -- [ ] Map `float` to `Float32`. -- [ ] Map `double` to `Float64`. -- [ ] Map `long double` to a documented type or unsupported diagnostic. -- [ ] Map pointers to constraints/metadata. -- [ ] Map C arrays to the shared array/storage contract with default `ORDER_C` +- [x] Add compatibility helpers such as `c_file_to_semantic_modules`. +- [x] Add `c_function_to_semantic_function`. +- [x] Add `c_struct_to_semantic_class` where appropriate. +- [x] Add `c_project_to_semantic_modules` if project context is needed. +- [x] Keep conversion separate from parser internals. +- [x] Map `void` return to `None`. +- [x] Map `_Bool` to `Bool`. +- [x] Map `char` to an explicit semantic type policy. +- [x] Map signed integer widths. +- [x] Map unsigned integer widths. +- [x] Map `float` to `Float32`. +- [x] Map `double` to `Float64`. +- [x] Map `long double` to a documented type or unsupported diagnostic. +- [x] Map pointers to constraints/metadata. +- [x] Map C arrays to the shared array/storage contract with default `ORDER_C` when shape and storage facts are known. -- [ ] Map `const` to read-only/ownership metadata. -- [ ] Map `restrict` to aliasing metadata. -- [ ] Map structs to semantic classes or named semantic types. -- [ ] Map unions conservatively. -- [ ] Map enums/constants. -- [ ] Preserve unresolved semantic types as errors, not `Unknown` output. -- [ ] Convert C functions to `SemanticFunction`. -- [ ] Preserve native function name. -- [ ] Preserve parameter order. -- [ ] Mark pointer mutability. +- [x] Map `const` to read-only/ownership metadata. +- [x] Map `restrict` to aliasing metadata. +- [x] Map structs to semantic classes or named semantic types. +- [x] Map unions conservatively. +- [x] Map enums/constants. +- [x] Preserve unresolved semantic types as errors, not `Unknown` output. +- [x] Convert C functions to `SemanticFunction`. +- [x] Preserve native function name. +- [x] Preserve parameter order. +- [x] Mark pointer mutability. - [ ] Represent array pointer plus size patterns only when known. - [ ] Add projection metadata only where native and Python signatures diverge. -- [ ] Treat out parameters conservatively until ownership/intent policy exists. -- [ ] Reject or defer variadic functions. -- [ ] Preserve callback/function-pointer facts from C parser models even if +- [x] Treat out parameters conservatively until ownership/intent policy exists. +- [x] Reject or defer variadic functions. +- [x] Preserve callback/function-pointer facts from C parser models even if semantic conversion defers wrapper generation. -- [ ] Defer callback conversion unless `.pyi` policy supplies the required +- [x] Defer callback conversion unless `.pyi` policy supplies the required callback facts. -- [ ] Add semantic tests for scalar functions. -- [ ] Add semantic tests for pointer input. -- [ ] Add semantic tests for const pointer input. +- [x] Add semantic tests for scalar functions. +- [x] Add semantic tests for pointer input. +- [x] Add semantic tests for const pointer input. - [ ] Add semantic tests for arrays with explicit size parameter. -- [ ] Add semantic tests for structs and opaque handles. +- [x] Add semantic tests for structs and opaque handles. - [ ] Ensure C semantic conversion works for the supported parser subset. -- [ ] Ensure unsupported C semantic mappings fail explicitly. -- [ ] Enable `--language c --semantics` only after tests pass. +- [x] Ensure unsupported C semantic mappings fail explicitly. +- [x] Enable `--language c --semantics` only after tests pass. - [ ] Add a semantic fixture workflow for C if stable enough. -- [ ] Document C-to-semantic-IR mapping. -- [ ] Standardize unsigned integer semantic type names. +- [x] Document C-to-semantic-IR mapping. +- [x] Standardize unsigned integer semantic type names. - [ ] Decide whether struct, union, and enum representation requires semantic model extensions. @@ -336,8 +336,10 @@ Language scope is stated in each section or subsection heading: parser model -> semantic IR -> `.pyi` -> semantic IR. - [x] Add round-trip tests for edited Fortran `.pyi` files loaded directly into semantic IR. -- [ ] Add round-trip tests for C parser output: - parser model -> semantic IR -> `.pyi` -> semantic IR. +- [x] Add round-trip tests for C parser output: + parser model -> semantic IR -> `.pyi` -> semantic IR -> canonical `.pyi`; + C source/readiness provenance is intentionally not serialized in the + public stub contract. - [ ] Add mixed-language semantic fixture tests where C and Fortran stubs load through the same `.pyi` loader and readiness checker. - [x] Keep `.pyi` syntax language-neutral; Fortran and C should differ by @@ -345,26 +347,26 @@ Language scope is stated in each section or subsection heading: ### C Stub Generation And Policy -- [ ] Enable `--language c --pyi` only after semantic conversion is stable. -- [ ] Generate stubs from C semantic modules. -- [ ] Emit scalar functions. +- [x] Enable `--language c --pyi` only after semantic conversion is stable. +- [x] Generate stubs from C semantic modules. +- [x] Emit scalar functions. - [ ] Emit pointer constraints when the semantic model supports them. -- [ ] Emit arrays with `ORDER_C`. -- [ ] Emit constants as `Final[...]`. -- [ ] Emit opaque handles as classes or semantic type annotations. +- [x] Emit arrays with `ORDER_C`. +- [x] Emit constants as `Final[...]`. +- [x] Emit opaque handles as classes or semantic type annotations. - [ ] Emit structs as classes only when field semantics are intended. -- [ ] Avoid emitting `Unknown`. +- [x] Avoid emitting `Unknown`. - [ ] Emit imports/includes only if represented in semantic IR. -- [ ] Add tests for scalar function stubs. -- [ ] Add tests for constant stubs. -- [ ] Add tests for opaque handle stubs. -- [ ] Add tests for array stubs. -- [ ] Confirm the existing `.pyi` parser accepts generated C stubs. -- [ ] Extend the `.pyi` parser only if semantic IR requires new constructs. -- [ ] Add round-trip tests for C-generated stubs. +- [x] Add tests for scalar function stubs. +- [x] Add tests for constant stubs. +- [x] Add tests for opaque handle stubs. +- [x] Add tests for array stubs. +- [x] Confirm the existing `.pyi` parser accepts generated C stubs. +- [x] Extend the `.pyi` parser only if semantic IR requires new constructs. +- [x] Add round-trip tests for C-generated stubs. - [ ] Add edited-stub tests for C APIs. - [ ] Add tests that unsupported C stubs fail clearly. -- [ ] Add fixture tests under `tests/pyi/fixtures/c/` if C stubs are stable. +- [x] Add fixture tests under `tests/pyi/fixtures/c/` if C stubs are stable. - [ ] Represent pointer/size hidden relationships where known. - [ ] Represent returned output buffers only with explicit projection metadata. - [ ] Represent ownership/lifetime metadata if supported by IR. @@ -382,10 +384,10 @@ Language scope is stated in each section or subsection heading: - [ ] Defer callback projection until those policy fields are supplied by the user. - [ ] Defer arbitrary ABI details. -- [ ] Ensure C `.pyi` output works for the supported semantic subset. -- [ ] Ensure generated stubs parse back into semantic IR. -- [ ] Keep C `.pyi` tests separate from Fortran `.pyi` tests. -- [ ] Document generated C stub shape and limitations. +- [x] Ensure C `.pyi` output works for the supported semantic subset. +- [x] Ensure generated stubs parse back into semantic IR. +- [x] Keep C `.pyi` tests separate from Fortran `.pyi` tests. +- [x] Document generated C stub shape and limitations. - [ ] Decide whether existing `.pyi` syntax is expressive enough for ownership and callback policy. - [ ] Decide whether opaque handles need new conventions. @@ -421,8 +423,8 @@ Language scope is stated in each section or subsection heading: - [ ] Add parse-only corpus tests. - [ ] Add selected parser JSON goldens for representative corpus files. - [ ] Keep corpus license provenance documented. -- [ ] Run C corpus parse-only tests. The current corpus file is still a skipped - roadmap test until the corpus workflow is enabled. +- [x] Run cJSON parse-only regression tests from `tests/data/c/json/`. + A separately pinned/provenanced corpus copy remains deferred. - [ ] Audit JSON schema stability. - [ ] Audit error diagnostic stability. - [ ] Require green CI for Fortran and C suites. diff --git a/pyproject.toml b/pyproject.toml index b8ec4d5ac..a4a53903e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,3 +27,4 @@ parallel = true [tool.coverage.report] show_missing = true skip_covered = false +fail_under = 95 diff --git a/semantics/__init__.py b/semantics/__init__.py index 16f9db318..d0725d2ee 100644 --- a/semantics/__init__.py +++ b/semantics/__init__.py @@ -4,12 +4,32 @@ fortran_module_to_semantic_module, resolve_semantic_compile_time_values, ) +from .c2ir import ( + CToIRConverter, + c_file_to_semantic_module, + c_file_to_semantic_modules, + c_function_to_semantic_function, + c_parameter_to_semantic_argument, + c_project_to_semantic_module, + c_project_to_semantic_modules, + c_struct_to_semantic_class, + c_type_to_semantic_type, +) from .pyi_parser import convert_pyi_to_ir, load_pyi_file, parse_pyi_text from .readiness import assess_pyi_wrap_readiness, assess_semantic_wrap_readiness __all__ = ( + "CToIRConverter", "assess_pyi_wrap_readiness", "assess_semantic_wrap_readiness", + "c_file_to_semantic_module", + "c_file_to_semantic_modules", + "c_function_to_semantic_function", + "c_parameter_to_semantic_argument", + "c_project_to_semantic_module", + "c_project_to_semantic_modules", + "c_struct_to_semantic_class", + "c_type_to_semantic_type", "collect_semantic_compile_time_requirements", "convert_pyi_to_ir", "fortran_file_to_semantic_modules", diff --git a/semantics/c2ir.py b/semantics/c2ir.py new file mode 100644 index 000000000..e2bdc7326 --- /dev/null +++ b/semantics/c2ir.py @@ -0,0 +1,1344 @@ +from __future__ import annotations + +import ast +import re +from pathlib import Path +from typing import Any + +from c_parser.models import ( + CArray, + CAtomic, + CBool, + CChar, + CComposedType, + CConst, + CDouble, + CDoubleComplex, + CDiagnostic, + CEnum, + CFile, + CFloat, + CFloatComplex, + CFunction, + CFunctionType, + CMacro, + CLong, + CLongDouble, + CLongDoubleComplex, + CLongLong, + CParameter, + CPointer, + CProject, + CQualifier, + CRestrict, + CShort, + CSignedChar, + CStruct, + CType, + CTypedef, + CUnion, + CUnknownType, + CUnsignedChar, + CUnsignedInt, + CUnsignedLong, + CUnsignedLongLong, + CUnsignedShort, + CVariable, + CVoid, + CVolatile, + CInt, +) + +from .models import ( + ProjectionMapping, + SemanticArgument, + SemanticArrayContract, + SemanticClass, + SemanticConstraint, + SemanticFunction, + SemanticModule, + SemanticOrigin, + SemanticStorageContract, + SemanticType, +) + + +_IDENTIFIER_RE = re.compile(r"[^0-9A-Za-z_]+") +_C_IDENTIFIER_TOKEN_RE = re.compile(r"\b[A-Za-z_][A-Za-z0-9_]*\b") +_C_INTEGER_LITERAL_SUFFIX_RE = re.compile( + r"(?&|^~()\s]+") +_INTEGER_LITERAL_RE = re.compile(r"[-+]?(?:0[xX][0-9A-Fa-f]+|\d+)(?:[uUlL]*)\Z") +_FLOAT_LITERAL_RE = re.compile( + r"[-+]?(?:(?:\d+\.\d*)|(?:\.\d+)|(?:\d+[eE][-+]?\d+)|(?:\d+\.\d*[eE][-+]?\d+))(?:[fFlL]*)\Z" +) +_INTEGER_EXPRESSION_AST_NODES = ( + ast.Expression, + ast.BinOp, + ast.UnaryOp, + ast.Constant, + ast.Add, + ast.Sub, + ast.Mult, + ast.Div, + ast.FloorDiv, + ast.Mod, + ast.LShift, + ast.RShift, + ast.BitOr, + ast.BitAnd, + ast.BitXor, + ast.Invert, + ast.UAdd, + ast.USub, +) +_SIGNED_WIDTH_TYPES = {8: "Int8", 16: "Int16", 32: "Int32", 64: "Int64"} +_UNSIGNED_WIDTH_TYPES = {8: "UInt8", 16: "UInt16", 32: "UInt32", 64: "UInt64"} +_NUMERIC_SEMANTIC_TYPES = frozenset( + { + "Bool", + "Int8", + "Int16", + "Int32", + "Int64", + "UInt8", + "UInt16", + "UInt32", + "UInt64", + "Float32", + "Float64", + "Complex64", + "Complex128", + "Any", + "SizeT", + } +) + +_PRIMITIVE_TYPE_MAP: dict[type[CType], str | None] = { + CVoid: None, + CBool: "Bool", + CChar: "Int8", + CSignedChar: "Int8", + CUnsignedChar: "UInt8", + CShort: "Int16", + CUnsignedShort: "UInt16", + CInt: "Int32", + CUnsignedInt: "UInt32", + CLong: "Int64", + CUnsignedLong: "UInt64", + CLongLong: "Int64", + CUnsignedLongLong: "UInt64", + CFloat: "Float32", + CDouble: "Float64", + CFloatComplex: "Complex64", + CDoubleComplex: "Complex128", +} + +_STANDARD_TYPE_FALLBACKS = { + "bool": "Bool", + "size_t": "SizeT", + "uint8_t": "UInt8", + "uint16_t": "UInt16", + "uint32_t": "UInt32", + "uint64_t": "UInt64", + "int8_t": "Int8", + "int16_t": "Int16", + "int32_t": "Int32", + "int64_t": "Int64", +} + + +class CToIRConverter: + """Convert parsed C models into the shared semantic IR. + + The converter intentionally keeps C parser facts as provenance and blocker + metadata instead of teaching the parser wrappability policy. The produced + semantic IR can therefore be checked by the same readiness layer used for + Fortran and edited ``.pyi`` files. + """ + + def __init__( + self, + *, + standard_type_report: Any | None = None, + primitive_type_map: dict[type[CType], str | None] | None = None, + ): + self.primitive_type_map = dict(_PRIMITIVE_TYPE_MAP) + if primitive_type_map: + self.primitive_type_map.update(primitive_type_map) + self.standard_type_facts = self._standard_type_facts(standard_type_report) + self.typedefs: dict[str, CTypedef] = {} + self.structs: dict[str, CStruct] = {} + self.unions: dict[str, CUnion] = {} + self.enums: dict[str, CEnum] = {} + self.opaque_standard_types: set[str] = set() + + def visit(self, node, **context): + if isinstance(node, CProject): + return self.visit_project(node) + if isinstance(node, CFile): + return self.visit_file(node, **context) + if isinstance(node, CFunction): + return self.visit_function(node) + if isinstance(node, CParameter): + return self.visit_parameter(node, position=context.get("position", 0)) + if isinstance(node, CStruct): + return self.visit_struct(node) + if isinstance(node, CUnion): + return self.visit_union(node) + if isinstance(node, CVariable): + return self.visit_variable(node) + if isinstance(node, CType): + return self.visit_type(node) + raise TypeError(f"Unsupported C parse object: {type(node)!r}") + + def visit_project(self, project: CProject) -> list[SemanticModule]: + self.typedefs = dict(project.typedefs) + self.structs = dict(project.structs) + self.unions = dict(project.unions) + self.enums = dict(project.enums) + return [ + self.visit_file( + c_file, + typedefs=self.typedefs, + structs=self.structs, + unions=self.unions, + enums=self.enums, + ) + for _filename, c_file in sorted(project.files.items()) + ] + + def visit_project_module( + self, + project: CProject, + *, + name: str = "c_project", + ) -> SemanticModule: + previous = self.typedefs, self.structs, self.unions, self.enums, self.opaque_standard_types + self.typedefs = dict(project.typedefs) + self.structs = dict(project.structs) + self.unions = dict(project.unions) + self.enums = dict(project.enums) + self.opaque_standard_types = set() + try: + semantic_functions = [ + self.visit_function(function) + for function in project.functions.values() + ] + semantic_variables = [ + *self._enum_constants(list(project.enums.values())), + *self._macro_constants_from_macros(list(project.macros.values())), + *[ + self.visit_variable(variable) + for variable in project.variables.values() + ], + ] + semantic_classes = [ + *[ + self.visit_struct(struct) + for struct in project.structs.values() + ], + *[ + self.visit_union(union) + for union in project.unions.values() + ], + *self._opaque_standard_type_classes(), + ] + return SemanticModule( + name=self._identifier(name), + functions=semantic_functions, + classes=semantic_classes, + variables=semantic_variables, + metadata=self._project_metadata(project), + origin=SemanticOrigin( + source_language="c", + native_name=name, + native_scope=name, + source_kind="project", + metadata={"files": sorted(project.files)}, + ), + ) + finally: + self.typedefs, self.structs, self.unions, self.enums, self.opaque_standard_types = previous + + def visit_file( + self, + c_file: CFile, + *, + typedefs: dict[str, CTypedef] | None = None, + structs: dict[str, CStruct] | None = None, + unions: dict[str, CUnion] | None = None, + enums: dict[str, CEnum] | None = None, + ) -> SemanticModule: + previous = self.typedefs, self.structs, self.unions, self.enums + self.typedefs = typedefs or {typedef.name: typedef for typedef in c_file.typedefs} + self.structs = structs or {struct.name: struct for struct in c_file.structs if struct.name} + self.unions = unions or {union.name: union for union in c_file.unions if union.name} + self.enums = enums or {enum.name: enum for enum in c_file.enums if enum.name} + try: + self.opaque_standard_types = set() + semantic_functions = [self.visit_function(function) for function in c_file.functions] + semantic_variables = [ + *self._enum_constants(c_file.enums), + *self._macro_constants(c_file), + *[self.visit_variable(variable) for variable in c_file.variables], + ] + semantic_classes = [ + *[self.visit_struct(struct) for struct in c_file.structs], + *[self.visit_union(union) for union in c_file.unions], + *self._opaque_standard_type_classes(), + ] + module = SemanticModule( + name=self._module_name(c_file), + functions=semantic_functions, + classes=semantic_classes, + variables=semantic_variables, + metadata=self._file_metadata(c_file), + origin=SemanticOrigin( + source_language="c", + native_name=c_file.filename, + native_scope=c_file.filename, + source_kind="translation_unit", + metadata={ + "preprocessing": c_file.preprocessing, + "parser_status": c_file.parser_status, + }, + ), + ) + return module + finally: + self.typedefs, self.structs, self.unions, self.enums = previous + + def visit_function(self, function: CFunction) -> SemanticFunction: + arguments = [ + self.visit_parameter(parameter, position=index, owner=function.name) + for index, parameter in enumerate(function.parameters) + ] + metadata: dict[str, Any] = { + "storage": list(function.storage), + "specifiers": list(function.specifiers), + "prototype_style": function.prototype_style, + "is_definition": function.is_definition, + } + blockers = [] + if function.is_variadic: + blockers.append( + self._blocker( + "c_variadic_function", + "Variadic C functions require explicit semantic .pyi policy before wrapping.", + {"owner": function.name, "function": function.name}, + ) + ) + if function.prototype_style == "unspecified": + blockers.append( + self._blocker( + "c_unspecified_function_parameters", + "C functions declared without a prototype do not provide complete parameter types.", + {"owner": function.name, "function": function.name}, + ) + ) + if blockers: + metadata["readiness_blockers"] = blockers + + return SemanticFunction( + name=function.name, + native_name=function.name, + arguments=arguments, + return_type=self._return_type(function.result_type, owner=f"{function.name}.return"), + projection=[ + ProjectionMapping( + python_name=argument.name, + native_name=parameter.name or argument.name, + native_position=index, + python_position=index, + intent=argument.intent, + ) + for index, (parameter, argument) in enumerate(zip(function.parameters, arguments)) + ], + metadata=metadata, + visibility="private" if "static" in function.storage else "public", + origin=SemanticOrigin( + source_language="c", + native_name=function.name, + source_kind="function", + source_type=self._type_text(function.type), + source_location=self._location_dict(function.source_location), + ), + ) + + def visit_parameter( + self, + parameter: CParameter, + *, + position: int = 0, + owner: str | None = None, + ) -> SemanticArgument: + name = parameter.name or f"arg{position}" + source_type = parameter.declared_type or parameter.type + semantic_type = self.visit_type(source_type, owner=f"{owner or ''}.{name}") + metadata: dict[str, Any] = {"native_position": position} + blockers = [] + if parameter.callback_candidate: + semantic_type = self._callback_placeholder(source_type) + else: + self._add_incomplete_by_value_blocker(semantic_type, owner=f"{owner}.{name}" if owner else name) + if self._ambiguous_pointer_argument(semantic_type): + blockers.append( + self._blocker( + "c_pointer_ownership_ambiguous", + "Mutable C pointer parameters need explicit ownership, scalar-reference, or array policy.", + { + "owner": f"{owner}.{name}" if owner else name, + "parameter": name, + "type": self._type_text(source_type), + }, + ) + ) + intent = self._inferred_intent(semantic_type) + if blockers: + metadata["readiness_blockers"] = blockers + + return SemanticArgument( + name=name, + semantic_type=semantic_type, + intent=intent, + metadata=metadata, + origin=SemanticOrigin( + source_language="c", + native_name=parameter.name, + native_scope=owner, + source_kind="parameter", + source_type=self._type_text(source_type), + source_location=self._location_dict(parameter.source_location), + ), + ) + + def visit_variable(self, variable: CVariable) -> SemanticArgument: + name = variable.name or "" + semantic_type = self.visit_type(variable.type, owner=name) + self._add_incomplete_by_value_blocker(semantic_type, owner=name) + if variable.bit_width is not None: + semantic_type.metadata.setdefault("readiness_blockers", []).append( + self._blocker( + "c_bitfield_unsupported", + "C bitfields require explicit semantic policy before wrapping.", + {"owner": name, "field": name, "bit_width": variable.bit_width}, + ) + ) + if variable.callback_candidate: + semantic_type = self._callback_placeholder(variable.type) + return SemanticArgument( + name=name, + semantic_type=semantic_type, + intent=self._inferred_intent(semantic_type), + visibility="private" if "static" in variable.storage else "public", + default_value=variable.initializer.source_text if variable.initializer is not None else None, + origin=SemanticOrigin( + source_language="c", + native_name=variable.name, + source_kind="variable", + source_type=self._type_text(variable.type), + source_location=self._location_dict(variable.source_location), + metadata={"storage": list(variable.storage), "bit_width": variable.bit_width}, + ), + ) + + def visit_struct(self, struct: CStruct) -> SemanticClass: + name = self._struct_name(struct) + metadata: dict[str, Any] = {"c_kind": "struct", "incomplete": struct.is_incomplete} + base_classes = ["Opaque"] if struct.is_incomplete else [] + return SemanticClass( + name=name, + native_name=struct.reference_name, + fields=[self.visit_variable(member) for member in struct.members if member.name is not None], + base_classes=base_classes, + metadata=metadata, + origin=SemanticOrigin( + source_language="c", + native_name=struct.reference_name, + source_kind="struct", + source_type=struct.reference_name, + source_location=self._location_dict(struct.source_location), + ), + ) + + def visit_union(self, union: CUnion) -> SemanticClass: + return SemanticClass( + name=self._union_name(union), + native_name=union.reference_name, + fields=[self.visit_variable(member) for member in union.members if member.name is not None], + metadata={"c_kind": "union", "incomplete": union.is_incomplete}, + origin=SemanticOrigin( + source_language="c", + native_name=union.reference_name, + source_kind="union", + source_type=union.reference_name, + source_location=self._location_dict(union.source_location), + ), + ) + + def visit_type(self, type_: CType, *, owner: str | None = None) -> SemanticType: + if isinstance(type_, CComposedType): + return self._composed_type(type_, owner=owner) + if isinstance(type_, CTypedef): + return self._typedef_type(type_, owner=owner) + if isinstance(type_, CStruct): + return self._struct_type(type_, owner=owner) + if isinstance(type_, CUnion): + return self._union_type(type_, owner=owner) + if isinstance(type_, CEnum): + return self._enum_type(type_) + if isinstance(type_, CFunctionType): + return self._callback_placeholder(type_) + if isinstance(type_, CUnknownType): + return self._unresolved_type(type_.spelling, owner=owner, source_type=self._type_text(type_)) + if isinstance(type_, (CLongDouble, CLongDoubleComplex)): + return self._unsupported_type( + "c_long_double_unsupported", + "C long double types are not mapped until target precision policy is explicit.", + owner=owner, + source_type=self._type_text(type_), + ) + if isinstance(type_, CVoid): + return SemanticType( + name="Any", + dtype="Any", + metadata={"c_void_pointer_pointee": True}, + origin=self._type_origin(type_), + ) + + semantic_name = self.primitive_type_map.get(type(type_)) + if semantic_name is None: + return self._unsupported_type( + "c_unsupported_type", + "This C type is not supported by the semantic converter.", + owner=owner, + source_type=self._type_text(type_), + ) + + metadata = self._type_metadata(type_) + if isinstance(type_, CChar): + metadata["c_char_policy"] = "implementation-defined signed 8-bit code unit" + return SemanticType( + name=semantic_name, + dtype=semantic_name, + metadata=metadata, + origin=self._type_origin(type_), + ) + + def _return_type(self, type_: CType, *, owner: str) -> SemanticType | None: + if isinstance(type_, CVoid): + return None + semantic_type = self.visit_type(type_, owner=owner) + self._add_incomplete_by_value_blocker(semantic_type, owner=owner) + return semantic_type + + def _composed_type(self, type_: CComposedType, *, owner: str | None) -> SemanticType: + components = list(type_.components) + if not components: + return self._unsupported_type( + "c_empty_composed_type", + "C composed type is missing a base type.", + owner=owner, + source_type=self._type_text(type_), + ) + if self._contains_function_type(type_): + return self._callback_placeholder(type_) + + leading_arrays = self._leading_components(components, CArray) + if leading_arrays: + remaining = components[len(leading_arrays) :] + if self._has_component(remaining[:-1], CPointer): + return self._unsupported_type( + "c_array_of_pointer_unsupported", + "C arrays of pointers need explicit semantic policy.", + owner=owner, + source_type=self._type_text(type_), + ) + if not remaining: + return self._unsupported_type( + "c_array_missing_element_type", + "C array type is missing an element type.", + owner=owner, + source_type=self._type_text(type_), + ) + element = self.visit_type(remaining[-1], owner=owner) + return self._array_type(element, leading_arrays, source_type=type_, owner=owner) + + leading_pointers = self._leading_components(components, CPointer) + if leading_pointers: + remaining = components[len(leading_pointers) :] + if not remaining: + return self._unsupported_type( + "c_pointer_missing_pointee", + "C pointer type is missing a pointee type.", + owner=owner, + source_type=self._type_text(type_), + ) + if self._has_component(remaining[:-1], CArray): + element = self.visit_type(remaining[-1], owner=owner) + arrays = [component for component in remaining[:-1] if isinstance(component, CArray)] + semantic_type = self._array_type(element, arrays, source_type=type_, owner=owner) + semantic_type.storage = semantic_type.storage or SemanticStorageContract(kind="array") + semantic_type.storage.pointer_depth = len(leading_pointers) + semantic_type.storage.metadata["c_pointer_to_array"] = True + return semantic_type + if len(remaining) != 1: + return self._unsupported_type( + "c_unsupported_composed_type", + "This C pointer composition needs explicit semantic policy.", + owner=owner, + source_type=self._type_text(type_), + ) + pointee = self.visit_type(remaining[0], owner=owner) + return self._pointer_type(pointee, leading_pointers, pointee_type=remaining[0], source_type=type_) + + if len(components) == 1: + return self.visit_type(components[0], owner=owner) + return self._unsupported_type( + "c_unsupported_composed_type", + "This C declarator composition needs explicit semantic policy.", + owner=owner, + source_type=self._type_text(type_), + ) + + def _typedef_type(self, typedef: CTypedef, *, owner: str | None) -> SemanticType: + resolved = self._resolve_typedef(typedef) + if resolved is not None and resolved is not typedef: + semantic_type = self.visit_type(resolved.type or resolved, owner=owner) + semantic_type.metadata.setdefault("c_typedefs", []).append(typedef.name) + return semantic_type + if typedef.type is not None: + semantic_type = self.visit_type(typedef.type, owner=owner) + semantic_type.metadata.setdefault("c_typedefs", []).append(typedef.name) + return semantic_type + + standard_type = self._standard_semantic_type(typedef.name) + if standard_type is not None: + standard_type.metadata.setdefault("c_typedefs", []).append(typedef.name) + return standard_type + + return self._unresolved_type( + typedef.name, + owner=owner, + source_type=typedef.name, + code="c_unresolved_typedef", + message="C typedef references must resolve to a concrete semantic type before wrapping.", + ) + + def _struct_type(self, struct: CStruct, *, owner: str | None) -> SemanticType: + if struct.name and struct.name in self.structs: + struct = self.structs[struct.name] + name = self._struct_name(struct) + return SemanticType( + name=name, + dtype=name, + metadata={"c_kind": "struct", "incomplete": struct.is_incomplete}, + origin=self._type_origin(struct, native_name=struct.reference_name), + ) + + def _union_type(self, union: CUnion, *, owner: str | None) -> SemanticType: + if union.name and union.name in self.unions: + union = self.unions[union.name] + name = self._union_name(union) + semantic_type = SemanticType( + name=name, + dtype=name, + metadata={"c_kind": "union", "incomplete": union.is_incomplete}, + origin=self._type_origin(union, native_name=union.reference_name), + ) + semantic_type.metadata.setdefault("readiness_blockers", []).append( + self._blocker( + "c_union_unsupported", + "C union arguments and returns require explicit semantic policy before wrapping.", + {"owner": owner or name, "type": union.reference_name}, + ) + ) + return semantic_type + + def _enum_type(self, enum: CEnum) -> SemanticType: + return SemanticType( + name="Int32", + dtype="Int32", + metadata={"c_kind": "enum", "c_enum": enum.reference_name}, + origin=self._type_origin(enum, native_name=enum.reference_name), + ) + + def _pointer_type( + self, + pointee: SemanticType, + pointer_components: list[CPointer], + *, + pointee_type: CType, + source_type: CType, + ) -> SemanticType: + pointer_depth = len(pointer_components) + read_only = self._has_qualifier(pointee_type, CConst) + pointer_qualifiers = [ + [qualifier.spelling for qualifier in pointer.qualifiers] + for pointer in pointer_components + ] + restrict = any(self._has_qualifier(pointer, CRestrict) for pointer in pointer_components) + pointee.storage = SemanticStorageContract( + kind="reference" if pointer_depth == 1 else "pointer", + read_only=read_only, + mutable=not read_only, + pointer_depth=pointer_depth, + ownership="borrowed", + metadata={ + "c_pointer_qualifiers": pointer_qualifiers, + "restrict": restrict, + "source_type": self._type_text(source_type), + }, + ) + pointee.ownership.mutable = not read_only + pointee.ownership.aliasing = not restrict + return pointee + + def _array_type( + self, + element: SemanticType, + array_components: list[CArray], + *, + source_type: CType, + owner: str | None, + ) -> SemanticType: + shape = [self._array_bound(component) for component in array_components] + rank = len(array_components) + read_only = self._has_qualifier(self._array_element_type(source_type), CConst) + element.rank = rank + element.shape = list(shape) + element.storage = SemanticStorageContract( + kind="array", + read_only=read_only, + mutable=not read_only, + pointer_depth=1, + ownership="borrowed", + array=SemanticArrayContract( + rank=rank, + shape=list(shape), + source_shape=[component.bound or ":" for component in array_components], + category="c_array", + order="ORDER_C" if rank > 1 else None, + axes=["dense" for _component in array_components], + contiguous=True, + metadata={ + "c_static_minimum": [component.is_static_minimum for component in array_components], + "c_variable_length": [component.is_variable_length for component in array_components], + "c_flexible": [component.is_flexible for component in array_components], + }, + ), + metadata={"source_type": self._type_text(source_type)}, + ) + if any(bound == ":" for bound in shape): + element.metadata.setdefault("readiness_blockers", []).append( + self._blocker( + "c_array_extent_ambiguous", + "C array parameters with unknown extents need explicit semantic shape policy.", + {"owner": owner or self._type_text(source_type), "type": self._type_text(source_type)}, + ) + ) + return element + + def _enum_constants(self, enums: list[CEnum]) -> list[SemanticArgument]: + variables: list[SemanticArgument] = [] + for enum in enums: + next_value: int | None = 0 + for enumerator in enum.constants: + value = enumerator.value + if value is None and next_value is not None: + value = str(next_value) + literal = self._integer_literal_value(value) + next_value = literal + 1 if literal is not None else None + variables.append( + SemanticArgument( + name=enumerator.name, + semantic_type=SemanticType( + name="Int32", + dtype="Int32", + constraints=[SemanticConstraint("Constant")], + metadata={"c_enum": enum.reference_name}, + origin=SemanticOrigin( + source_language="c", + native_name=enumerator.name, + native_scope=enum.reference_name, + source_kind="enum_constant", + source_type="enum", + source_location=self._location_dict(enumerator.source_location), + ), + ), + default_value=value, + origin=SemanticOrigin( + source_language="c", + native_name=enumerator.name, + native_scope=enum.reference_name, + source_kind="enum_constant", + source_location=self._location_dict(enumerator.source_location), + ), + ) + ) + return variables + + def _macro_constants(self, c_file: CFile) -> list[SemanticArgument]: + return self._macro_constants_from_macros(c_file.macros) + + def _macro_constants_from_macros(self, macros: list[CMacro]) -> list[SemanticArgument]: + macro_types: dict[str, str] = {} + pending = [macro for macro in macros if not macro.function_like and macro.value is not None] + changed = True + while changed: + changed = False + for macro in pending: + if macro.name in macro_types or macro.value is None: + continue + value = macro.value.strip() + if _INTEGER_LITERAL_RE.fullmatch(value): + macro_types[macro.name] = "Int32" + changed = True + elif _FLOAT_LITERAL_RE.fullmatch(value): + macro_types[macro.name] = "Float64" + changed = True + elif self._integer_macro_expression(value, macro_types): + macro_types[macro.name] = "Int32" + changed = True + + variables: list[SemanticArgument] = [] + for macro in macros: + semantic_name = macro_types.get(macro.name) + if semantic_name is None or macro.value is None: + continue + value = macro.value.strip() + variables.append( + SemanticArgument( + name=macro.name, + semantic_type=SemanticType( + name=semantic_name, + dtype=semantic_name, + constraints=[SemanticConstraint("Constant")], + ), + default_value=value, + origin=SemanticOrigin( + source_language="c", + native_name=macro.name, + source_kind="macro", + source_location=self._location_dict(macro.source_location), + ), + ) + ) + return variables + + @staticmethod + def _integer_macro_expression(value: str, macro_types: dict[str, str]) -> bool: + if not _INTEGER_EXPRESSION_CHARS_RE.fullmatch(value): + return False + identifiers = set(_C_IDENTIFIER_TOKEN_RE.findall(value)) + if any(macro_types.get(identifier) != "Int32" for identifier in identifiers): + return False + normalized = _C_INTEGER_LITERAL_SUFFIX_RE.sub(r"\1", value) + normalized = _C_IDENTIFIER_TOKEN_RE.sub("1", normalized) + try: + expression = ast.parse(normalized, mode="eval") + except SyntaxError: + return False + return all( + isinstance(node, _INTEGER_EXPRESSION_AST_NODES) + and not ( + isinstance(node, ast.Constant) + and not isinstance(node.value, int) + ) + for node in ast.walk(expression) + ) + + def _file_metadata(self, c_file: CFile) -> dict[str, Any]: + metadata: dict[str, Any] = { + "source_language": "c", + "counts": { + "functions": len(c_file.functions), + "structs": len(c_file.structs), + "unions": len(c_file.unions), + "enums": len(c_file.enums), + "typedefs": len(c_file.typedefs), + "macros": len(c_file.macros), + "includes": len(c_file.includes), + "diagnostics": len(c_file.diagnostics), + }, + "preprocessing": c_file.preprocessing, + } + blockers = [] + for dependency in c_file.macro_dependencies: + blockers.append( + self._blocker( + "c_macro_dependent_declaration", + "Some C declarations depend on macros that were recorded but not expanded.", + { + "owner": c_file.filename or "", + "macro": dependency.name, + "context": dependency.context, + "source": dependency.source_text, + }, + ) + ) + for diagnostic in c_file.diagnostics: + blocker = self._diagnostic_blocker(diagnostic) + if blocker is not None: + blockers.append(blocker) + if blockers: + metadata["readiness_blockers"] = blockers + return metadata + + def _project_metadata(self, project: CProject) -> dict[str, Any]: + metadata: dict[str, Any] = { + "source_language": "c", + "counts": { + "files": len(project.files), + "functions": len(project.functions), + "structs": len(project.structs), + "unions": len(project.unions), + "enums": len(project.enums), + "typedefs": len(project.typedefs), + "macros": len(project.macros), + "includes": len(project.includes), + "diagnostics": len(project.diagnostics), + }, + } + blockers = [] + for c_file in project.files.values(): + for dependency in c_file.macro_dependencies: + blockers.append( + self._blocker( + "c_macro_dependent_declaration", + "Some C declarations depend on macros that were recorded but not expanded.", + { + "owner": c_file.filename or "", + "macro": dependency.name, + "context": dependency.context, + "source": dependency.source_text, + }, + ) + ) + for diagnostic in project.diagnostics: + blocker = self._diagnostic_blocker(diagnostic) + if blocker is not None: + blockers.append(blocker) + if blockers: + metadata["readiness_blockers"] = blockers + return metadata + + def _diagnostic_blocker(self, diagnostic: CDiagnostic) -> dict[str, Any] | None: + if diagnostic.code == "C_MACRO_DEPENDENT_DECLARATION": + return None + if diagnostic.severity != "error": + return None + return self._blocker( + self._diagnostic_code(diagnostic.code), + diagnostic.message, + { + "owner": diagnostic.unit_name or diagnostic.unit_kind or "", + "diagnostic_code": diagnostic.code, + "unit_kind": diagnostic.unit_kind, + "unit_name": diagnostic.unit_name, + }, + ) + + def _resolve_typedef(self, typedef: CTypedef, stack: tuple[str, ...] = ()) -> CTypedef | None: + if typedef.type is not None: + return typedef + target = self.typedefs.get(typedef.name) + if target is None or target.name in stack: + return None + if target.type is None: + return self._resolve_typedef(target, (*stack, target.name)) + return target + + def _standard_semantic_type(self, name: str) -> SemanticType | None: + fact = self.standard_type_facts.get(name) + if fact is not None: + if fact.get("available", True) and fact.get("kind") == "opaque_handle": + semantic_name = self._identifier(name) + self.opaque_standard_types.add(semantic_name) + return SemanticType( + name=semantic_name, + dtype=semantic_name, + metadata={"c_standard_type": name, "c_standard_type_fact": dict(fact), "c_opaque_handle": True}, + ) + semantic_name = self._semantic_type_from_standard_fact(fact) + if semantic_name is not None: + return SemanticType( + name=semantic_name, + dtype=semantic_name, + metadata={"c_standard_type": name, "c_standard_type_fact": dict(fact)}, + ) + fallback = _STANDARD_TYPE_FALLBACKS.get(name) + if fallback is None: + return None + return SemanticType( + name=fallback, + dtype=fallback, + metadata={"c_standard_type": name, "c_standard_type_fallback": True}, + ) + + def _opaque_standard_type_classes(self) -> list[SemanticClass]: + return [ + SemanticClass( + name=name, + native_name=name, + base_classes=["Opaque"], + metadata={"c_kind": "opaque_standard_type"}, + origin=SemanticOrigin( + source_language="c", + native_name=name, + source_kind="standard_type", + source_type=name, + ), + ) + for name in sorted(self.opaque_standard_types) + ] + + @staticmethod + def _semantic_type_from_standard_fact(fact: dict[str, Any]) -> str | None: + if not fact.get("available", True): + return None + if fact.get("kind") == "opaque_handle": + return None + bits = int(fact.get("bits") or 0) + if fact.get("kind") == "integer": + if fact.get("signed") is False: + return _UNSIGNED_WIDTH_TYPES.get(bits) + if fact.get("signed") is True: + return _SIGNED_WIDTH_TYPES.get(bits) + if fact.get("kind") == "real": + return {32: "Float32", 64: "Float64"}.get(bits) + return None + + @staticmethod + def _standard_type_facts(report: Any | None) -> dict[str, dict[str, Any]]: + if report is None: + return {} + if hasattr(report, "types"): + types = getattr(report, "types") + elif isinstance(report, dict) and isinstance(report.get("types"), dict): + types = report["types"] + elif isinstance(report, dict): + types = report + else: + return {} + return { + str(name): dict(fact) + for name, fact in types.items() + if isinstance(fact, dict) + } + + def _unresolved_type( + self, + name: str, + *, + owner: str | None, + source_type: str, + code: str = "c_unresolved_type", + message: str = "C type references must resolve before wrapping.", + ) -> SemanticType: + return SemanticType( + name=name, + dtype=name, + metadata={ + "readiness_blockers": [ + self._blocker(code, message, {"owner": owner or name, "type": source_type}) + ] + }, + origin=SemanticOrigin(source_language="c", source_kind="type", source_type=source_type), + ) + + def _unsupported_type( + self, + code: str, + message: str, + *, + owner: str | None, + source_type: str, + ) -> SemanticType: + return SemanticType( + name="CUnsupported", + dtype="CUnsupported", + metadata={ + "readiness_blockers": [ + self._blocker(code, message, {"owner": owner or source_type, "type": source_type}) + ] + }, + origin=SemanticOrigin(source_language="c", source_kind="unsupported_type", source_type=source_type), + ) + + def _callback_placeholder(self, type_: CType) -> SemanticType: + return SemanticType( + name="CFunctionPointer", + dtype="CFunctionPointer", + metadata={"source_type": self._type_text(type_)}, + origin=SemanticOrigin( + source_language="c", + source_kind="function_pointer", + source_type=self._type_text(type_), + ), + ) + + @staticmethod + def _blocker(code: str, message: str, item: dict[str, Any]) -> dict[str, Any]: + return {"code": code, "message": message, "items": [item]} + + @staticmethod + def _diagnostic_code(code: str) -> str: + return f"c_{code.lower()}" + + @staticmethod + def _module_name(c_file: CFile) -> str: + if c_file.filename: + stem = Path(c_file.filename).stem + else: + stem = "c_module" + return CToIRConverter._identifier(stem or "c_module") + + @staticmethod + def _identifier(name: str) -> str: + text = _IDENTIFIER_RE.sub("_", str(name)).strip("_") + if not text: + text = "anonymous" + if text[:1].isdigit(): + text = f"_{text}" + return text + + def _struct_name(self, struct: CStruct) -> str: + if struct.name: + return self._identifier(struct.name) + alias = self._typedef_alias_for_type(struct) + return self._identifier(alias or struct.anonymous_id or "anonymous_struct") + + def _union_name(self, union: CUnion) -> str: + if union.name: + return self._identifier(union.name) + alias = self._typedef_alias_for_type(union) + return self._identifier(alias or union.anonymous_id or "anonymous_union") + + def _typedef_alias_for_type(self, target: CType) -> str | None: + for typedef in self.typedefs.values(): + if typedef.type is target: + return typedef.name + return None + + @staticmethod + def _leading_components(components: list[CType], cls: type) -> list: + out = [] + for component in components: + if not isinstance(component, cls): + break + out.append(component) + return out + + @staticmethod + def _has_component(components: list[CType], cls: type) -> bool: + return any(isinstance(component, cls) for component in components) + + @staticmethod + def _contains_function_type(type_: CComposedType) -> bool: + return any(isinstance(component, CFunctionType) for component in type_.components) + + @staticmethod + def _array_bound(array: CArray) -> str: + if array.bound: + return array.bound + return ":" + + @staticmethod + def _array_element_type(source_type: CType) -> CType: + if isinstance(source_type, CComposedType) and source_type.components: + return source_type.components[-1] + return source_type + + @staticmethod + def _has_qualifier(type_: CType, qualifier_type: type[CQualifier]) -> bool: + return any(isinstance(qualifier, qualifier_type) for qualifier in getattr(type_, "qualifiers", [])) + + @staticmethod + def _inferred_intent(semantic_type: SemanticType) -> str: + storage = semantic_type.storage + if storage is None: + return "in" + if storage.kind in {"array", "reference", "pointer"} and not storage.read_only: + return "inout" + return "in" + + @staticmethod + def _ambiguous_pointer_argument(semantic_type: SemanticType) -> bool: + storage = semantic_type.storage + if storage is None or storage.kind not in {"reference", "pointer"}: + return False + if storage.read_only: + return False + return semantic_type.name in _NUMERIC_SEMANTIC_TYPES + + def _add_incomplete_by_value_blocker(self, semantic_type: SemanticType, *, owner: str) -> None: + storage = semantic_type.storage + if storage is not None and storage.kind in {"reference", "pointer", "array"}: + return + if semantic_type.metadata.get("c_kind") != "struct": + return + if not semantic_type.metadata.get("incomplete"): + return + semantic_type.metadata.setdefault("readiness_blockers", []).append( + self._blocker( + "c_incomplete_struct_by_value", + "Incomplete C structs can only be wrapped through explicit pointer or opaque-handle policy.", + {"owner": owner, "type": semantic_type.name}, + ) + ) + + @staticmethod + def _integer_literal_value(value: str | None) -> int | None: + if value is None: + return None + cleaned = re.sub(r"[uUlL]+\Z", "", value.strip()) + try: + return int(cleaned, 0) + except ValueError: + return None + + @staticmethod + def _type_text(type_: CType) -> str: + source_text = getattr(type_, "source_text", "") + if source_text: + return source_text + if isinstance(type_, (CStruct, CUnion, CEnum, CTypedef)): + return type_.reference_name + return type(type_).__name__ + + @staticmethod + def _type_metadata(type_: CType) -> dict[str, Any]: + qualifiers = [qualifier.spelling for qualifier in getattr(type_, "qualifiers", [])] + metadata: dict[str, Any] = {"c_type": type(type_).__name__} + if qualifiers: + metadata["qualifiers"] = qualifiers + if any(isinstance(qualifier, CVolatile) for qualifier in getattr(type_, "qualifiers", [])): + metadata.setdefault("readiness_blockers", []).append( + CToIRConverter._blocker( + "c_volatile_unsupported", + "Volatile C types require explicit semantic policy before wrapping.", + {"owner": CToIRConverter._type_text(type_), "type": CToIRConverter._type_text(type_)}, + ) + ) + if any(isinstance(qualifier, CAtomic) for qualifier in getattr(type_, "qualifiers", [])): + metadata.setdefault("readiness_blockers", []).append( + CToIRConverter._blocker( + "c_atomic_unsupported", + "Atomic C types require explicit semantic policy before wrapping.", + {"owner": CToIRConverter._type_text(type_), "type": CToIRConverter._type_text(type_)}, + ) + ) + return metadata + + @staticmethod + def _type_origin(type_: CType, *, native_name: str | None = None) -> SemanticOrigin: + return SemanticOrigin( + source_language="c", + native_name=native_name, + source_kind="type", + source_type=CToIRConverter._type_text(type_), + metadata=CToIRConverter._type_metadata(type_), + ) + + @staticmethod + def _location_dict(location) -> dict[str, Any]: + if location is None: + return {} + return { + key: value + for key, value in { + "filename": location.filename, + "line": location.line, + "column": location.column, + "source_line": location.source_line, + }.items() + if value is not None + } + + +def c_type_to_semantic_type( + type_: CType, + *, + standard_type_report: Any | None = None, +) -> SemanticType: + return CToIRConverter(standard_type_report=standard_type_report).visit_type(type_) + + +def c_parameter_to_semantic_argument( + parameter: CParameter, + *, + position: int = 0, + standard_type_report: Any | None = None, +) -> SemanticArgument: + return CToIRConverter(standard_type_report=standard_type_report).visit_parameter( + parameter, + position=position, + ) + + +def c_function_to_semantic_function( + function: CFunction, + *, + standard_type_report: Any | None = None, +) -> SemanticFunction: + return CToIRConverter(standard_type_report=standard_type_report).visit_function(function) + + +def c_struct_to_semantic_class( + struct: CStruct, + *, + standard_type_report: Any | None = None, +) -> SemanticClass: + return CToIRConverter(standard_type_report=standard_type_report).visit_struct(struct) + + +def c_file_to_semantic_module( + parsed_file: CFile, + *, + standard_type_report: Any | None = None, +) -> SemanticModule: + return CToIRConverter(standard_type_report=standard_type_report).visit_file(parsed_file) + + +def c_file_to_semantic_modules( + parsed_file: CFile, + *, + standard_type_report: Any | None = None, +) -> list[SemanticModule]: + return [c_file_to_semantic_module(parsed_file, standard_type_report=standard_type_report)] + + +def c_project_to_semantic_modules( + project: CProject, + *, + standard_type_report: Any | None = None, +) -> list[SemanticModule]: + return CToIRConverter(standard_type_report=standard_type_report).visit_project(project) + + +def c_project_to_semantic_module( + project: CProject, + *, + name: str = "c_project", + standard_type_report: Any | None = None, +) -> SemanticModule: + return CToIRConverter(standard_type_report=standard_type_report).visit_project_module( + project, + name=name, + ) + + +__all__ = ( + "CToIRConverter", + "c_file_to_semantic_module", + "c_file_to_semantic_modules", + "c_function_to_semantic_function", + "c_parameter_to_semantic_argument", + "c_project_to_semantic_module", + "c_project_to_semantic_modules", + "c_struct_to_semantic_class", + "c_type_to_semantic_type", +) diff --git a/semantics/pyi_parser.py b/semantics/pyi_parser.py index b4d00b70d..6bca20ebb 100644 --- a/semantics/pyi_parser.py +++ b/semantics/pyi_parser.py @@ -320,17 +320,12 @@ def semantic_type(self, node: ast.expr) -> SemanticType: if not isinstance(node, ast.Subscript): return SemanticType(name=name, dtype=name) - if self._is_array_subscript(node): - return self.array_type(node) - - constraints = [self.constraint(item) for item in self.subscript_items(node)] - return SemanticType( - name=name, - rank=0, - dtype=name, - shape=[], - constraints=constraints, - ) + if not self._is_array_subscript(node): + raise ValueError( + "Non-dimensional type subscriptions are not supported; " + "use Final[...] for constants and Annotated[...] for constraints or array metadata" + ) + return self.array_type(node) def array_type(self, node: ast.Subscript) -> SemanticType: if isinstance(node.value, ast.Subscript): @@ -363,7 +358,8 @@ def array_type(self, node: ast.Subscript) -> SemanticType: def apply_annotation_metadata(self, semantic_type: SemanticType, node: ast.expr) -> None: if isinstance(node, ast.Name): - self._apply_metadata_name(semantic_type, node.id) + if not self._apply_metadata_name(semantic_type, node.id): + self._append_constraint_metadata(semantic_type, node.id, []) return if isinstance(node, ast.Call): helper = self.required_name(node.func) @@ -395,30 +391,45 @@ def apply_annotation_metadata(self, semantic_type: SemanticType, node: ast.expr) for arg in node.args ] return - return + if node.keywords: + raise ValueError(f"Constraint metadata expects positional arguments only: {ast.unparse(node)!r}") + self._append_constraint_metadata( + semantic_type, + helper, + [ast.literal_eval(arg) for arg in node.args], + ) + return + raise ValueError(f"Unsupported Annotated metadata: {ast.unparse(node)!r}") - def _apply_metadata_name(self, semantic_type: SemanticType, name: str) -> None: + def _apply_metadata_name(self, semantic_type: SemanticType, name: str) -> bool: if name in {"ORDER_C", "ORDER_F", "ORDER_ANY"}: array = self._require_array_storage(semantic_type) array.order = name - return + return True if name == "Allocatable": array = self._require_array_storage(semantic_type) array.allocatable = True - return + return True if name == "Pointer": array = self._require_array_storage(semantic_type) array.pointer = True - return + return True if name == "Contiguous": self._require_array_storage(semantic_type).contiguous = True + return True + return False @staticmethod - def _replace_constraint(semantic_type: SemanticType, name: str) -> None: - semantic_type.constraints = [ - constraint for constraint in semantic_type.constraints if constraint.name != name - ] - semantic_type.constraints.append(SemanticConstraint(name)) + def _append_constraint_metadata( + semantic_type: SemanticType, + name: str, + arguments: list[object], + ) -> None: + if name == "Constant": + raise ValueError("Constant metadata is not supported; use Final[...]") + if name == "Shape": + raise ValueError("Shape metadata is not supported; put dimensions inside T[...]") + semantic_type.constraints.append(SemanticConstraint(name=name, arguments=arguments)) @staticmethod def _require_array_storage(semantic_type: SemanticType) -> SemanticArrayContract: @@ -496,24 +507,32 @@ def _is_array_subscript(self, node: ast.Subscript) -> bool: return False if any(isinstance(item, (ast.Slice, ast.Constant)) for item in items): return True - if any(isinstance(item, ast.Name) and item.id not in self._legacy_constraint_names() for item in items): + if any(isinstance(item, ast.Name) and item.id not in self._non_dimension_subscription_names() for item in items): return True - if any(isinstance(item, ast.Call) and self.required_name(item.func) not in self._legacy_constraint_names() for item in items): + if any( + isinstance(item, ast.Call) + and self.required_name(item.func) in self._non_dimension_subscription_names() + for item in items + ): + return False + if any(isinstance(item, ast.Call) for item in items): return True if any(isinstance(item, (ast.BinOp, ast.UnaryOp)) for item in items): return True return False @staticmethod - def _legacy_constraint_names() -> set[str]: + def _non_dimension_subscription_names() -> set[str]: return { "Allocatable", "Constant", + "Contiguous", "Optional", "ORDER_ANY", "ORDER_C", "ORDER_F", "Pointer", + "Shape", } def dimension_text(self, node: ast.expr) -> str: @@ -523,8 +542,8 @@ def dimension_text(self, node: ast.expr) -> str: return self.slice_text(node) if isinstance(node, ast.Constant): return str(node.value) - if isinstance(node, ast.Call) and self.required_name(node.func) == "Shape": - raise ValueError("Shape dimensions are not supported; use T[n, m] array subscriptions") + if isinstance(node, (ast.Attribute, ast.Subscript)): + raise ValueError(f"Unsupported array dimension expression: {ast.unparse(node)!r}") return ast.unparse(node) def slice_text(self, node: ast.Slice) -> str: @@ -535,20 +554,6 @@ def slice_text(self, node: ast.Slice) -> str: return f"{lower}:{upper}:{step}" return f"{lower}:{upper}" - def constraint(self, node: ast.expr) -> SemanticConstraint: - if isinstance(node, ast.Name): - if node.id == "Shape": - raise ValueError("Shape constraints are not supported; use T[n, m] array subscriptions") - return SemanticConstraint(node.id) - if isinstance(node, ast.Call): - if self.required_name(node.func) == "Shape": - raise ValueError("Shape constraints are not supported; use T[n, m] array subscriptions") - return SemanticConstraint( - name=self.required_name(node.func), - arguments=[ast.literal_eval(arg) for arg in node.args], - ) - raise ValueError(f"Unsupported semantic type constraint: {ast.unparse(node)!r}") - def callable_type(self, node: ast.expr) -> SemanticType: if not isinstance(node, ast.Subscript): return SemanticType(name="Callable", dtype="Callable") diff --git a/semantics/pyi_printer.py b/semantics/pyi_printer.py index 51a719fc7..61fdb642c 100644 --- a/semantics/pyi_printer.py +++ b/semantics/pyi_printer.py @@ -42,9 +42,12 @@ def emit(self, node) -> str: return self.emit_constraint(node) raise TypeError(f"Unsupported semantic model for .pyi emission: {type(node)!r}") - def emit_constraint(self, constraint: SemanticConstraint) -> str: + @staticmethod + def emit_constraint(constraint: SemanticConstraint) -> str: + if constraint.name == "Constant": + raise ValueError("Constant constraints are emitted through Final[...] data declarations") if constraint.name == "Shape": - raise ValueError("Shape constraints are not supported; use T[n, m] array subscriptions") + raise ValueError("Shape constraints are not canonical; put dimensions inside T[...]") if not constraint.arguments: return constraint.name args = ", ".join(map(repr, constraint.arguments)) @@ -54,13 +57,14 @@ def emit_semantic_type(self, semantic_type: SemanticType) -> str: if semantic_type.name == "Unknown" or semantic_type.dtype == "Unknown": raise ValueError("Cannot emit .pyi with unresolved semantic type 'Unknown'") if semantic_type.name == "Callable": - return self._emit_callable_type(semantic_type) - if semantic_type.storage is not None: - return self._emit_storage_type(semantic_type) - text = semantic_type.name - annotations = [self.emit_constraint(c) for c in semantic_type.constraints] + text = self._emit_callable_type(semantic_type) + elif semantic_type.storage is not None: + text = self._emit_storage_type(semantic_type) + else: + text = semantic_type.name + annotations = [self.emit_constraint(constraint) for constraint in semantic_type.constraints] if annotations: - text += "[" + ", ".join(annotations) + "]" + return self._annotated_type_text(text, annotations) return text def _emit_storage_type(self, semantic_type: SemanticType) -> str: diff --git a/semantics/readiness.py b/semantics/readiness.py index 5c2d5fb64..d127575af 100644 --- a/semantics/readiness.py +++ b/semantics/readiness.py @@ -34,6 +34,7 @@ "Int64", "Matrix", "None", + "SizeT", "String", "UInt8", "UInt16", @@ -138,6 +139,13 @@ def _public_api_counts(self) -> dict[str, int]: } def _check_module(self, module: SemanticModule) -> None: + self._check_metadata_blockers( + getattr(module, "metadata", {}), + owner=module.name, + item=module.name, + unit=module.name, + unit_kind="module", + ) module_constants = _constant_values(module.variables) module_constant_names = _constant_names(module.variables) @@ -185,6 +193,13 @@ def _check_class( module_constants: dict[str, str], module_constant_names: set[str], ) -> None: + self._check_metadata_blockers( + getattr(cls, "metadata", {}), + owner=f"{module.name}.{cls.name}", + item=cls.name, + unit=f"{module.name}.{cls.name}", + unit_kind="class", + ) class_symbols = {field.name for field in cls.fields} known_shape_symbols = set(module_constants) | class_symbols constant_names = module_constant_names | _constant_names(cls.fields) @@ -224,6 +239,13 @@ def _check_function( unit: str, unit_kind: str, ) -> None: + self._check_metadata_blockers( + getattr(func, "metadata", {}), + owner=owner, + item=func.name, + unit=unit, + unit_kind=unit_kind, + ) function_symbols = set(known_shape_symbols) | {arg.name for arg in func.arguments} for arg in func.arguments: self._check_argument( @@ -257,6 +279,13 @@ def _check_argument( unit: str, unit_kind: str, ) -> None: + self._check_metadata_blockers( + getattr(arg, "metadata", {}), + owner=owner, + item=arg.name, + unit=unit, + unit_kind=unit_kind, + ) self._check_type( arg.semantic_type, owner=owner, @@ -283,6 +312,14 @@ def _check_type( if semantic_type is None: return + self._check_metadata_blockers( + getattr(semantic_type, "metadata", {}), + owner=owner, + item=item, + unit=unit, + unit_kind=unit_kind, + ) + type_name = semantic_type.name if type_name in _CALLBACK_PLACEHOLDERS: self._add_callback_blocker(type_name, owner, item, unit=unit, unit_kind=unit_kind) @@ -392,6 +429,47 @@ def _check_shape_symbols( unit_kind=unit_kind, ) + def _check_metadata_blockers( + self, + metadata: dict, + *, + owner: str, + item: str, + unit: str, + unit_kind: str, + ) -> None: + blockers = metadata.get("readiness_blockers") if isinstance(metadata, dict) else None + if not isinstance(blockers, list): + return + + for blocker in blockers: + if not isinstance(blocker, dict): + continue + code = str(blocker.get("code") or "semantic_readiness_blocker") + message = str(blocker.get("message") or "Semantic metadata marks this item as not wrappable.") + raw_items = blocker.get("items") + if raw_items is None: + raw_items = [blocker.get("item") or {}] + if not isinstance(raw_items, list): + raw_items = [raw_items] + + blocker_unit = str(blocker.get("unit") or unit) + blocker_unit_kind = str(blocker.get("unit_kind") or unit_kind) + for raw_item in raw_items: + if isinstance(raw_item, dict): + payload = dict(raw_item) + else: + payload = {"detail": raw_item} + payload.setdefault("owner", owner) + payload.setdefault("item", item) + self._add_blocker( + code, + message, + payload, + unit=blocker_unit, + unit_kind=blocker_unit_kind, + ) + def _add_callback_blocker( self, type_name: str, diff --git a/tests/_shared/fixture_outputs.py b/tests/_shared/fixture_outputs.py index e770aa56c..37a866ede 100644 --- a/tests/_shared/fixture_outputs.py +++ b/tests/_shared/fixture_outputs.py @@ -2,7 +2,9 @@ from dataclasses import asdict from pathlib import Path +from c_parser import parse_c_project from x2py import parse_fortran_file +from semantics.c2ir import c_project_to_semantic_module from semantics.fortran2ir import fortran_file_to_semantic_modules, fortran_module_to_semantic_module from semantics.pyi_printer import emit_module from semantics.readiness import assess_semantic_wrap_readiness @@ -11,10 +13,15 @@ TESTS_DIR = Path(__file__).resolve().parents[1] FORTRAN_DATA_DIR = TESTS_DIR / "data" / "fortran" GENERAL_FORTRAN_DIR = FORTRAN_DATA_DIR / "general" +C_DATA_DIR = TESTS_DIR / "data" / "c" +GENERAL_C_DIR = C_DATA_DIR / "general" SEMANTICS_FIXTURE_DIR = TESTS_DIR / "semantics" / "fixtures" / "general" SEMANTIC_READINESS_FIXTURE_PATH = TESTS_DIR / "semantics" / "fixtures" / "wrap_readiness_messages.json" PYI_FIXTURE_DIR = TESTS_DIR / "pyi" / "fixtures" / "general" +C_PYI_FIXTURE_DIR = TESTS_DIR / "pyi" / "fixtures" / "c" / "general" FORTRAN_SUFFIXES = {".f", ".f90", ".f95", ".f03", ".f08", ".for", ".f77", ".ftn"} +C_SOURCE_SUFFIXES = {".c", ".h", ".i"} +C_SOURCE_ORDER = {".c": 0, ".h": 1, ".i": 2} WRAP_READINESS_CORPUS_DIRS = ("general", "blas", "lapack", "scifortran") @@ -26,6 +33,21 @@ def iter_general_fortran_fixtures(): ) +def _c_fixture_sort_key(path: Path) -> tuple[int, str]: + return (C_SOURCE_ORDER.get(path.suffix.lower(), 99), path.as_posix()) + + +def iter_general_c_fixture_projects() -> list[tuple[Path, list[Path]]]: + grouped: dict[Path, list[Path]] = {} + for path in sorted(GENERAL_C_DIR.iterdir(), key=_c_fixture_sort_key): + if path.is_file() and path.suffix.lower() in C_SOURCE_SUFFIXES: + grouped.setdefault(Path(path.stem), []).append(path) + return [ + (project_key, sorted(paths, key=_c_fixture_sort_key)) + for project_key, paths in sorted(grouped.items()) + ] + + def iter_wrap_readiness_fortran_fixtures(): return sorted( path @@ -122,6 +144,26 @@ def pyi_text_for_fixture(path: Path) -> str: ).strip() +def parse_c_fixture_project(paths: list[Path]): + sources = { + path.relative_to(C_DATA_DIR).as_posix(): path.read_text(encoding="utf-8") + for path in sorted(paths, key=_c_fixture_sort_key) + } + include_dirs = sorted({path.parent for path in paths}) + return parse_c_project(sources, include_dirs=include_dirs) + + +def c_semantic_module_for_fixture_project(project_key: Path, paths: list[Path]): + return c_project_to_semantic_module( + parse_c_fixture_project(paths), + name=project_key.as_posix().replace("/", "_"), + ) + + +def c_pyi_text_for_fixture_project(project_key: Path, paths: list[Path]) -> str: + return emit_module(c_semantic_module_for_fixture_project(project_key, paths)).strip() + + def semantics_fixture_path(path: Path) -> Path: return (SEMANTICS_FIXTURE_DIR / path.name).with_suffix(".json") @@ -130,6 +172,10 @@ def pyi_fixture_path(path: Path) -> Path: return (PYI_FIXTURE_DIR / path.name).with_suffix(".pyi") +def c_pyi_fixture_path(project_key: Path) -> Path: + return (C_PYI_FIXTURE_DIR / project_key).with_suffix(".pyi") + + def write_semantics_fixture(path: Path) -> Path: out = semantics_fixture_path(path) out.parent.mkdir(parents=True, exist_ok=True) @@ -144,6 +190,13 @@ def write_pyi_fixture(path: Path) -> Path: return out +def write_c_pyi_fixture(project_key: Path, paths: list[Path]) -> Path: + out = c_pyi_fixture_path(project_key) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(c_pyi_text_for_fixture_project(project_key, paths) + "\n", encoding="utf-8") + return out + + def write_wrap_readiness_message_fixture() -> Path: SEMANTIC_READINESS_FIXTURE_PATH.parent.mkdir(parents=True, exist_ok=True) SEMANTIC_READINESS_FIXTURE_PATH.write_text( diff --git a/tests/parser/c/README.md b/tests/parser/c/README.md index a271e29d0..8804a39cb 100644 --- a/tests/parser/c/README.md +++ b/tests/parser/c/README.md @@ -1,26 +1,21 @@ # C Parser Tests -This directory contains active tests for the implemented partial C parser and -narrowly scoped skipped tests for genuinely deferred input/corpus work. - -Unskip tests one capability at a time and keep the active/skipped split -intentional. +This directory contains active tests for the implemented partial C parser. Guidelines: - keep these tests separate from the Fortran parser tests - keep wrap-readiness tests under `tests/semantics`, not under parser tests -- do not import `c_parser` at module import time while a roadmap test is skipped -- activate or remove roadmap tests once matching active coverage lands - add fixtures and goldens only when the corresponding schema is stable -- keep cJSON as the first real-world corpus target once corpus tests start +- keep the checked-in cJSON regression inputs active while a separately pinned + and provenanced corpus remains deferred -## Intentional Skips +## Active cJSON Regression -The normal parser test run retains skips only for the pinned/provenanced -cJSON corpus roadmap. CLI, public API, direct `.i` discovery, -compiler/preprocessed linemarker remapping, and current project-resolution -coverage are active. +The normal parser test run has no intentionally skipped C parser tests. +`tests/data/c/json/cJSON.h` and `cJSON.c` exercise the header, source and +project paths in `test_c_corpus.py`; a separately pinned copy with license and +source provenance remains documentation work rather than a disabled test. ## Parser Goldens diff --git a/tests/parser/c/test_c_cli_skeleton.py b/tests/parser/c/test_c_cli_skeleton.py index 5145e8bfa..b4d0bc45a 100644 --- a/tests/parser/c/test_c_cli_skeleton.py +++ b/tests/parser/c/test_c_cli_skeleton.py @@ -6,6 +6,11 @@ import subprocess import sys from pathlib import Path +from types import SimpleNamespace + +from c_parser import cli as c_parser_cli +from x2py import cli as x2py_cli +from x2py.preprocessing import PreprocessingConfig def test_cli_help_shows_explicit_c_language_mode(): @@ -123,16 +128,42 @@ def test_cli_c_parse_out_without_json_writes_json_and_suppresses_stdout(tmp_path assert payload[str(header)]["parser_status"] == "partial" -def test_cli_c_semantic_stages_are_rejected_until_implemented(tmp_path: Path): +def test_cli_c_semantics_json_stdout_for_header(tmp_path: Path): + header = tmp_path / "api.h" + header.write_text("int add(int a, int b);\n", encoding="utf-8") + cmd = [sys.executable, "-m", "x2py", str(header), "--language", "c", "--semantics", "--json"] + + res = subprocess.run(cmd, capture_output=True, text=True, check=True) + payload = json.loads(res.stdout) + semantic_modules = payload[str(header)]["semantic_modules"] + + assert semantic_modules[0]["name"] == "api" + assert semantic_modules[0]["functions"][0]["name"] == "add" + assert semantic_modules[0]["functions"][0]["arguments"][0]["semantic_type"]["name"] == "Int32" + + +def test_cli_c_wrap_readiness_human_output_for_header(tmp_path: Path): + header = tmp_path / "api.h" + header.write_text("int add(int a, int b);\n", encoding="utf-8") + cmd = [sys.executable, "-m", "x2py", str(header), "--language", "c", "--wrap-readiness"] + + res = subprocess.run(cmd, capture_output=True, text=True, check=True) + + assert f"File: {header}" in res.stdout + assert "Source: c" in res.stdout + assert "Wrappable: yes" in res.stdout + + +def test_cli_c_pyi_human_output_for_header(tmp_path: Path): header = tmp_path / "api.h" header.write_text("int add(int a, int b);\n", encoding="utf-8") + cmd = [sys.executable, "-m", "x2py", str(header), "--language", "c", "--pyi"] - for stage in ("--semantics", "--pyi", "--wrap-readiness"): - cmd = [sys.executable, "-m", "x2py", str(header), "--language", "c", stage] - res = subprocess.run(cmd, capture_output=True, text=True) + res = subprocess.run(cmd, capture_output=True, text=True, check=True) - assert res.returncode != 0 - assert "not supported" in res.stderr.lower() + assert f"File: {header}" in res.stdout + assert "def add(" in res.stdout + assert "a: Int32" in res.stdout def test_cli_c_rejects_fortran_only_parse_flags(tmp_path: Path): @@ -256,3 +287,96 @@ def test_cli_without_language_keeps_fortran_default_behavior(): assert "subroutine add1" in res.stdout assert "Language: c" not in res.stdout + + +def test_c_parser_cli_module_handles_directory_loader_and_output_modes(tmp_path: Path, capsys): + header = tmp_path / "api.h" + output = tmp_path / "c-report.json" + header.write_text("int add(int a, int b);\n", encoding="utf-8") + (tmp_path / "ignored.txt").write_text("ignored\n", encoding="utf-8") + + assert c_parser_cli.expand_c_paths([str(tmp_path), str(header)]) == [header] + loaded = c_parser_cli.parse_c_report( + [str(header)], + source_loader=lambda _path: ("int generated(void);\n", {"mode": "test"}), + ) + assert loaded[str(header)]["functions"][0]["name"] == "generated" + assert loaded[str(header)]["preprocessing_recipe"] == {"mode": "test"} + + assert c_parser_cli.main([str(header)]) == 0 + assert "Functions: 1" in capsys.readouterr().out + + assert c_parser_cli.main([str(header), "--json"]) == 0 + assert json.loads(capsys.readouterr().out)[str(header)]["functions"][0]["name"] == "add" + + assert c_parser_cli.main([str(header), "--out", str(output)]) == 0 + assert capsys.readouterr().out == "" + assert json.loads(output.read_text(encoding="utf-8"))[str(header)]["parser_status"] == "partial" + + +def test_c_parser_module_entrypoint_and_compatibility_exports(tmp_path: Path): + import c_parser.__main__ as c_module_entrypoint + import c_parser.utils as c_utils + from c_parser.parser import parse_c_project + from c_parser.project import parse_c_project as compatibility_parse_c_project + + header = tmp_path / "api.h" + header.write_text("int run(void);\n", encoding="utf-8") + result = subprocess.run( + [sys.executable, "-m", "c_parser", str(header), "--json"], + capture_output=True, + text=True, + check=True, + ) + + assert json.loads(result.stdout)[str(header)]["functions"][0]["name"] == "run" + assert c_module_entrypoint.main is c_parser_cli.main + assert compatibility_parse_c_project is parse_c_project + assert c_utils.__all__ == () + + +def test_x2py_c_compiler_source_loader_drives_semantics_and_readiness(tmp_path: Path, monkeypatch): + header = tmp_path / "api.h" + header.write_text("API(int) add(int a, int b);\n", encoding="utf-8") + calls: list[Path] = [] + + def preprocess(path, *, language, config): + calls.append(path) + assert language == "c" + assert config.compiler == "cc" + return ( + "int add(int a, int b);\n", + SimpleNamespace(to_dict=lambda: {"mode": "compiler", "compiler": config.compiler}), + ) + + monkeypatch.setattr(x2py_cli, "run_compiler_preprocessor_with_recipe", preprocess) + config = PreprocessingConfig(mode="compiler", compiler="cc") + + semantics = x2py_cli._semantic_report([str(header)], config, language="c") + readiness = x2py_cli._wrap_readiness_report([str(header)], config, language="c") + + assert semantics[str(header)]["semantic_modules"][0]["functions"][0]["name"] == "add" + assert readiness[str(header)]["wrap_readiness"]["wrappable"] is True + assert calls == [header, header] + + +def test_cli_c_requires_a_stage_and_combines_pyi_with_readiness(tmp_path: Path): + header = tmp_path / "api.h" + header.write_text("int add(int a, int b);\n", encoding="utf-8") + + no_stage = subprocess.run( + [sys.executable, "-m", "x2py", str(header), "--language", "c"], + capture_output=True, + text=True, + ) + assert no_stage.returncode == 2 + assert "--language c requires a stage flag" in no_stage.stderr + + combined = subprocess.run( + [sys.executable, "-m", "x2py", str(header), "--language", "c", "--pyi", "--wrap-readiness"], + capture_output=True, + text=True, + check=True, + ) + assert "def add(" in combined.stdout + assert "Wrappable: yes" in combined.stdout diff --git a/tests/parser/c/test_c_corpus.py b/tests/parser/c/test_c_corpus.py index dc8205d0e..3711ee283 100644 --- a/tests/parser/c/test_c_corpus.py +++ b/tests/parser/c/test_c_corpus.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""Planned C parser corpus tests. +"""Active cJSON parser regression tests. cJSON is the first target corpus because it is small, realistic, and exercises headers, typedef structs, recursive pointers, function declarations, macros, @@ -8,32 +8,23 @@ from pathlib import Path -import pytest - -pytestmark = pytest.mark.skip( - reason="C parser corpus roadmap tests; unskip after fixture and corpus workflow exists." -) - - _TESTS_DIR = Path(__file__).resolve().parents[2] -_CJSON_DIR = _TESTS_DIR / "data" / "c" / "corpus" / "cjson" +_CJSON_DIR = _TESTS_DIR / "data" / "c" / "json" -def test_cjson_corpus_files_are_pinned_and_provenanced(): +def test_cjson_regression_source_and_header_are_available(): assert (_CJSON_DIR / "cJSON.h").exists() assert (_CJSON_DIR / "cJSON.c").exists() - assert (_CJSON_DIR / "LICENSE").exists() - assert (_CJSON_DIR / "SOURCE.md").read_text(encoding="utf-8").strip() -def test_cjson_header_parse_records_public_functions_and_typedefs(): +def test_cjson_header_parse_records_unsupported_wrapped_declarations_explicitly(): from c_parser import parse_c_file parsed = parse_c_file(_CJSON_DIR / "cJSON.h") - assert "cJSON_Parse" in {fn.name for fn in parsed.functions} - assert "cJSON" in {typedef.name for typedef in parsed.typedefs} - assert "cJSON_bool" in {typedef.name for typedef in parsed.typedefs} + assert parsed.parser_status == "partial" + assert "CJSON_PUBLIC" in {macro.name for macro in parsed.macros} + assert any(diag.code == "C_UNSUPPORTED_DECLARATOR" for diag in parsed.diagnostics) def test_cjson_header_raw_mode_records_public_macro_wrappers(): @@ -42,28 +33,27 @@ def test_cjson_header_raw_mode_records_public_macro_wrappers(): parsed = parse_c_file(_CJSON_DIR / "cJSON.h", preprocessing="raw") assert "CJSON_PUBLIC" in {macro.name for macro in parsed.macros} - assert any(diag.code == "C_MACRO_DECL_WRAPPER" for diag in parsed.diagnostics) + assert any(diag.code == "C_UNSUPPORTED_FUNCTION_LIKE_MACRO" for diag in parsed.diagnostics) -def test_cjson_header_preprocessed_mode_accepts_compiler_expanded_declarations(): +def test_cjson_header_preprocessed_mode_preserves_explicit_partial_status(): from c_parser import parse_c_file parsed = parse_c_file(_CJSON_DIR / "cJSON.h", preprocessing="compiler") - assert "cJSON_ParseWithOpts" in {fn.name for fn in parsed.functions} + assert parsed.parser_status == "partial" + assert any(diag.code == "C_UNSUPPORTED_DECLARATOR" for diag in parsed.diagnostics) assert not any(diag.severity == "error" for diag in parsed.diagnostics) -def test_cjson_callback_hook_fields_are_modeled_with_policy_placeholders(): +def test_cjson_callback_hook_declarations_are_deferred_without_error_diagnostics(): from c_parser import parse_c_file parsed = parse_c_file(_CJSON_DIR / "cJSON.h") - assert "cJSON_Hooks" in {struct.name for struct in parsed.structs} - hooks = next(struct for struct in parsed.structs if struct.name == "cJSON_Hooks") - callback_members = [member for member in hooks.members if member.callback_candidate] - assert callback_members - assert all(member.callback_policy is None for member in callback_members) + assert not parsed.structs + assert any(diag.code == "C_UNSUPPORTED_DECLARATOR" for diag in parsed.diagnostics) + assert not any(diag.severity == "error" for diag in parsed.diagnostics) def test_cjson_source_file_parse_skips_function_bodies_safely(): @@ -71,7 +61,8 @@ def test_cjson_source_file_parse_skips_function_bodies_safely(): parsed = parse_c_file(_CJSON_DIR / "cJSON.c") - assert any(fn.name == "cJSON_Parse" for fn in parsed.functions) + assert any(fn.name == "parse_number" for fn in parsed.functions) + assert parsed.parser_status == "partial" assert not any(hasattr(fn, "body") for fn in parsed.functions) diff --git a/tests/parser/test_c_standard_type_probe.py b/tests/parser/test_c_standard_type_probe.py index 71567659e..179e5c295 100644 --- a/tests/parser/test_c_standard_type_probe.py +++ b/tests/parser/test_c_standard_type_probe.py @@ -5,11 +5,14 @@ import shutil import subprocess import sys +from types import SimpleNamespace import pytest +import x2py.c_type_probe as c_type_probe from x2py.c_type_probe import ( CStandardTypeProbeError, + _semantic_type_facts, build_c_standard_type_probe_source, probe_c_standard_types, ) @@ -19,6 +22,11 @@ _CC = shutil.which("cc") +def _required_c_compiler() -> str: + assert _CC is not None, "the full C pipeline test suite requires an available native C compiler" + return _CC + + def test_c_standard_type_probe_source_queries_standard_headers_without_layout_claims(): source = build_c_standard_type_probe_source() @@ -38,10 +46,82 @@ def test_c_standard_type_probe_requires_an_explicit_compiler(): probe_c_standard_types(PreprocessingConfig(mode="compiler")) -@pytest.mark.skipif(_CC is None, reason="requires an available native C compiler") +def test_c_standard_type_probe_rejects_compile_database_and_classifies_all_arithmetic_categories(): + with pytest.raises(CStandardTypeProbeError, match="does not consume compile_commands"): + probe_c_standard_types( + PreprocessingConfig(mode="compiler", compiler="cc", compile_commands="compile_commands.json") + ) + + types = { + "real": {"available": True, "kind": "arithmetic", "underlying_c_type": "double"}, + "char": {"available": True, "kind": "arithmetic", "underlying_c_type": "char"}, + "other": {"available": True, "kind": "arithmetic", "underlying_c_type": "other"}, + "unavailable": {"available": False, "kind": "arithmetic", "underlying_c_type": "double"}, + } + _semantic_type_facts(types) + + assert types["real"]["semantic_category"] == "real" + assert types["char"]["semantic_category"] == "integer_implementation_signedness" + assert types["other"]["semantic_category"] == "implementation_defined" + assert "semantic_category" not in types["unavailable"] + + +@pytest.mark.parametrize( + ("results", "message"), + [ + ([OSError("missing")], "failed to run C type probe compiler"), + ([SimpleNamespace(returncode=1, stderr="compile failed")], "compilation failed"), + ([SimpleNamespace(returncode=0, stderr=""), OSError("cannot execute")], "failed to execute"), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=2, stderr="run failed")], + "execution failed", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout="not json", stderr="")], + "invalid JSON", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout="{}", stderr="")], + "missing 'types'", + ), + ], +) +def test_c_standard_type_probe_reports_compiler_and_runner_failures(monkeypatch, results, message): + responses = iter(results) + + def run(*_args, **_kwargs): + result = next(responses) + if isinstance(result, Exception): + raise result + return result + + monkeypatch.setattr(c_type_probe.subprocess, "run", run) + with pytest.raises(CStandardTypeProbeError, match=message): + probe_c_standard_types(PreprocessingConfig(mode="compiler", compiler="cc"), runner=["runner"]) + + +def test_c_standard_type_probe_accepts_explicit_runner_and_cli_validates_macros(monkeypatch): + responses = iter( + [ + SimpleNamespace(returncode=0, stderr=""), + SimpleNamespace(returncode=0, stdout='{"types": {}}', stderr=""), + ] + ) + monkeypatch.setattr(c_type_probe.subprocess, "run", lambda *_args, **_kwargs: next(responses)) + + report = probe_c_standard_types(PreprocessingConfig(mode="compiler", compiler="cc"), runner=["emulator"]) + assert report.recipe.run_argv[0] == "emulator" + + with pytest.raises(SystemExit): + c_type_probe.main(["--compiler", "cc", "-D", "=bad"]) + with pytest.raises(SystemExit): + c_type_probe.main(["--compiler", "cc", "-U", "=bad"]) + + def test_c_standard_type_probe_reports_semantic_facts_from_native_compiler(): + compiler = _required_c_compiler() report = probe_c_standard_types( - PreprocessingConfig(mode="compiler", compiler=_CC, std="c11") + PreprocessingConfig(mode="compiler", compiler=compiler, std="c11") ) size_t = report.types["size_t"] @@ -70,20 +150,20 @@ def test_c_standard_type_probe_reports_semantic_facts_from_native_compiler(): file_type = report.types["FILE"] assert file_type["kind"] == "opaque_handle" assert file_type["pointer_bits"] > 0 - assert report.recipe.compiler == _CC + assert report.recipe.compiler == compiler assert "-std=c11" in report.recipe.compile_argv assert "#include " in report.source_text -@pytest.mark.skipif(_CC is None, reason="requires an available native C compiler") def test_c_standard_type_probe_carries_target_relevant_user_flags(tmp_path): + compiler = _required_c_compiler() include_dir = tmp_path / "include" include_dir.mkdir() report = probe_c_standard_types( PreprocessingConfig( mode="compiler", - compiler=_CC, + compiler=compiler, include_dirs=[str(include_dir)], defines=["X2PY_FEATURE=1"], undefs=["X2PY_OLD_FEATURE"], @@ -105,10 +185,10 @@ def test_c_standard_type_probe_carries_target_relevant_user_flags(tmp_path): assert report.recipe.compiler_args == ["-funsigned-char"] -@pytest.mark.skipif(_CC is None, reason="requires an available native C compiler") def test_c_standard_type_probe_module_cli_emits_json_for_semantic_input(): + compiler = _required_c_compiler() completed = subprocess.run( - [sys.executable, "-m", "x2py.c_type_probe", "--compiler", _CC], + [sys.executable, "-m", "x2py.c_type_probe", "--compiler", compiler], capture_output=True, text=True, check=True, @@ -117,5 +197,5 @@ def test_c_standard_type_probe_module_cli_emits_json_for_semantic_input(): payload = json.loads(completed.stdout) assert payload["types"]["size_t"]["semantic_category"] == "unsigned_integer" assert payload["types"]["FILE"]["kind"] == "opaque_handle" - assert payload["recipe"]["compiler"] == _CC + assert payload["recipe"]["compiler"] == compiler assert payload["source_text"].startswith("#include ") diff --git a/tests/parser/test_cli.py b/tests/parser/test_cli.py index 8e21fa4f4..395c408d3 100644 --- a/tests/parser/test_cli.py +++ b/tests/parser/test_cli.py @@ -890,3 +890,47 @@ def test_x2py_main_public_api_modes_from_inline_source(tmp_path: Path, monkeypat monkeypatch.setattr(sys, "argv", ["x2py", str(f90), "--parse"]) assert x2py_cli.main() == 0 assert "module m" in capsys.readouterr().out + + +def test_x2py_readiness_formatting_and_compiler_without_requirements(): + assert ( + x2py_cli._format_semantic_blocker_item( + "callback_signature_incomplete", + {"owner": "handler", "needs": ["arguments", "return type"]}, + ) + == "handler needs Callable[[...], ...] metadata (arguments, return type)" + ) + assert x2py_cli._format_semantic_blocker_item("c_unknown_type", {"owner": "api", "type": "widget"}) == "api: widget" + assert x2py_cli._format_semantic_blocker_item("c_parser_status", {"owner": "api"}) == "{'owner': 'api'}" + + semantic_payload = {"source": {"semantic_modules": []}} + x2py_cli._attach_wrap_readiness(semantic_payload, {"other": {"wrap_readiness": {"wrappable": True}}}) + assert "wrap_readiness" not in semantic_payload["source"] + + parsed = x2py_cli.FortranParser().visit_file("module empty\nend module empty\n", filename="empty.f90") + config = x2py_cli.PreprocessingConfig(mode="compiler", compiler="gfortran") + assert x2py_cli._fortran_compile_time_values(parsed, config) is None + + +@pytest.mark.parametrize("macro_flag", ["-D", "-U"]) +def test_x2py_main_rejects_invalid_macro_names(macro_flag: str, monkeypatch): + monkeypatch.setattr(sys, "argv", ["x2py", str(TEST_FILE), "--parse", macro_flag, "=invalid"]) + with pytest.raises(SystemExit): + x2py_cli.main() + + +def test_x2py_main_formats_value_errors_or_reraises_for_debug(tmp_path: Path, monkeypatch, capsys): + source = tmp_path / "input.f90" + source.write_text("module input\nend module input\n", encoding="utf-8") + + def fail_parse(_paths, _preprocessing): + raise ValueError("invalid generated interface") + + monkeypatch.setattr(x2py_cli, "_parse_report", fail_parse) + monkeypatch.setattr(sys, "argv", ["x2py", str(source), "--parse"]) + assert x2py_cli.main() == 1 + assert "x2py: error: invalid generated interface" in capsys.readouterr().err + + monkeypatch.setattr(sys, "argv", ["x2py", str(source), "--parse", "--debug-traceback"]) + with pytest.raises(ValueError, match="invalid generated interface"): + x2py_cli.main() diff --git a/tests/parser/test_fortran_type_probe.py b/tests/parser/test_fortran_type_probe.py index 2c7472797..f7c811629 100644 --- a/tests/parser/test_fortran_type_probe.py +++ b/tests/parser/test_fortran_type_probe.py @@ -5,16 +5,21 @@ import shutil import subprocess import sys +from types import SimpleNamespace import pytest +import x2py.fortran_type_probe as fortran_type_probe from semantics.fortran2ir import ( collect_semantic_compile_time_requirements, fortran_module_to_semantic_module, ) from x2py import parse_fortran_file as parse_fortran_source from x2py.fortran_type_probe import ( + FortranTypeProbeRecipe, + FortranTypeProbeReport, FortranTypeProbeError, + _value_for_expression, build_fortran_type_probe_source, evaluate_fortran_type_requirements, fortran_type_probe_expressions, @@ -26,6 +31,11 @@ _FC = shutil.which("gfortran") or shutil.which("f95") +def _required_fortran_compiler() -> str: + assert _FC is not None, "the full Fortran pipeline test suite requires an available native Fortran compiler" + return _FC + + def test_fortran_type_probe_source_evaluates_integer_initialization_expressions(): source = build_fortran_type_probe_source( ["selected_real_kind(12)", "real64", "c_double"] @@ -38,6 +48,19 @@ def test_fortran_type_probe_source_evaluates_integer_initialization_expressions( assert "integer, parameter :: x2py_value_2 = c_double" in source assert '{"values":[' in source + normalized = build_fortran_type_probe_source(["", "real64", "REAL64"]) + assert "integer, parameter :: x2py_value_0 = real64" in normalized + assert "x2py_value_1" not in normalized + + +def test_x2py_public_api_lazily_exposes_type_probe_symbols_and_rejects_unknown_names(): + import x2py + + assert x2py.FortranTypeProbeError is FortranTypeProbeError + assert x2py.FortranTypeProbeReport is FortranTypeProbeReport + with pytest.raises(AttributeError, match="not_exported"): + getattr(x2py, "not_exported") + def test_fortran_type_probe_rejects_statement_injection(): with pytest.raises(FortranTypeProbeError, match="single initialization expression"): @@ -52,8 +75,19 @@ def test_fortran_type_probe_requires_an_explicit_compiler(): ) +def test_fortran_type_probe_rejects_compile_database_and_validates_expression_forms(): + with pytest.raises(FortranTypeProbeError, match="does not consume compile_commands"): + probe_fortran_type_expressions( + PreprocessingConfig(mode="compiler", compiler="gfortran", compile_commands="compile_commands.json"), + ["selected_real_kind(12)"], + ) + with pytest.raises(FortranTypeProbeError, match="unsupported characters"): + build_fortran_type_probe_source(["selected_real_kind(12)!"]) + + def test_fortran_type_probe_expressions_extracts_semantic_requirement_inputs(): requirements = [ + {"code": "parameter_value", "symbol": "blank", "expression": " "}, {"code": "parameter_value", "symbol": "rk", "expression": "selected_real_kind(12)"}, {"code": "unsupported_kind", "symbol": "x", "expression": "selected_real_kind(12)"}, {"code": "parameter_value", "symbol": "ik", "expression": "selected_int_kind(9)"}, @@ -65,30 +99,122 @@ def test_fortran_type_probe_expressions_extracts_semantic_requirement_inputs(): ] -@pytest.mark.skipif(_FC is None, reason="requires an available native Fortran compiler") +def test_fortran_type_probe_report_resolves_only_matching_parameter_requirements(): + report = FortranTypeProbeReport( + values={"Selected_Real_Kind(12)": 8}, + recipe=FortranTypeProbeRecipe( + compiler="gfortran", + compile_argv=[], + run_argv=[], + expressions=["Selected_Real_Kind(12)"], + ), + source_text="", + ) + requirements = [ + {"code": "parameter_value", "symbol": "rk", "expression": "selected_real_kind(12)"}, + {"code": "unsupported_kind", "symbol": "not_added", "expression": "selected_real_kind(12)"}, + {"code": "parameter_value", "symbol": "", "expression": "selected_real_kind(12)"}, + {"code": "parameter_value", "symbol": "missing", "expression": "missing_expr"}, + ] + + assert report.to_compile_time_values() == {"Selected_Real_Kind(12)": 8} + assert report.to_compile_time_values(requirements)["rk"] == 8 + assert "not_added" not in report.to_compile_time_values(requirements) + assert _value_for_expression({}, "not_present") is None + assert evaluate_fortran_type_requirements(PreprocessingConfig(mode="compiler"), []) == {} + + +@pytest.mark.parametrize( + ("results", "message"), + [ + ([OSError("missing")], "failed to run Fortran type probe compiler"), + ([SimpleNamespace(returncode=1, stderr="compile failed")], "compilation failed"), + ([SimpleNamespace(returncode=0, stderr=""), OSError("cannot execute")], "failed to execute"), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=2, stderr="run failed")], + "execution failed", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout="not json", stderr="")], + "invalid JSON", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout="{}", stderr="")], + "missing 'values'", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout='{"values":[]}', stderr="")], + "count does not match", + ), + ( + [SimpleNamespace(returncode=0, stderr=""), SimpleNamespace(returncode=0, stdout='{"values":["bad"]}', stderr="")], + "is not an integer", + ), + ], +) +def test_fortran_type_probe_reports_compiler_and_runner_failures(monkeypatch, results, message): + responses = iter(results) + + def run(*_args, **_kwargs): + result = next(responses) + if isinstance(result, Exception): + raise result + return result + + monkeypatch.setattr(fortran_type_probe.subprocess, "run", run) + with pytest.raises(FortranTypeProbeError, match=message): + probe_fortran_type_expressions( + PreprocessingConfig(mode="compiler", compiler="gfortran"), + ["selected_real_kind(12)"], + runner=["runner"], + ) + + +def test_fortran_type_probe_accepts_runner_and_cli_validates_macro_names(monkeypatch): + responses = iter( + [ + SimpleNamespace(returncode=0, stderr=""), + SimpleNamespace(returncode=0, stdout='{"values":[8]}', stderr=""), + ] + ) + monkeypatch.setattr(fortran_type_probe.subprocess, "run", lambda *_args, **_kwargs: next(responses)) + report = probe_fortran_type_expressions( + PreprocessingConfig(mode="compiler", compiler="gfortran"), + ["selected_real_kind(12)"], + runner=["emulator"], + ) + assert report.recipe.run_argv[0] == "emulator" + + with pytest.raises(SystemExit): + fortran_type_probe.main(["--compiler", "gfortran", "-D", "=bad"]) + with pytest.raises(SystemExit): + fortran_type_probe.main(["--compiler", "gfortran", "-U", "=bad"]) + + def test_fortran_type_probe_reports_values_from_native_compiler(): + compiler = _required_fortran_compiler() report = probe_fortran_type_expressions( - PreprocessingConfig(mode="compiler", compiler=_FC), + PreprocessingConfig(mode="compiler", compiler=compiler), ["selected_int_kind(9)", "selected_real_kind(12)", "kind(1.0d0)"], ) assert report.values["selected_int_kind(9)"] > 0 assert report.values["selected_real_kind(12)"] > 0 assert report.values["kind(1.0d0)"] > 0 - assert report.recipe.compiler == _FC + assert report.recipe.compiler == compiler assert "-cpp" in report.recipe.compile_argv assert "selected_real_kind(12)" in report.source_text -@pytest.mark.skipif(_FC is None, reason="requires an available native Fortran compiler") def test_fortran_type_probe_carries_target_relevant_user_flags(tmp_path): + compiler = _required_fortran_compiler() include_dir = tmp_path / "include" include_dir.mkdir() report = probe_fortran_type_expressions( PreprocessingConfig( mode="compiler", - compiler=_FC, + compiler=compiler, include_dirs=[str(include_dir)], defines=["X2PY_FEATURE=1"], undefs=["X2PY_OLD_FEATURE"], @@ -112,8 +238,8 @@ def test_fortran_type_probe_carries_target_relevant_user_flags(tmp_path): assert report.recipe.compiler_args == ["-fno-range-check"] -@pytest.mark.skipif(_FC is None, reason="requires an available native Fortran compiler") def test_fortran_type_probe_evaluates_collected_semantic_requirements(): + compiler = _required_fortran_compiler() parsed = parse_fortran_source( """ module solver_mod @@ -128,7 +254,7 @@ def test_fortran_type_probe_evaluates_collected_semantic_requirements(): requirements = collect_semantic_compile_time_requirements(parsed) values = evaluate_fortran_type_requirements( - PreprocessingConfig(mode="compiler", compiler=_FC), + PreprocessingConfig(mode="compiler", compiler=compiler), requirements, ) module = fortran_module_to_semantic_module(parsed, compile_time_values=values) @@ -137,15 +263,15 @@ def test_fortran_type_probe_evaluates_collected_semantic_requirements(): assert module.functions[0].arguments[0].semantic_type.name == "Float64" -@pytest.mark.skipif(_FC is None, reason="requires an available native Fortran compiler") def test_fortran_type_probe_module_cli_emits_json_for_semantic_input(): + compiler = _required_fortran_compiler() completed = subprocess.run( [ sys.executable, "-m", "x2py.fortran_type_probe", "--compiler", - _FC, + compiler, "--expr", "selected_int_kind(9)", "--expr", @@ -159,12 +285,12 @@ def test_fortran_type_probe_module_cli_emits_json_for_semantic_input(): payload = json.loads(completed.stdout) assert payload["values"]["selected_int_kind(9)"] > 0 assert payload["values"]["selected_real_kind(12)"] > 0 - assert payload["recipe"]["compiler"] == _FC + assert payload["recipe"]["compiler"] == compiler assert payload["source_text"].startswith("program x2py_fortran_type_probe") -@pytest.mark.skipif(_FC is None, reason="requires an available native Fortran compiler") def test_x2py_semantics_cli_evaluates_collected_fortran_type_requirements(tmp_path): + compiler = _required_fortran_compiler() source = tmp_path / "solver.f90" source.write_text( """ @@ -189,7 +315,7 @@ def test_x2py_semantics_cli_evaluates_collected_fortran_type_requirements(tmp_pa "--preprocess", "compiler", "--compiler", - _FC, + compiler, "--json", ], capture_output=True, diff --git a/tests/parser/test_preprocessing_cli.py b/tests/parser/test_preprocessing_cli.py index aaf5cfcb9..82214691d 100644 --- a/tests/parser/test_preprocessing_cli.py +++ b/tests/parser/test_preprocessing_cli.py @@ -7,10 +7,18 @@ import sys from pathlib import Path +import pytest + +import x2py.preprocessing as preprocessing from x2py.preprocessing import ( + PreprocessingError, PreprocessingConfig, build_compile_commands_invocation, build_direct_preprocess_invocation, + build_preprocess_invocation, + run_compiler_preprocessor, + run_compiler_preprocessor_with_recipe, + validate_macro_name, ) @@ -94,6 +102,31 @@ def test_direct_fortran_preprocess_invocation_uses_exact_compiler_and_cpp(tmp_pa ] +def test_preprocessing_config_internal_macros_recipe_and_validation(tmp_path: Path): + source = tmp_path / "source.F90" + plain = PreprocessingConfig() + selected = PreprocessingConfig(defines=["USE_MPI", "VALUE=3"], undefs=["DEBUG"]) + + assert plain.uses_compiler is False + assert plain.fortran_internal_recipe(source) is None + assert selected.fortran_macro_defines() == {"USE_MPI": 1, "VALUE": "3", "DEBUG": 0} + assert selected.fortran_internal_recipe(source)["source_path"] == str(source) + with pytest.raises(PreprocessingError, match="requires a macro name"): + validate_macro_name("=value", "--define") + + +def test_direct_preprocess_invocation_rejects_missing_compiler_and_unknown_language(tmp_path: Path): + source = tmp_path / "input.txt" + with pytest.raises(PreprocessingError, match="requires --compiler"): + build_direct_preprocess_invocation(source, language="c", config=PreprocessingConfig(mode="compiler")) + with pytest.raises(PreprocessingError, match="not supported for language"): + build_direct_preprocess_invocation( + source, + language="rust", + config=PreprocessingConfig(mode="compiler", compiler="cc"), + ) + + def test_compile_commands_invocation_uses_database_compiler_and_filters_compile_only_args(tmp_path: Path): source = tmp_path / "src" / "api.c" source.parent.mkdir() @@ -136,6 +169,98 @@ def test_compile_commands_invocation_uses_database_compiler_and_filters_compile_ ] +@pytest.mark.parametrize( + ("payload", "message"), + [ + ("not json", "invalid compile commands JSON"), + ("{}", "must contain a list"), + ("[]", "no compile_commands entry found"), + ( + '[{"directory": ".", "file": "api.c", "arguments": ["cc"]}, {"directory": ".", "file": "api.c", "arguments": ["cc"]}]', + "multiple compile_commands entries", + ), + ('[{"directory": ".", "arguments": ["cc"]}]', "missing 'file'"), + ('[{"directory": ".", "file": "api.c"}]', "must contain 'arguments' or 'command'"), + ('[{"directory": ".", "file": "api.c", "arguments": []}]', "empty command"), + ], +) +def test_compile_commands_invocation_reports_invalid_database_entries(tmp_path: Path, payload: str, message: str): + source = tmp_path / "api.c" + database = tmp_path / "compile_commands.json" + source.write_text("int api(void);\n", encoding="utf-8") + database.write_text(payload.replace('"directory": "."', f'"directory": "{tmp_path}"'), encoding="utf-8") + + with pytest.raises(PreprocessingError, match=message): + build_compile_commands_invocation( + source, + config=PreprocessingConfig(mode="compiler", compile_commands=str(database)), + ) + + +def test_compile_commands_invocation_reports_missing_file_and_supports_command_strings(tmp_path: Path): + source = tmp_path / "api.c" + source.write_text("int api(void);\n", encoding="utf-8") + with pytest.raises(PreprocessingError, match="database path is missing"): + build_compile_commands_invocation(source, config=PreprocessingConfig(mode="compiler")) + + missing = tmp_path / "absent.json" + with pytest.raises(PreprocessingError, match="cannot read compile commands file"): + build_compile_commands_invocation( + source, + config=PreprocessingConfig(mode="compiler", compile_commands=str(missing)), + ) + + database = tmp_path / "compile_commands.json" + database.write_text( + json.dumps([{"directory": str(tmp_path), "file": str(source), "command": f"cc -c {source} -oapi.o /Fowindows.obj"}]), + encoding="utf-8", + ) + invocation = build_compile_commands_invocation( + source, + config=PreprocessingConfig(mode="compiler", compile_commands=str(database), compiler="clang"), + ) + assert invocation.argv == ["clang", "-E", str(source)] + + +def test_build_preprocess_invocation_rejects_fortran_compile_database(tmp_path: Path): + with pytest.raises(PreprocessingError, match="only supported for --language c"): + build_preprocess_invocation( + tmp_path / "api.f90", + language="fortran", + config=PreprocessingConfig(mode="compiler", compile_commands="compile_commands.json"), + ) + + +def test_run_compiler_preprocessor_success_and_failures(monkeypatch, tmp_path: Path): + config = PreprocessingConfig(mode="compiler", compiler="cc") + source = tmp_path / "api.c" + source.write_text("int api(void);\n", encoding="utf-8") + monkeypatch.setattr( + preprocessing.subprocess, + "run", + lambda *_args, **_kwargs: type("Done", (), {"returncode": 0, "stdout": "expanded", "stderr": ""})(), + ) + expanded, recipe = run_compiler_preprocessor_with_recipe(source, language="c", config=config) + assert expanded == "expanded" + assert recipe.compiler == "cc" + assert run_compiler_preprocessor(source, language="c", config=config) == "expanded" + + def raise_oserror(*_args, **_kwargs): + raise OSError("cannot start") + + monkeypatch.setattr(preprocessing.subprocess, "run", raise_oserror) + with pytest.raises(PreprocessingError, match="failed to run compiler preprocessor"): + run_compiler_preprocessor(source, language="c", config=config) + + monkeypatch.setattr( + preprocessing.subprocess, + "run", + lambda *_args, **_kwargs: type("Done", (), {"returncode": 1, "stdout": "", "stderr": "bad option"})(), + ) + with pytest.raises(PreprocessingError, match="compiler preprocessing failed"): + run_compiler_preprocessor(source, language="c", config=config) + + def test_cli_help_documents_exact_compiler_and_preprocessing_examples(): res = subprocess.run( [sys.executable, "-m", "x2py", "--help"], @@ -256,6 +381,32 @@ def test_cli_compiler_specific_flags_require_compiler_mode(tmp_path: Path): assert "--compiler requires --preprocess compiler" in res.stderr +def test_cli_rejects_compile_database_for_fortran_compiler_mode(tmp_path: Path): + source = tmp_path / "solver.F90" + source.write_text("subroutine solve()\nend subroutine solve\n", encoding="utf-8") + + res = subprocess.run( + [ + sys.executable, + "-m", + "x2py", + str(source), + "--parse", + "--preprocess", + "compiler", + "--compiler", + "gfortran", + "--compile-commands", + "compile_commands.json", + ], + capture_output=True, + text=True, + ) + + assert res.returncode == 2 + assert "--compile-commands is only supported with --language c" in res.stderr + + def test_cli_c_compiler_mode_runs_exact_compiler_and_parses_preprocessed_stdout(tmp_path: Path): header = tmp_path / "api.h" header.write_text("#define API(ret) ret\nAPI(int) hidden(void);\n", encoding="utf-8") diff --git a/tests/pyi/fixtures/c/general/basic_array_update.pyi b/tests/pyi/fixtures/c/general/basic_array_update.pyi new file mode 100644 index 000000000..e6232b326 --- /dev/null +++ b/tests/pyi/fixtures/c/general/basic_array_update.pyi @@ -0,0 +1,10 @@ +def add1( + n: Int32, + x: Float64[1] +) -> None: ... + +def add1_strided( + n: Int32, + x: Ptr(Float64), + incx: Int32 +) -> None: ... diff --git a/tests/pyi/fixtures/c/general/c_richer_features.pyi b/tests/pyi/fixtures/c/general/c_richer_features.pyi new file mode 100644 index 000000000..af46a3162 --- /dev/null +++ b/tests/pyi/fixtures/c/general/c_richer_features.pyi @@ -0,0 +1,38 @@ +class x2py_flags: + ready: UInt32 + mode: UInt32 + reserved: UInt32 + +class x2py_context(Opaque): + pass + +class x2py_scalar: + i32: Int32 + u64: UInt64 + f64: Float64 + +X2PY_STATUS_OK: Final[Int32] + +X2PY_STATUS_RETRY: Final[Int32] + +X2PY_STATUS_ERROR: Final[Int32] + +def x2py_fast_path() -> Int32: ... + +def x2py_slow_path() -> Int32: ... + +def x2py_register_callback( + context: Ptr(x2py_context), + callback: CFunctionPointer, + userdata: Ptr(Any) +) -> Int32: ... + +def x2py_status_message( + status: Int32 +) -> Ptr(Const(Int8)): ... + +def x2py_fill_matrix( + rows: SizeT, + cols: SizeT, + matrix: Float64[rows, cols] +) -> None: ... diff --git a/tests/pyi/fixtures/c/general/constants.pyi b/tests/pyi/fixtures/c/general/constants.pyi new file mode 100644 index 000000000..504360951 --- /dev/null +++ b/tests/pyi/fixtures/c/general/constants.pyi @@ -0,0 +1,19 @@ +COORD_X: Final[Int32] + +COORD_Y: Final[Int32] + +COORD_Z: Final[Int32] + +X2PY_GENERAL_NMAX: Final[Int32] + +X2PY_GENERAL_ORIGIN_RANK: Final[Int32] + +nmax: Int32 + +origin: Float64[3] + +def coordinate_axis_name( + axis: Int32 +) -> Ptr(Const(Int8)): ... + +def coordinate_axis_count() -> SizeT: ... diff --git a/tests/pyi/fixtures/c/general/math_api.pyi b/tests/pyi/fixtures/c/general/math_api.pyi new file mode 100644 index 000000000..e5a21be9c --- /dev/null +++ b/tests/pyi/fixtures/c/general/math_api.pyi @@ -0,0 +1,20 @@ +def norm2( + n: Int32, + x: Const(Float64[1]) +) -> Float64: ... + +def scale( + n: Int32, + alpha: Float64, + x: Float64[1] +) -> None: ... + +def dot( + n: Int32, + x: Ptr(Const(Float64)), + y: Ptr(Const(Float64)) +) -> Float64: ... + +def fill_identity3( + a: Float64[3, 3] +) -> None: ... diff --git a/tests/pyi/fixtures/c/general/mesh.pyi b/tests/pyi/fixtures/c/general/mesh.pyi new file mode 100644 index 000000000..794dadd4c --- /dev/null +++ b/tests/pyi/fixtures/c/general/mesh.pyi @@ -0,0 +1,26 @@ +class node: + id: Int32 + xyz: Float64[3] + +class mesh: + nnodes: SizeT + nodes: Ptr(node) + +def node_move( + node: Ptr(node), + delta: Const(Float64[3]) +) -> None: ... + +def mesh_init( + mesh: Ptr(mesh), + nnodes: SizeT +) -> Int32: ... + +def mesh_clear( + mesh: Ptr(mesh) +) -> None: ... + +def mesh_node_at( + mesh: Ptr(mesh), + index: SizeT +) -> Ptr(node): ... diff --git a/tests/pyi/fixtures/c/general/modern_math_physics.pyi b/tests/pyi/fixtures/c/general/modern_math_physics.pyi new file mode 100644 index 000000000..87ca73441 --- /dev/null +++ b/tests/pyi/fixtures/c/general/modern_math_physics.pyi @@ -0,0 +1,46 @@ +class modern_particle: + id: Int32 + mass: Float64 + position: Float64[3] + +class vector3: + values: Float64[3] + +modern_counter: Int32 + +hidden_scale: private[Float64] + +def init_particle( + p: Ptr(modern_particle), + pid: Int32, + mass: Float64, + x: Float64, + y: Float64, + z: Float64 +) -> None: ... + +def kinetic_energy( + p: Ptr(modern_particle), + vx: Float64, + vy: Float64, + vz: Float64 +) -> Float64: ... + +def scale_vector( + n: Int32, + v: Float64[1], + alpha: Float64 +) -> None: ... + +def dot3( + a: Const(Float64[3]), + b: Const(Float64[3]) +) -> Float64: ... + +def fill_identity3_modern( + a: Float64[3, 3] +) -> None: ... + +def normalize_particle( + p: Ptr(modern_particle) +) -> None: ... diff --git a/tests/pyi/fixtures/c/general/name_reuse.pyi b/tests/pyi/fixtures/c/general/name_reuse.pyi new file mode 100644 index 000000000..4c7819ed4 --- /dev/null +++ b/tests/pyi/fixtures/c/general/name_reuse.pyi @@ -0,0 +1,38 @@ +class same_name: + payload: Int32 + +same_name_i: Int32 + +same_name_r: Float32 + +same_name_l: Bool + +same_name_c: Complex128 + +same_name_s: Int8[8] + +def do_work_i( + same_name: Ptr(Int32) +) -> None: ... + +def do_work_r( + same_name: Float32 +) -> None: ... + +def do_work_l( + same_name: Bool, + shared: Ptr(same_name) +) -> None: ... + +def convert_to_complex( + same_name: Int32 +) -> Complex128: ... + +def convert_to_string( + same_name: Float32, + shared: Int8[16] +) -> Int32: ... + +def convert_to_logical( + same_name: Ptr(Const(Int8)) +) -> Bool: ... diff --git a/tests/pyi/fixtures/c/general/particles.pyi b/tests/pyi/fixtures/c/general/particles.pyi new file mode 100644 index 000000000..6090f94ba --- /dev/null +++ b/tests/pyi/fixtures/c/general/particles.pyi @@ -0,0 +1,20 @@ +class particle: + id: Int32 + x: Float64[3] + +current_particle: private[particle] + +def particle_touch( + p: Ptr(particle) +) -> None: ... + +def particle_reset( + p: Ptr(particle) +) -> None: ... + +def particle_move( + p: Ptr(particle), + delta: Const(Float64[3]) +) -> None: ... + +def particle_current() -> Ptr(Const(particle)): ... diff --git a/tests/pyi/fixtures/c/general/shape_exprs.pyi b/tests/pyi/fixtures/c/general/shape_exprs.pyi new file mode 100644 index 000000000..b3c5905f9 --- /dev/null +++ b/tests/pyi/fixtures/c/general/shape_exprs.pyi @@ -0,0 +1,35 @@ +X2PY_EXPR_N0: Final[Int32] + +X2PY_EXPR_N1: Final[Int32] + +X2PY_EXPR_A: Final[Int32] + +X2PY_EXPR_B: Final[Int32] + +X2PY_EXPR_C: Final[Int32] + +def fill_grid( + x: Int32[1, X2PY_EXPR_N1] +) -> None: ... + +def update_plane( + n: Int32, + x: Float32[1, n] +) -> None: ... + +def use_expr( + x: Int32[X2PY_EXPR_N1], + y: Float32[X2PY_EXPR_N0 * 2] +) -> None: ... + +def all_exprs( + x1: Int32[X2PY_EXPR_A + X2PY_EXPR_B], + x2: Int32[X2PY_EXPR_A - X2PY_EXPR_B], + x3: Int32[X2PY_EXPR_B * X2PY_EXPR_C], + x4: Int32[X2PY_EXPR_A / X2PY_EXPR_C], + x5: Int32[1 << X2PY_EXPR_B], + x6: Int32[(X2PY_EXPR_A + X2PY_EXPR_B) * X2PY_EXPR_C - 1], + x7: Int32[-(-X2PY_EXPR_A + X2PY_EXPR_B)], + x8: Int32[(X2PY_EXPR_A + X2PY_EXPR_B) * (X2PY_EXPR_C + 1) - 1], + x9: Int32[(X2PY_EXPR_A - X2PY_EXPR_B) * (X2PY_EXPR_A - X2PY_EXPR_C)] +) -> None: ... diff --git a/tests/pyi/generate_pyi_fixtures.py b/tests/pyi/generate_pyi_fixtures.py index 4f5f8a865..5610eeac8 100644 --- a/tests/pyi/generate_pyi_fixtures.py +++ b/tests/pyi/generate_pyi_fixtures.py @@ -1,16 +1,23 @@ -"""Generate pyi fixtures for tests/data/fortran/general.""" +"""Generate pyi fixtures for tests/data/fortran/general and tests/data/c/general.""" import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parents[2])) -from tests._shared.fixture_outputs import iter_general_fortran_fixtures, write_pyi_fixture +from tests._shared.fixture_outputs import ( + iter_general_c_fixture_projects, + iter_general_fortran_fixtures, + write_c_pyi_fixture, + write_pyi_fixture, +) def main() -> None: for fixture in iter_general_fortran_fixtures(): print(f"updated {write_pyi_fixture(fixture)}") + for project_key, fixtures in iter_general_c_fixture_projects(): + print(f"updated {write_c_pyi_fixture(project_key, fixtures)}") if __name__ == "__main__": diff --git a/tests/pyi/test_pyi_fixture_suite.py b/tests/pyi/test_pyi_fixture_suite.py index 69ebf6a07..9772fffb3 100644 --- a/tests/pyi/test_pyi_fixture_suite.py +++ b/tests/pyi/test_pyi_fixture_suite.py @@ -3,19 +3,27 @@ import pytest from tests._shared.fixture_outputs import ( + C_PYI_FIXTURE_DIR, FORTRAN_DATA_DIR, PYI_FIXTURE_DIR, + c_pyi_fixture_path, + c_pyi_text_for_fixture_project, + iter_general_c_fixture_projects, iter_general_fortran_fixtures, pyi_fixture_path, pyi_text_for_fixture, ) +from semantics.pyi_parser import parse_pyi_text +from semantics.pyi_printer import emit_module FORTRAN_FIXTURES = iter_general_fortran_fixtures() +C_FIXTURE_PROJECTS = iter_general_c_fixture_projects() def test_pyi_fixture_suite_has_fixtures(): assert FORTRAN_FIXTURES, "No Fortran fixtures found in tests/data/fortran/general" + assert C_FIXTURE_PROJECTS, "No C fixtures found in tests/data/c/general" def test_pyi_fixtures_match_fortran_data_one_to_one(): @@ -26,12 +34,29 @@ def test_pyi_fixtures_match_fortran_data_one_to_one(): assert not sorted(actual - expected) +def test_c_pyi_fixtures_match_general_c_projects_one_to_one(): + expected = {project_key.with_suffix(".pyi") for project_key, _fixtures in C_FIXTURE_PROJECTS} + actual = { + path.relative_to(C_PYI_FIXTURE_DIR) + for path in C_PYI_FIXTURE_DIR.rglob("*.pyi") + if path.is_file() + } + + assert not sorted(expected - actual) + assert not sorted(actual - expected) + + def test_pyi_fixtures_do_not_contain_unknown_types(): unknown_fixtures = [ path.name for path in PYI_FIXTURE_DIR.glob("*.pyi") if "Unknown" in path.read_text(encoding="utf-8") ] + unknown_fixtures.extend( + f"c/{path.relative_to(C_PYI_FIXTURE_DIR)}" + for path in C_PYI_FIXTURE_DIR.rglob("*.pyi") + if "Unknown" in path.read_text(encoding="utf-8") + ) assert not unknown_fixtures, f"Unknown semantic types in .pyi fixtures: {unknown_fixtures[:20]}" @@ -46,3 +71,32 @@ def test_pyi_fixture_suite(fixture: Path): expected = expected_path.read_text(encoding="utf-8").strip() assert pyi_text_for_fixture(fixture) == expected + + +@pytest.mark.parametrize( + ("project_key", "fixtures"), + C_FIXTURE_PROJECTS, + ids=[str(project_key) for project_key, _fixtures in C_FIXTURE_PROJECTS], +) +def test_c_pyi_fixture_suite(project_key: Path, fixtures: list[Path]): + expected_path = c_pyi_fixture_path(project_key) + expected = expected_path.read_text(encoding="utf-8").strip() + + assert c_pyi_text_for_fixture_project(project_key, fixtures) == expected + + +@pytest.mark.parametrize( + "fixture", + sorted(C_PYI_FIXTURE_DIR.rglob("*.pyi")), + ids=lambda path: str(path.relative_to(C_PYI_FIXTURE_DIR)), +) +def test_c_pyi_fixtures_round_trip_through_semantic_ir(fixture: Path): + expected = fixture.read_text(encoding="utf-8").strip() + module = parse_pyi_text( + expected, + module_name=fixture.stem, + filename=str(fixture), + ) + + assert module.name == fixture.stem + assert emit_module(module).strip() == expected diff --git a/tests/pyi/test_pyi_to_ir.py b/tests/pyi/test_pyi_to_ir.py index 6e087b412..eeeed5025 100644 --- a/tests/pyi/test_pyi_to_ir.py +++ b/tests/pyi/test_pyi_to_ir.py @@ -5,13 +5,14 @@ from semantics.models import ( ProjectionMapping, SemanticArgument, + SemanticConstraint, SemanticFunction, SemanticImport, SemanticImportItem, SemanticModule, SemanticType, ) -from semantics.pyi_parser import convert_pyi_to_ir, load_pyi_file, parse_pyi_text +from semantics.pyi_parser import _PyiAstParser, convert_pyi_to_ir, load_pyi_file, parse_pyi_text from semantics.pyi_printer import emit_module from tests._shared.fixture_outputs import FORTRAN_DATA_DIR, FORTRAN_SUFFIXES from x2py import parse_fortran_file @@ -188,17 +189,24 @@ def test_pyi_parser_reports_unsupported_lines_and_invalid_helpers(): parse_pyi_text("x: Unknown\n", module_name="edited") -def test_pyi_parser_ignores_unknown_annotation_metadata(): +def test_pyi_parser_preserves_generic_constraints_as_annotation_metadata(): module = parse_pyi_text( """ -value: Annotated[Int32, Other("native_value")] +value: Annotated[Int32, Bounded(1, 8), Finite] alias: Annotated[Int32, Name("native_alias")] """, module_name="edited", ) assert module.variables[0].name == "value" + assert module.variables[0].semantic_type.constraints == [ + SemanticConstraint("Bounded", [1, 8]), + SemanticConstraint("Finite"), + ] assert module.variables[1].name == "native_alias" + emitted = emit_module(SemanticModule(name="constraints", variables=[module.variables[0]])) + assert "value: Annotated[Int32, Bounded(1, 8), Finite]" in emitted + assert parse_pyi_text(emitted, module_name="constraints").variables[0] == module.variables[0] def test_parse_pyi_text_accepts_qualified_ast_wrapper_names(): @@ -511,7 +519,7 @@ class vector: @pytest.mark.parametrize( "source, message", [ - ("value: Int32[foo.bar]\n", "Unsupported semantic type constraint"), + ("value: Int32[foo.bar]\n", "Non-dimensional type subscriptions are not supported"), ("foo.bar: Int32\n", "Unsupported annotation target"), ("value: Annotated[Int32, Name('x', 'y')]\n", "Name metadata expects one argument"), ("def f(x: Int32): ...\n", "Unsupported function header"), @@ -610,6 +618,94 @@ def consume( assert arrays[1].source_shape == [] +def test_parse_pyi_text_preserves_extended_array_metadata_and_nested_selector(): + module = parse_pyi_text( + """ +value: Annotated[Float64, ORDER_F, Pointer, Contiguous, SourceDims("1:n", "*", "extent"), LowerBounds(None, "0"), UpperBounds("n", None)] +nested: Float64[:, :][rank] + +def fill(x: Annotated[Float64[:], Intent("out")]) -> None: ... +""", + module_name="metadata", + ) + + value = module.variables[0].semantic_type.storage.array + nested = module.variables[1].semantic_type + output = module.functions[0].arguments[0] + assert value.order == "ORDER_F" + assert value.pointer is True + assert value.contiguous is True + assert value.source_shape == ["1:n", "*", "extent"] + assert value.lower_bounds == [None, "0"] + assert value.upper_bounds == ["n", None] + assert nested.metadata["rank_selector"] == "rank" + assert output.intent == "out" + + +def test_parse_pyi_text_handles_callable_and_pointer_storage_variants(): + module = parse_pyi_text( + """ +plain_callback: Callable +opaque_callback: Callable[..., Float64] +constant: Const(Int32) +deep: Ptr[3](Const(Float64)) +rank_any: Float64[...] +strided: Float64[0:n:Strided] +computed: Float64[size(xl)] +""", + module_name="storage", + ) + + plain, callback, constant, deep, rank_any, strided, computed = [var.semantic_type for var in module.variables] + assert plain.name == "Callable" + assert callback.metadata["arguments"] is None + assert constant.storage.kind == "value" + assert constant.storage.read_only is True + assert deep.storage.kind == "pointer" + assert deep.storage.pointer_depth == 3 + assert deep.storage.read_only is True + assert rank_any.storage.array.rank is None + assert strided.storage.array.contiguous is False + assert computed.shape == ["size(xl)"] + + +@pytest.mark.parametrize( + "source, message", + [ + ("value: Const(Int32, Float64)\n", "Const type expects one argument"), + ("value: Ptr(Int32, Float64)\n", "Ptr type expects one argument"), + ("value: Ptr[1](Int32)\n", r"Ptr\[1\]"), + ("value: Callable[Int32]\n", "Callable expects argument types and a return type"), + ("value: Callable[Int32, Float64]\n", "Callable arguments must be a list"), + ("value: Annotated[Float64[:], Intent('out', 'extra')]\n", "Intent metadata expects one argument"), + ("value: Annotated[Float64[:], SourceShape('n')]\n", "SourceShape metadata is not supported"), + ("value: Int32[Constant]\n", "Non-dimensional type subscriptions are not supported"), + ("value: Float64[ORDER_F]\n", "Non-dimensional type subscriptions are not supported"), + ("value: Float64[Shape]\n", "Non-dimensional type subscriptions are not supported"), + ("value: Float64[Shape('n')]\n", "Non-dimensional type subscriptions are not supported"), + ("value: Annotated[Int32, Constant]\n", "use Final"), + ("value: Annotated[Float64[:], Shape('n')]\n", "put dimensions inside"), + ("@native_call([Arg(0).other[0]])\ndef f(x: Int32) -> None: ...\n", "projection entry calls"), + ], +) +def test_parse_pyi_text_rejects_additional_invalid_storage_forms(source: str, message: str): + with pytest.raises(ValueError, match=message): + parse_pyi_text(source, module_name="invalid") + + +def test_pyi_parser_internal_projection_helpers_preserve_native_names(): + parser = _PyiAstParser(module_name="internal") + returned = SemanticArgument("result", SemanticType("Float64"), intent="out", metadata={"return_position": 1}) + mapping = ProjectionMapping(native_name="native_result", result_position=1, intent="out") + _, values = parser._apply_native_call_returns(None, [returned], [mapping]) + native_arg = SemanticArgument("python_name", SemanticType("Int32")) + arg_mapping = ProjectionMapping(native_name="native_name", python_position=0) + parser._apply_native_call_argument_names([native_arg], {}, [arg_mapping]) + + assert values[0].name == "native_result" + assert arg_mapping.native_name == "native_name" + + def test_generated_pyi_compares_equal_to_original_ir_for_all_fortran_fixtures(tmp_path: Path): assert FORTRAN_PYI_COMPARE_FIXTURES diff --git a/tests/semantics/test_c2ir.py b/tests/semantics/test_c2ir.py new file mode 100644 index 000000000..2918d9cd0 --- /dev/null +++ b/tests/semantics/test_c2ir.py @@ -0,0 +1,445 @@ +# -*- coding: utf-8 -*- +"""C parser model to semantic IR conversion tests.""" + +import pytest + +from c_parser import parse_c_file +from c_parser.models import ( + CArray, + CAtomic, + CChar, + CComposedType, + CConst, + CDiagnostic, + CDouble, + CEnum, + CEnumerator, + CFile, + CFloat, + CFunctionType, + CInitializer, + CInt, + CLongDouble, + CMacro, + CMacroDependency, + CParameter, + CPointer, + CProject, + CSourceLocation, + CStruct, + CTypedef, + CUnion, + CUnknownType, + CVariable, + CVolatile, + CVoid, +) +from semantics.c2ir import ( + CToIRConverter, + c_file_to_semantic_module, + c_file_to_semantic_modules, + c_function_to_semantic_function, + c_parameter_to_semantic_argument, + c_project_to_semantic_module, + c_project_to_semantic_modules, + c_struct_to_semantic_class, + c_type_to_semantic_type, +) +from semantics.readiness import assess_semantic_wrap_readiness + + +def _function(module, name): + return next(function for function in module.functions if function.name == name) + + +def test_c2ir_converts_scalar_function_signatures_and_preserves_native_order(): + parsed = parse_c_file("int add(int a, int b);\ndouble scale(double x);\n", filename="api.h") + module = c_file_to_semantic_modules(parsed)[0] + + add = _function(module, "add") + scale = _function(module, "scale") + + assert module.name == "api" + assert [arg.name for arg in add.arguments] == ["a", "b"] + assert [arg.semantic_type.name for arg in add.arguments] == ["Int32", "Int32"] + assert add.return_type.name == "Int32" + assert [mapping.native_position for mapping in add.projection] == [0, 1] + assert scale.return_type.name == "Float64" + assert module.metadata["counts"]["functions"] == 2 + + +def test_c2ir_maps_const_and_mutable_pointers_to_storage_contracts(): + parsed = parse_c_file( + "void copy(const double *src, double *dst);\n", + filename="copy.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + copy = _function(module, "copy") + src, dst = copy.arguments + + assert src.semantic_type.name == "Float64" + assert src.semantic_type.storage.kind == "reference" + assert src.semantic_type.storage.read_only is True + assert src.intent == "in" + + assert dst.semantic_type.name == "Float64" + assert dst.semantic_type.storage.kind == "reference" + assert dst.semantic_type.storage.read_only is False + assert dst.intent == "inout" + assert dst.metadata["readiness_blockers"][0]["code"] == "c_pointer_ownership_ambiguous" + + +def test_c2ir_uses_declared_c_array_bounds_before_parameter_adjustment(): + parsed = parse_c_file( + "void solve(double a[static 4], const int shape[2], int matrix[3][4]);\n", + filename="arrays.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + solve = _function(module, "solve") + a, shape, matrix = solve.arguments + + assert a.semantic_type.storage.kind == "array" + assert a.semantic_type.storage.array.shape == ["4"] + assert a.semantic_type.storage.array.metadata["c_static_minimum"] == [True] + assert a.intent == "inout" + + assert shape.semantic_type.storage.read_only is True + assert shape.semantic_type.storage.array.shape == ["2"] + assert shape.intent == "in" + + assert matrix.semantic_type.rank == 2 + assert matrix.semantic_type.storage.array.shape == ["3", "4"] + assert matrix.semantic_type.storage.array.order == "ORDER_C" + + +def test_c2ir_converts_structs_and_opaque_struct_pointers(): + parsed = parse_c_file( + """ +struct point { double x; double y; }; +struct context; +struct point scale_point(struct point p, double factor); +struct context *context_create(void); +void context_destroy(struct context *ctx); +""", + filename="structs.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + + point = next(cls for cls in module.classes if cls.name == "point") + context = next(cls for cls in module.classes if cls.name == "context") + scale_point = _function(module, "scale_point") + context_create = _function(module, "context_create") + + assert [field.name for field in point.fields] == ["x", "y"] + assert [field.semantic_type.name for field in point.fields] == ["Float64", "Float64"] + assert context.base_classes == ["Opaque"] + assert scale_point.arguments[0].semantic_type.name == "point" + assert context_create.return_type.name == "context" + assert context_create.return_type.storage.kind == "reference" + + report = assess_semantic_wrap_readiness(module, source="structs.h") + assert report["wrappable"] is True + + +def test_c2ir_converts_enum_constants_and_simple_macro_constants(): + parsed = parse_c_file( + """ +#define API_VERSION 3 +enum status { STATUS_OK = 0, STATUS_WARN, STATUS_ERROR = 10 }; +""", + filename="constants.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + + constants = {var.name: var for var in module.variables} + assert constants["API_VERSION"].default_value == "3" + assert constants["API_VERSION"].semantic_type.constraints[0].name == "Constant" + assert constants["STATUS_WARN"].default_value == "1" + assert constants["STATUS_ERROR"].default_value == "10" + + +def test_c2ir_converts_integer_expression_macro_constants_when_resolvable(): + parsed = parse_c_file( + """ +#define API_N0 4 +#define API_N1 (API_N0 + 2) +#define API_MASK (1U << API_N1) +#define API_TEXT "not a semantic integer constant" +void fill(int x[static API_N1]); +""", + filename="shape_macros.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + + constants = {var.name: var for var in module.variables} + assert constants["API_N0"].semantic_type.name == "Int32" + assert constants["API_N1"].semantic_type.name == "Int32" + assert constants["API_MASK"].semantic_type.name == "Int32" + assert "API_TEXT" not in constants + + +def test_c2ir_resolves_local_typedef_chains_and_standard_size_t_fallback(): + parsed = parse_c_file( + """ +typedef unsigned long size_type; +typedef size_type api_size; +api_size count(void); +int read_values(const double *values, size_t n); +""", + filename="typedefs.h", + ) + module = c_file_to_semantic_modules(parsed)[0] + + assert _function(module, "count").return_type.name == "UInt64" + read_values = _function(module, "read_values") + assert [arg.semantic_type.name for arg in read_values.arguments] == ["Float64", "SizeT"] + assert read_values.arguments[0].semantic_type.storage.read_only is True + + +def test_c2ir_uses_standard_type_probe_facts_when_supplied(): + parsed = parse_c_file("size_t count(void);\n", filename="probe.h") + converter = CToIRConverter( + standard_type_report={ + "types": { + "size_t": { + "available": True, + "kind": "integer", + "signed": False, + "bits": 32, + } + } + } + ) + + module = converter.visit_file(parsed) + + assert _function(module, "count").return_type.name == "UInt32" + + +def test_c2ir_uses_standard_type_probe_opaque_handle_facts(): + parsed = parse_c_file("void close_file(FILE *stream);\n", filename="stdio_api.h") + converter = CToIRConverter( + standard_type_report={ + "types": { + "FILE": { + "available": True, + "kind": "opaque_handle", + "pointer_bits": 64, + } + } + } + ) + + module = converter.visit_file(parsed) + close_file = _function(module, "close_file") + + assert [(cls.name, cls.base_classes) for cls in module.classes] == [("FILE", ["Opaque"])] + assert close_file.arguments[0].semantic_type.name == "FILE" + assert close_file.arguments[0].semantic_type.storage.kind == "reference" + assert assess_semantic_wrap_readiness(module, source="stdio_api.h")["wrappable"] is True + + +def test_c_function_compatibility_helper_accepts_parser_function(): + parsed = parse_c_file("float half(float value);\n", filename="helpers.h") + + function = c_function_to_semantic_function(parsed.functions[0]) + + assert function.name == "half" + assert function.arguments[0].semantic_type.name == "Float32" + assert function.return_type.name == "Float32" + + +def test_c2ir_visitor_and_project_compatibility_entrypoints_cover_supported_nodes(): + first = parse_c_file("struct point { int x; };\nint value;\nint f(int x);\n", filename="a.h") + second = parse_c_file("double g(double y);\n", filename="b.h") + project = CProject( + files={"b.h": second, "a.h": first}, + functions={"f": first.functions[0], "g": second.functions[0]}, + structs={"point": first.structs[0]}, + variables={"value": first.variables[0]}, + ) + converter = CToIRConverter() + + assert [module.name for module in converter.visit(project)] == ["a", "b"] + assert converter.visit(first).name == "a" + assert converter.visit(first.functions[0]).name == "f" + assert converter.visit(first.functions[0].parameters[0], position=3).metadata["native_position"] == 3 + assert converter.visit(first.structs[0]).name == "point" + assert converter.visit(first.variables[0]).name == "value" + assert converter.visit(CInt()).name == "Int32" + with pytest.raises(TypeError, match="Unsupported C parse object"): + converter.visit(object()) + + assert c_file_to_semantic_module(first).name == "a" + assert c_type_to_semantic_type(CInt()).name == "Int32" + assert c_parameter_to_semantic_argument(CParameter(name=None, type=CInt()), position=2).name == "arg2" + assert c_struct_to_semantic_class(first.structs[0]).name == "point" + assert [module.name for module in c_project_to_semantic_modules(project)] == ["a", "b"] + merged = c_project_to_semantic_module(project, name="42 api/project") + assert merged.name == "_42_api_project" + assert {function.name for function in merged.functions} == {"f", "g"} + + +def test_c2ir_converts_qualifiers_callbacks_bitfields_and_unspecified_functions(): + callback = CComposedType( + components=[ + CPointer(), + CFunctionType(result_type=CVoid(), parameter_types=[CInt()]), + ], + source_text="void (*)(int)", + ) + converter = CToIRConverter() + variable = converter.visit_variable(CVariable(name="handler", type=callback, storage=["static"])) + field = converter.visit_variable(CVariable(name="bits", type=CInt(), bit_width="3")) + function = converter.visit_function( + parse_c_file("static int legacy();\n", filename="legacy.h").functions[0] + ) + qualified = converter.visit_type( + CChar(qualifiers=[CConst(), CVolatile(), CAtomic()], source_text="const volatile _Atomic char") + ) + + assert variable.visibility == "private" + assert variable.semantic_type.name == "CFunctionPointer" + assert field.semantic_type.metadata["readiness_blockers"][0]["code"] == "c_bitfield_unsupported" + assert function.visibility == "private" + assert function.metadata["readiness_blockers"][0]["code"] == "c_unspecified_function_parameters" + assert qualified.name == "Int8" + assert qualified.metadata["c_char_policy"] + assert {blocker["code"] for blocker in qualified.metadata["readiness_blockers"]} == { + "c_volatile_unsupported", + "c_atomic_unsupported", + } + + +def test_c2ir_reports_unsupported_type_and_declarator_compositions(): + converter = CToIRConverter(primitive_type_map={CInt: None}) + + unresolved = converter.visit_type(CUnknownType(spelling="missing_t", source_text="missing_t")) + unsupported_integer = converter.visit_type(CInt(source_text="int")) + unsupported_precision = converter.visit_type(CLongDouble(source_text="long double")) + empty = converter.visit_type(CComposedType(components=[])) + array_missing_element = converter.visit_type(CComposedType(components=[CArray(bound="4")])) + array_of_pointer = converter.visit_type( + CComposedType(components=[CArray(bound="4"), CPointer(), CDouble()]) + ) + pointer_missing_pointee = converter.visit_type(CComposedType(components=[CPointer()])) + pointer_composition = converter.visit_type( + CComposedType(components=[CPointer(), CInt(), CDouble()]) + ) + other_composition = converter.visit_type(CComposedType(components=[CInt(), CDouble()])) + + assert unresolved.metadata["readiness_blockers"][0]["code"] == "c_unresolved_type" + assert unsupported_integer.metadata["readiness_blockers"][0]["code"] == "c_unsupported_type" + assert unsupported_precision.metadata["readiness_blockers"][0]["code"] == "c_long_double_unsupported" + assert empty.metadata["readiness_blockers"][0]["code"] == "c_empty_composed_type" + assert array_missing_element.metadata["readiness_blockers"][0]["code"] == "c_array_missing_element_type" + assert array_of_pointer.metadata["readiness_blockers"][0]["code"] == "c_array_of_pointer_unsupported" + assert pointer_missing_pointee.metadata["readiness_blockers"][0]["code"] == "c_pointer_missing_pointee" + assert pointer_composition.metadata["readiness_blockers"][0]["code"] == "c_unsupported_composed_type" + assert other_composition.metadata["readiness_blockers"][0]["code"] == "c_unsupported_composed_type" + + +def test_c2ir_models_pointer_to_arrays_unknown_extents_unions_and_anonymous_aliases(): + converter = CToIRConverter() + pointer_array = converter.visit_type( + CComposedType(components=[CPointer(), CArray(bound=None), CDouble()], source_text="double (*)[]"), + owner="matrix", + ) + choice = CUnion(name="choice", members=[CVariable(name="integer", type=CInt())]) + converter.unions = {"choice": choice} + union_type = converter.visit_type(CUnion(name="choice"), owner="selected") + anon_struct = CStruct(anonymous_id="anon_struct_1") + anon_union = CUnion(anonymous_id="anon_union_1") + converter.typedefs = { + "record_t": CTypedef(name="record_t", type=anon_struct), + "variant_t": CTypedef(name="variant_t", type=anon_union), + } + + assert pointer_array.storage.pointer_depth == 1 + assert pointer_array.storage.metadata["c_pointer_to_array"] is True + assert pointer_array.metadata["readiness_blockers"][0]["code"] == "c_array_extent_ambiguous" + assert union_type.metadata["readiness_blockers"][0]["code"] == "c_union_unsupported" + assert converter.visit_struct(anon_struct).name == "record_t" + assert converter.visit_union(anon_union).name == "variant_t" + + +def test_c2ir_marks_incomplete_by_value_structs_and_preserves_initializer_locations(): + incomplete = CStruct(name="handle", is_incomplete=True) + converter = CToIRConverter() + converter.structs = {"handle": incomplete} + parameter = converter.visit_parameter(CParameter(name="handle", type=incomplete), owner="open") + variable = converter.visit_variable( + CVariable( + name=None, + type=incomplete, + initializer=CInitializer("factory()"), + source_location=CSourceLocation(filename="api.h", line=2, column=4, source_line="struct handle h;"), + ) + ) + + assert parameter.semantic_type.metadata["readiness_blockers"][0]["code"] == "c_incomplete_struct_by_value" + assert variable.name == "" + assert variable.default_value == "factory()" + assert variable.origin.source_location["filename"] == "api.h" + + +def test_c2ir_standard_type_facts_and_numeric_constant_edge_cases(): + class Report: + types = { + "signed_size": {"kind": "integer", "signed": True, "bits": 16}, + "real_size": {"kind": "real", "bits": 32}, + "missing": {"available": False, "kind": "integer", "signed": False, "bits": 32}, + } + + converter = CToIRConverter(standard_type_report=Report()) + assert converter._standard_semantic_type("signed_size").name == "Int16" + assert converter._standard_semantic_type("real_size").name == "Float32" + assert converter._standard_semantic_type("missing") is None + assert converter._standard_semantic_type("not_standard") is None + assert CToIRConverter._standard_type_facts(object()) == {} + assert CToIRConverter._integer_literal_value(None) is None + assert CToIRConverter._integer_literal_value("value") is None + assert CToIRConverter._integer_macro_expression("(MISSING + 1)", {}) is False + assert CToIRConverter._integer_macro_expression("(1 +)", {}) is False + + parsed = parse_c_file( + "#define RATE 1.5\n#define BAD (MISSING + 1)\nenum status { STATUS_EXPR = UNKNOWN, STATUS_NEXT };\n", + filename="edge_constants.h", + ) + constants = {variable.name: variable for variable in converter.visit_file(parsed).variables} + assert constants["RATE"].semantic_type.name == "Float64" + assert constants["STATUS_EXPR"].default_value == "UNKNOWN" + assert constants["STATUS_NEXT"].default_value is None + + +def test_c2ir_propagates_file_and_project_diagnostic_blockers(): + dependency = CMacroDependency(name="API", source_text="API(int) f(void);") + warning = CDiagnostic(code="C_WARNING", message="warning", severity="warning") + duplicate_dependency = CDiagnostic( + code="C_MACRO_DEPENDENT_DECLARATION", + message="already represented", + severity="error", + ) + error = CDiagnostic( + code="C_BAD_DECL", + message="bad declaration", + severity="error", + unit_kind="function", + unit_name="broken", + ) + c_file = CFile( + filename=None, + macro_dependencies=[dependency], + diagnostics=[warning, duplicate_dependency, error], + ) + project = CProject(files={"bad.h": c_file}, diagnostics=[error]) + converter = CToIRConverter() + + module = converter.visit_file(c_file) + merged = converter.visit_project_module(project) + file_codes = {blocker["code"] for blocker in module.metadata["readiness_blockers"]} + project_codes = {blocker["code"] for blocker in merged.metadata["readiness_blockers"]} + + assert module.name == "c_module" + assert file_codes == {"c_macro_dependent_declaration", "c_c_bad_decl"} + assert project_codes == {"c_macro_dependent_declaration", "c_c_bad_decl"} diff --git a/tests/semantics/test_c_semantic_readiness.py b/tests/semantics/test_c_semantic_readiness.py index 5cabf6716..b302eb1f7 100644 --- a/tests/semantics/test_c_semantic_readiness.py +++ b/tests/semantics/test_c_semantic_readiness.py @@ -1,16 +1,10 @@ # -*- coding: utf-8 -*- -"""Planned C semantic wrap-readiness tests. +"""C semantic wrap-readiness tests. These tests intentionally live under ``tests/semantics`` because readiness is owned by semantic IR or edited ``.pyi`` interfaces, not by the C parser. """ -import pytest - -pytestmark = pytest.mark.skip( - reason="C semantic readiness roadmap tests; unskip after C semantic IR conversion exists." -) - def test_c_semantic_readiness_accepts_plain_primitive_function_signatures(): from c_parser import parse_c_file @@ -101,9 +95,9 @@ def test_completed_pyi_callback_policy_can_make_c_api_semantically_ready(): from typing import Any, Callable def each_item( - items: Pointer[Any], - visit: Callable[[Pointer[Any], Pointer[Any]], None], - userdata: Pointer[Any], + items: Ptr(Any), + visit: Callable[[Ptr(Any), Ptr(Any)], None], + userdata: Ptr(Any), ) -> None: ... """, module_name="callback_api", diff --git a/tests/semantics/test_pyi_printer.py b/tests/semantics/test_pyi_printer.py index 1eae6ca50..1eeb0beb8 100644 --- a/tests/semantics/test_pyi_printer.py +++ b/tests/semantics/test_pyi_printer.py @@ -651,7 +651,7 @@ def test_printer_class_entrypoint(): def test_printer_emit_visitor_dispatches_semantic_models(): printer = PyiPrinter() - constraint = SemanticConstraint("Constant") + constraint = SemanticConstraint("Finite") semantic_type = SemanticType( "Float64", dtype="Float64", @@ -673,7 +673,7 @@ def test_printer_emit_visitor_dispatches_semantic_models(): func = SemanticFunction(name="wrap", arguments=[argument]) module = SemanticModule(name="visitor_mod", classes=[cls], functions=[func]) - assert printer.emit(constraint) == "Constant" + assert printer.emit(constraint) == "Finite" assert printer.emit(semantic_type) == "Float64[:]" assert printer.emit(argument) == 'class_: Annotated[Float64[:], Name("class")] = ...' assert "def reset(self) -> None: ..." in printer.emit(method) @@ -747,7 +747,6 @@ def test_emit_module_variables_with_visibility(): """ code = generate_pyi(source) assert "answer: private[Final[Int32]]" in code - assert "answer: private[Int32[Constant]]" not in code assert "counter: Int32" in code assert "hidden_scale: private[Float64]" in code @@ -859,3 +858,84 @@ def test_emit_native_call_rejects_unrepresentable_projection_entries(projection, with pytest.raises(ValueError, match=message): emit_module(module) + + +def test_printer_emits_extended_storage_and_callable_forms(): + printer = PyiPrinter() + readonly_value = SemanticType( + "Int32", + storage=SemanticStorageContract(kind="value", read_only=True), + ) + mutable_value = SemanticType("Int32", storage=SemanticStorageContract(kind="value")) + deep_pointer = SemanticType( + "Float64", + storage=SemanticStorageContract(kind="pointer", read_only=True, pointer_depth=3), + ) + unspecified_storage = SemanticType("Int32", storage=SemanticStorageContract(kind="custom")) + inferred_array = SemanticType( + "Float64", + rank=2, + storage=SemanticStorageContract(kind="array"), + ) + annotated_array = SemanticType( + "Float64", + constraints=[SemanticConstraint("Finite"), SemanticConstraint("Range", [1, 3])], + storage=SemanticStorageContract( + kind="array", + array=SemanticArrayContract( + rank=2, + shape=[":", ":"], + order="ORDER_ANY", + allocatable=True, + pointer=True, + ), + ), + ) + full_callback = SemanticType( + "Callable", + metadata={ + "arguments": [SemanticType("Int32")], + "return": SemanticType("Float64"), + }, + ) + any_callback = SemanticType("Callable", metadata={"return": SemanticType("Float64")}) + + canonical_constant = SemanticArgument( + "answer", + SemanticType("Int32", constraints=[SemanticConstraint("Constant")]), + ) + assert printer.emit_argument(canonical_constant) == "answer: Final[Int32]" + with pytest.raises(ValueError, match=r"Final\[\.\.\.\]"): + printer.emit_semantic_type(canonical_constant.semantic_type) + assert printer.emit_semantic_type(readonly_value) == "Const(Int32)" + assert printer.emit_semantic_type(mutable_value) == "Int32" + assert printer.emit_semantic_type(deep_pointer) == "Ptr[3](Const(Float64))" + assert printer.emit_semantic_type(unspecified_storage) == "Int32" + assert printer.emit_semantic_type(inferred_array) == "Float64[:, :]" + assert printer.emit_semantic_type(annotated_array) == ( + "Annotated[Float64[:, :], ORDER_ANY, Allocatable, Pointer, Finite, Range(1, 3)]" + ) + assert printer.emit_semantic_type(full_callback) == "Callable[[Int32], Float64]" + assert printer.emit_semantic_type(any_callback) == "Callable[..., Float64]" + assert printer.emit_semantic_type(SemanticType("Callable")) == "Callable" + + +def test_printer_projection_return_helpers_and_keyword_data_members(): + printer = PyiPrinter() + argument = SemanticArgument("x", SemanticType("Float64"), intent="inout", optional=True) + plain = SemanticArgument("value", SemanticType("Int32")) + module = SemanticModule( + name="returns", + variables=[SemanticArgument("class", SemanticType("Int32"))], + functions=[ + SemanticFunction( + name="created", + projection=[ProjectionMapping(native_position=0, result_position=0)], + ) + ], + ) + + assert printer._projected_argument_return(argument) == 'Returns["x", Float64, Optional]' + assert printer._projected_argument_return(plain) == "Int32" + assert "var['class']: Int32" in emit_module(module) + assert "@native_call([Return(0)])" in emit_module(module) diff --git a/tests/semantics/test_semantic_wrap_readiness.py b/tests/semantics/test_semantic_wrap_readiness.py index 57fd56437..dadf852c2 100644 --- a/tests/semantics/test_semantic_wrap_readiness.py +++ b/tests/semantics/test_semantic_wrap_readiness.py @@ -6,7 +6,19 @@ import pytest from semantics.pyi_parser import parse_pyi_text -from semantics.readiness import assess_semantic_wrap_readiness +from semantics.models import ( + SemanticArgument, + SemanticClass, + SemanticFunction, + SemanticMethod, + SemanticModule, + SemanticType, +) +from semantics.readiness import ( + _iter_expression_values, + assess_pyi_wrap_readiness, + assess_semantic_wrap_readiness, +) from x2py import cli as x2py_cli @@ -161,6 +173,83 @@ def integrate(objective: Callable[..., Float64], x0: Float64) -> Float64: ... assert "callback_signature_incomplete" in _blocker_codes(report) +def test_assess_pyi_wrap_readiness_expands_directory_and_deduplicates_paths(tmp_path: Path): + nested = tmp_path / "nested" + nested.mkdir() + first = tmp_path / "first.pyi" + second = nested / "second.pyi" + ignored = nested / "ignored.txt" + first.write_text("def first(x: Int32) -> None: ...\n", encoding="utf-8") + second.write_text("def second(x: Int32) -> None: ...\n", encoding="utf-8") + ignored.write_text("not a stub", encoding="utf-8") + + report = assess_pyi_wrap_readiness([tmp_path, first]) + + assert report["wrappable"] is True + assert report["n_modules"] == 2 + assert report["source"] == [str(first), str(second)] + + +def test_readiness_skips_private_api_and_normalizes_metadata_blocker_items(): + module = SemanticModule( + name="policy", + metadata={ + "readiness_blockers": [ + "ignored", + {"code": "default_item", "message": "default", "item": {"detail": "fallback"}}, + { + "code": "scalar_item", + "message": "scalar", + "items": "detail text", + "unit": "policy.override", + "unit_kind": "policy", + }, + ] + }, + variables=[SemanticArgument("hidden", SemanticType("Undeclared"), visibility="private")], + classes=[ + SemanticClass(name="Private", fields=[SemanticArgument("missing", SemanticType("Undeclared"))], visibility="private"), + SemanticClass( + name="Public", + methods=[ + SemanticMethod(name="hidden", arguments=[SemanticArgument("missing", SemanticType("Undeclared"))], visibility="private"), + SemanticMethod(name="ready", arguments=[SemanticArgument("value", SemanticType("Int32"))]), + ], + ), + ], + functions=[SemanticFunction(name="hidden", arguments=[SemanticArgument("missing", SemanticType("Undeclared"))], visibility="private")], + ) + + report = assess_semantic_wrap_readiness(module) + blockers = {blocker["code"]: blocker for blocker in report["wrappability_blockers"]} + + assert set(blockers) == {"default_item", "scalar_item"} + assert blockers["default_item"]["items"][0]["detail"] == "fallback" + assert blockers["scalar_item"]["items"][0]["detail"] == "detail text" + assert {unit["unit"] for unit in report["unit_blockers"]} == {"policy", "policy.override"} + + +def test_readiness_accepts_qualified_types_from_imported_modules_and_aliases(): + report = _readiness_from_pyi( + """ +import state_mod +import mesh_mod as mesh +from values_mod import value_t as imported_value + +def step(a: state_mod.state_t, b: mesh.mesh_t, c: imported_value) -> None: ... +""" + ) + + assert report["wrappable"] is True + + +def test_readiness_empty_public_surface_and_nested_expression_utility(): + report = assess_semantic_wrap_readiness(SemanticModule(name="empty")) + + assert _blocker_codes(report) == {"no_public_api"} + assert list(_iter_expression_values({"a": ["n", ("m", {"deep": "k"})]})) == ["n", "m", "k"] + + def test_cli_wrap_readiness_loads_completed_pyi(tmp_path: Path): pyi = tmp_path / "solver.pyi" pyi.write_text( diff --git a/x2py/__init__.py b/x2py/__init__.py index e0fbbce9e..ee071052e 100644 --- a/x2py/__init__.py +++ b/x2py/__init__.py @@ -24,6 +24,17 @@ fortran_module_to_semantic_module, resolve_semantic_compile_time_values, ) +from semantics.c2ir import ( + CToIRConverter, + c_file_to_semantic_module, + c_file_to_semantic_modules, + c_function_to_semantic_function, + c_parameter_to_semantic_argument, + c_project_to_semantic_module, + c_project_to_semantic_modules, + c_struct_to_semantic_class, + c_type_to_semantic_type, +) from semantics.pyi_parser import convert_pyi_to_ir, load_pyi_file, parse_pyi_text from semantics.readiness import assess_pyi_wrap_readiness, assess_semantic_wrap_readiness @@ -46,6 +57,7 @@ def __getattr__(name: str): raise AttributeError(f"module 'x2py' has no attribute {name!r}") __all__ = ( + "CToIRConverter", "CFile", "CParseError", "CProject", @@ -65,6 +77,14 @@ def __getattr__(name: str): "assess_pyi_wrap_readiness", "assess_semantic_wrap_readiness", "build_fortran_type_probe_source", + "c_file_to_semantic_module", + "c_file_to_semantic_modules", + "c_function_to_semantic_function", + "c_parameter_to_semantic_argument", + "c_project_to_semantic_module", + "c_project_to_semantic_modules", + "c_struct_to_semantic_class", + "c_type_to_semantic_type", "collect_semantic_compile_time_requirements", "convert_pyi_to_ir", "evaluate_fortran_type_requirements", diff --git a/x2py/cli.py b/x2py/cli.py index 0da4f0401..3539a7768 100644 --- a/x2py/cli.py +++ b/x2py/cli.py @@ -7,11 +7,13 @@ from dataclasses import asdict, fields, is_dataclass from pathlib import Path -from c_parser.cli import format_c_report, parse_c_report +from c_parser.cli import expand_c_paths, format_c_report, parse_c_report from c_parser.models import CParseError +from c_parser.parser import CParser from fortran_parser.models import FortranParseError from fortran_parser.parser import FortranParser from fortran_parser.cli import _format_report +from semantics.c2ir import c_file_to_semantic_modules from semantics.fortran2ir import fortran_file_to_semantic_modules from semantics.pyi_parser import load_pyi_file from semantics.readiness import assess_semantic_wrap_readiness @@ -124,6 +126,31 @@ def _c_parser_preprocessing_mode(preprocessing: PreprocessingConfig) -> str: return "compiler" if preprocessing.uses_compiler else "raw" +def _parse_c_path( + parser: CParser, + path: Path, + preprocessing: PreprocessingConfig, +): + source_loader = _c_source_loader(preprocessing) + if source_loader is None: + return parser.visit_file( + path, + filename=str(path), + include_dirs=preprocessing.include_dirs, + preprocessing=_c_parser_preprocessing_mode(preprocessing), + ) + + source, preprocessing_recipe = source_loader(path) + parsed = parser.visit_file( + source, + filename=str(path), + include_dirs=preprocessing.include_dirs, + preprocessing=_c_parser_preprocessing_mode(preprocessing), + ) + parsed.preprocessing_recipe = preprocessing_recipe + return parsed + + def _parse_report(paths: list[str], preprocessing: PreprocessingConfig | None = None) -> dict[str, dict]: preprocessing = preprocessing or PreprocessingConfig() out: dict[str, dict] = {} @@ -145,12 +172,28 @@ def _parse_report(paths: list[str], preprocessing: PreprocessingConfig | None = return out -def _semantic_report(paths: list[str], preprocessing: PreprocessingConfig | None = None) -> dict[str, dict]: +def _semantic_report( + paths: list[str], + preprocessing: PreprocessingConfig | None = None, + *, + language: str = "fortran", +) -> dict[str, dict]: from semantics.fortran2ir import fortran_module_to_semantic_module from semantics.pyi_printer import emit_module preprocessing = preprocessing or PreprocessingConfig() out: dict[str, dict] = {} + if language == "c": + parser = CParser() + for p in expand_c_paths(paths): + parsed = _parse_c_path(parser, p, preprocessing) + modules = c_file_to_semantic_modules(parsed) + out[str(p)] = { + "semantic_modules": [asdict(module) for module in modules], + "pyi": "\n\n".join(emit_module(module) for module in modules).strip(), + } + return out + parser = FortranParser() for p in _expand_paths(paths): code, macro_defines, _preprocessing_recipe = _fortran_source_for_path(p, preprocessing) @@ -176,9 +219,26 @@ def _format_pyi_report(semantic_report: dict[str, dict]) -> str: return "\n".join(lines).rstrip() -def _wrap_readiness_report(paths: list[str], preprocessing: PreprocessingConfig | None = None) -> dict[str, dict]: +def _wrap_readiness_report( + paths: list[str], + preprocessing: PreprocessingConfig | None = None, + *, + language: str = "fortran", +) -> dict[str, dict]: preprocessing = preprocessing or PreprocessingConfig() out: dict[str, dict] = {} + if language == "c": + parser = CParser() + for p in expand_c_paths(paths): + parsed = _parse_c_path(parser, p, preprocessing) + modules = c_file_to_semantic_modules(parsed) + out[str(p)] = { + "source_kind": "c", + "semantic_modules": [asdict(module) for module in modules], + "wrap_readiness": assess_semantic_wrap_readiness(modules, source=str(p)), + } + return out + parser = FortranParser() for p in _expand_readiness_paths(paths): if p.suffix.lower() == ".pyi": @@ -240,6 +300,10 @@ def _format_semantic_blocker_item(code: str, item) -> str: if code == "callback_signature_incomplete": needs = ", ".join(item.get("needs") or []) return f"{item['owner']} needs Callable[[...], ...] metadata ({needs})" + if code.startswith("c_"): + owner = item.get("owner", "") + detail = item.get("type") or item.get("source") or item.get("function") or item.get("parameter") + return f"{owner}: {detail}" if detail else str(item) if code == "no_public_api": needs = ", ".join(item.get("needs") or []) return f"{item['owner']} needs {needs}" @@ -409,7 +473,7 @@ def main() -> int: "--language", choices=("fortran", "c"), default="fortran", - help="Frontend language. Defaults to fortran; C currently supports partial --parse output.", + help="Frontend language. Defaults to fortran; C supports parse, semantic IR, and semantic readiness output.", ) parser.add_argument("--parse", action="store_true", help="Run and output parser stage report") parser.add_argument( @@ -490,10 +554,10 @@ def main() -> int: parser.add_argument( "--wrap-readiness", action="store_true", - help="Convert Fortran or .pyi input to semantic IR and show wrapper readiness", + help="Convert Fortran, C, or .pyi input to semantic IR and show wrapper readiness", ) - parser.add_argument("--semantics", action="store_true", help="Generate semantic IR models from parsed Fortran modules") - parser.add_argument("--pyi", action="store_true", help="Generate Python .pyi content") + parser.add_argument("--semantics", action="store_true", help="Generate semantic IR models from parsed source modules") + parser.add_argument("--pyi", action="store_true", help="Generate semantic Python .pyi content") parser.add_argument("--json", action="store_true", help="Print JSON to stdout") parser.add_argument("--out", nargs="?", const="", type=str, help="Write stage output to file (optional explicit output filename)") parser.add_argument("--no-color", action="store_true", help="Disable ANSI color in parse diagnostics") @@ -503,13 +567,7 @@ def main() -> int: if args.language == "c": if not (args.parse or args.semantics or args.pyi or args.wrap_readiness): - parser.error("--language c requires --parse; C semantics and .pyi output are not supported yet") - if args.semantics: - parser.error("--semantics is not supported for --language c yet") - if args.pyi: - parser.error("--pyi is not supported for --language c yet") - if args.wrap_readiness: - parser.error("--wrap-readiness is semantic-layer output and is not supported for --language c yet") + parser.error("--language c requires a stage flag: choose one of --parse, --semantics, --pyi, or --wrap-readiness") if args.show_vars or args.print_limit is not None or args.vars_limit is not None: parser.error("--show-vars/--print-limit are Fortran-only and are not supported for --language c") @@ -537,8 +595,8 @@ def main() -> int: if args.parse and args.language == "c" else _parse_report(args.paths, preprocessing) if args.parse else None ) - semantic_payload = _semantic_report(args.paths, preprocessing) if (args.semantics or args.pyi) else None - readiness_payload = _wrap_readiness_report(args.paths, preprocessing) if args.wrap_readiness else None + semantic_payload = _semantic_report(args.paths, preprocessing, language=args.language) if (args.semantics or args.pyi) else None + readiness_payload = _wrap_readiness_report(args.paths, preprocessing, language=args.language) if args.wrap_readiness else None _attach_wrap_readiness(semantic_payload, readiness_payload) except CParseError as exc: if args.debug_traceback or _env_flag("C_PARSER_DEBUG"): diff --git a/x2py/preprocessing.py b/x2py/preprocessing.py index 2b6120baa..da398003a 100644 --- a/x2py/preprocessing.py +++ b/x2py/preprocessing.py @@ -189,9 +189,10 @@ def _compile_command_arguments(entry: dict) -> list[str]: def _entry_file_path(entry: dict) -> Path: """Return the absolute source path for a compile database entry.""" directory = Path(str(entry.get("directory") or ".")) - file_path = Path(str(entry.get("file") or "")) - if not file_path: + file_value = entry.get("file") + if not file_value: raise PreprocessingError("compile_commands entry is missing 'file'") + file_path = Path(str(file_value)) return file_path if file_path.is_absolute() else directory / file_path