diff --git a/README.md b/README.md index b8183cbca..c4193e55c 100644 --- a/README.md +++ b/README.md @@ -199,18 +199,18 @@ python -m x2py tests/data/c/general/math_api.h --language c --pyi ```python File: tests/data/c/general/math_api.h def norm2( - n: Int32, + n: Int, x: Const(Float64[1]) ) -> Float64: ... def scale( - n: Int32, + n: Int, alpha: Float64, x: Float64[1] ) -> None: ... def dot( - n: Int32, + n: Int, x: Ptr(Const(Float64)), y: Ptr(Const(Float64)) ) -> Float64: ... diff --git a/docs/c_parser.md b/docs/c_parser.md index 872e64e37..258e756a2 100644 --- a/docs/c_parser.md +++ b/docs/c_parser.md @@ -281,9 +281,11 @@ python -m x2py.c_type_probe --compiler /usr/bin/gcc-13 --std c11 ``` The report records arithmetic category, underlying C spelling, bit width, and -alignment for `size_t`, available `uint32_t`, and `time_t`; it records opaque -handle and pointer ABI facts for `FILE`. It also retains the generated C source -and exact compile/run commands. +alignment for builtin C `int`, `size_t`, available `uint32_t`, and `time_t`; it +records opaque handle and pointer ABI facts for `FILE`. It also retains the +generated C source and exact compile/run commands. Semantic conversion keeps +the name `Int` for builtin C `int` and stores the measured concrete dtype and +probe fact separately. The probe must be run with the same target profile as the source being parsed. It carries `-I`, `-D`, `-U`, and `--compiler-arg` options into the compile diff --git a/docs/developper_guide.md b/docs/developper_guide.md index c161ad1dd..11b03b75b 100644 --- a/docs/developper_guide.md +++ b/docs/developper_guide.md @@ -349,6 +349,9 @@ from `semantics/models.py`. - `semantics/c2ir.py` maps C functions, variables, structs/opaque structs, enums, typedef chains, standard-type probe facts, macros, pointer/array storage, and C-specific readiness blockers. +- C `int` keeps the semantic name `Int` while its compiler-probed concrete + precision is stored on the semantic type. C enums are open named semantic + declarations with unscoped module-level enumerator constants. - `semantics/pyi_printer.py` emits editable user contracts. - `semantics/pyi_parser.py` loads edited contracts back into semantic IR. - `semantics/readiness.py` decides whether that IR is complete enough for diff --git a/docs/examples.md b/docs/examples.md index 61d9c5ba9..bc191fc21 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -435,9 +435,9 @@ Output: ```text def add( - a: Int32, - b: Int32 -) -> Int32: ... + a: Int, + b: Int +) -> Int: ... True ``` diff --git a/docs/semantics.md b/docs/semantics.md index 5e59b6c9f..1579c5f54 100644 --- a/docs/semantics.md +++ b/docs/semantics.md @@ -21,6 +21,7 @@ and eventual NumPy-oriented wrapper code. | Semantic dtype | NumPy equivalent | Notes | | --- | --- | --- | | `Bool` | `numpy.bool_` | Boolean scalar. | +| `Int` | Target-dependent signed NumPy integer | Ordinary C `int`; the concrete `Int16`/`Int32`/`Int64` dtype and compiler fact are stored separately. | | `Int8`, `Int16`, `Int32`, `Int64` | `numpy.int8`, `numpy.int16`, `numpy.int32`, `numpy.int64` | Signed integers. | | `UInt8`, `UInt16`, `UInt32`, `UInt64` | `numpy.uint8`, `numpy.uint16`, `numpy.uint32`, `numpy.uint64` | Unsigned integers. | | `Float32`, `Float64` | `numpy.float32`, `numpy.float64` | Binary floating-point scalars. | @@ -57,7 +58,8 @@ and eventual NumPy-oriented wrapper code. | `char`, `signed char` | `Int8` | `numpy.int8` | | `unsigned char` | `UInt8` | `numpy.uint8` | | `short`, `unsigned short` | `Int16`, `UInt16` | `numpy.int16`, `numpy.uint16` | -| `int`, `unsigned int` | `Int32`, `UInt32` | `numpy.int32`, `numpy.uint32` | +| `int` / `CInt` | `Int` with concrete probed dtype | Matching signed NumPy integer for the target | +| `unsigned int` | `UInt32` | `numpy.uint32` | | `long`, `long long` | `Int64` | `numpy.int64` | | `unsigned long`, `unsigned long long` | `UInt64` | `numpy.uint64` | | `float`, `double`, `long double` | `Float32`, `Float64`, `Float128` | `numpy.float32`, `numpy.float64`, `numpy.longdouble` | @@ -66,9 +68,12 @@ and eventual NumPy-oriented wrapper code. | `uint8_t`, `uint16_t`, `uint32_t`, `uint64_t` | `UInt8`, `UInt16`, `UInt32`, `UInt64` | Matching unsigned NumPy integer | | `size_t` | `SizeT` or probed unsigned width | `numpy.uintp` or matching `numpy.uint*` | -C integer spellings such as `long` are ABI-dependent in general C, but the -current semantic policy maps parsed primitive `long` and `long long` to 64-bit -semantic dtypes. Standard-library typedefs are refined through the compiler +C integer spellings are ABI-dependent. Ordinary C `int` keeps the stable +semantic identity `Int`; its concrete dtype and the compiler fact used to +derive it are stored on `SemanticType`. Without a supplied compiler report, +the concrete dtype uses a clearly marked 32-bit fallback. The current semantic +policy still maps parsed primitive `long` and `long long` to 64-bit semantic +dtypes. Standard-library typedefs are refined through the compiler standard-type probe when facts are supplied. ## C To Semantic IR Mapping @@ -87,9 +92,11 @@ policy is documented in the datatype mapping section above. - `_Bool` -> `Bool`. - `char` -> `Int8` with `c_char_policy` metadata; `signed char` -> `Int8`; `unsigned char` -> `UInt8`. -- `short`, `int`, `long`, and `long long` map to fixed signed integer names - using the current Linux-oriented defaults: `Int16`, `Int32`, `Int64`, - `Int64`. +- `int` maps to `Int`. Its concrete dtype is derived from a supplied + `x2py.c_type_probe` report and stored with the fact source; without one it + carries a marked `Int32` fallback. +- `short`, `long`, and `long long` map to fixed signed integer names using the + current Linux-oriented defaults: `Int16`, `Int64`, `Int64`. - Unsigned integer spellings map to `UInt16`, `UInt32`, `UInt64`, and `UInt64`; fixed-width typedef spellings such as `uint32_t` map to the matching `UInt*` fallback. @@ -103,8 +110,20 @@ policy is documented in the datatype mapping section above. `Int*`, `UInt*`, or `Float*` semantic names. - Opaque standard-type probe facts such as `FILE` create named opaque semantic classes when referenced by converted declarations. -- Object-like numeric macros and enum constants become `Final`-style semantic - variables through the `Constant` constraint. +- Enum definitions become open `SemanticEnum` declarations. Named enum + arguments and returns keep the enum datatype instead of flattening to an + integer. +- C enumerators remain unscoped module-level `Final[enum_name]` variables with + their known values. An open enum may still carry any value representable by + its underlying integer type; the listed enumerators are named constants, not + closed validation choices. +- Native enumerator expressions remain stored in semantic IR. The `.pyi` + initializer is emitted only when it can be represented as valid Python + expression syntax. +- Enum underlying storage currently assumes C `int` and records that + assumption unless an enum-specific compiler fact is supplied. +- Object-like numeric macros become `Final`-style semantic variables through + the `Constant` constraint. - Struct definitions become `SemanticClass` entries. Incomplete structs become opaque classes and may be used through direct `Ptr(...)` identity contracts. - Explicit multi-header conversion resolves a struct to the header that defines @@ -118,6 +137,25 @@ policy is documented in the datatype mapping section above. metadata. `const` on the pointee makes the storage read-only, and `restrict` is preserved as aliasing metadata. +For example: + +```c +enum status { STATUS_OK = 0, STATUS_ERROR = 10 }; +void set_status(enum status value); +``` + +becomes: + +```python +class status(Enum[Int]): + pass + +STATUS_OK: Final[status] = 0 +STATUS_ERROR: Final[status] = 10 + +def set_status(value: status) -> None: ... +``` + ### Conservative Blockers The converter does not silently invent wrapper policy. It attaches diff --git a/docs/tutorial.md b/docs/tutorial.md index 084d2096d..14ff0b458 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -211,18 +211,18 @@ Expected output: ```python File: tests/data/c/general/math_api.h def norm2( - n: Int32, + n: Int, x: Const(Float64[1]) ) -> Float64: ... def scale( - n: Int32, + n: Int, alpha: Float64, x: Float64[1] ) -> None: ... def dot( - n: Int32, + n: Int, x: Ptr(Const(Float64)), y: Ptr(Const(Float64)) ) -> Float64: ... diff --git a/semantics/__init__.py b/semantics/__init__.py index 06f652ae7..a1c46051f 100644 --- a/semantics/__init__.py +++ b/semantics/__init__.py @@ -7,6 +7,7 @@ ) from .c2ir import ( CToIRConverter, + c_enum_to_semantic_enum, c_file_to_semantic_module, c_file_to_semantic_modules, c_function_to_semantic_function, @@ -24,6 +25,7 @@ "CToIRConverter", "assess_pyi_wrap_readiness", "assess_semantic_wrap_readiness", + "c_enum_to_semantic_enum", "c_file_to_semantic_module", "c_file_to_semantic_modules", "c_function_to_semantic_function", diff --git a/semantics/c2ir.py b/semantics/c2ir.py index 943b0440f..a752d94c1 100644 --- a/semantics/c2ir.py +++ b/semantics/c2ir.py @@ -55,7 +55,9 @@ SemanticArgument, SemanticArrayContract, SemanticClass, + SemanticCoercion, SemanticConstraint, + SemanticEnum, SemanticFunction, SemanticModule, SemanticOrigin, @@ -68,6 +70,7 @@ _IDENTIFIER_RE = re.compile(r"[^0-9A-Za-z_]+") _C_IDENTIFIER_TOKEN_RE = re.compile(r"\b[A-Za-z_][A-Za-z0-9_]*\b") _C_INTEGER_LITERAL_SUFFIX_RE = re.compile(r"(?&|^~()\s]+") _INTEGER_LITERAL_RE = re.compile(r"[-+]?(?:0[xX][0-9A-Fa-f]+|\d+)(?:[uUlL]*)\Z") _FLOAT_LITERAL_RE = re.compile( @@ -98,6 +101,7 @@ _NUMERIC_SEMANTIC_TYPES = frozenset( { "Bool", + "Int", "Int8", "Int16", "Int32", @@ -125,7 +129,7 @@ CUnsignedChar: "UInt8", CShort: "Int16", CUnsignedShort: "UInt16", - CInt: "Int32", + CInt: "Int", CUnsignedInt: "UInt32", CLong: "Int64", CUnsignedLong: "UInt64", @@ -152,6 +156,14 @@ "int64_t": "Int64", } +_C_INT_FALLBACK_FACT = { + "available": True, + "kind": "integer", + "signed": True, + "bits": 32, + "underlying_c_type": "int", +} + class CToIRConverter: """Convert parsed C models into the shared semantic IR. @@ -229,8 +241,9 @@ def visit_project_module( self.opaque_standard_types = set() try: semantic_functions = [self.visit_function(function) for function in project.functions.values()] + semantic_enums = [self.visit_enum(enum) for enum in self._project_enum_declarations(project)] semantic_variables = [ - *self._enum_constants(list(project.enums.values())), + *[enumerator for enum in semantic_enums for enumerator in enum.enumerators], *self._macro_constants_from_macros(list(project.macros.values())), *[self.visit_variable(variable) for variable in project.variables.values()], ] @@ -242,7 +255,7 @@ def visit_project_module( return SemanticModule( name=self._identifier(name), functions=semantic_functions, - classes=semantic_classes, + classes=[*semantic_enums, *semantic_classes], variables=semantic_variables, metadata=self._project_metadata(project), origin=SemanticOrigin( @@ -273,8 +286,9 @@ def visit_file( try: self.opaque_standard_types = set() semantic_functions = [self.visit_function(function) for function in c_file.functions] + semantic_enums = [self.visit_enum(enum) for enum in c_file.enums] semantic_variables = [ - *self._enum_constants(c_file.enums), + *[enumerator for enum in semantic_enums for enumerator in enum.enumerators], *self._macro_constants(c_file), *[self.visit_variable(variable) for variable in c_file.variables], ] @@ -286,7 +300,7 @@ def visit_file( module = SemanticModule( name=self._module_name(c_file), functions=semantic_functions, - classes=semantic_classes, + classes=[*semantic_enums, *semantic_classes], variables=semantic_variables, metadata=self._file_metadata(c_file), origin=SemanticOrigin( @@ -473,6 +487,34 @@ def visit_union(self, union: CUnion) -> SemanticClass: ), ) + def visit_enum(self, enum: CEnum) -> SemanticEnum: + enum = self._resolved_enum(enum) + name = self._enum_name(enum) + underlying_type = self._enum_underlying_type(enum) + metadata: dict[str, Any] = { + "c_kind": "enum", + "c_open": True, + } + if underlying_type.metadata.get("c_enum_type_fact_source") == "compiler_probe": + metadata["c_underlying_type_fact_source"] = "compiler_probe" + else: + metadata["c_underlying_type_assumption"] = "int" + return SemanticEnum( + name=name, + native_name=enum.reference_name, + underlying_type=underlying_type, + enumerators=self._enum_constants_for_enum(enum), + open=True, + metadata=metadata, + origin=SemanticOrigin( + source_language="c", + native_name=enum.reference_name, + source_kind="enum", + source_type=enum.reference_name, + source_location=self._location_dict(enum.source_location), + ), + ) + def visit_type(self, type_: CType, *, owner: str | None = None) -> SemanticType: if isinstance(type_, CComposedType): return self._composed_type(type_, owner=owner) @@ -506,14 +548,21 @@ def visit_type(self, type_: CType, *, owner: str | None = None) -> SemanticType: ) origin = self._type_origin(type_) - metadata = {} + metadata: dict[str, Any] = {} if "readiness_blockers" in origin.metadata: metadata["readiness_blockers"] = list(origin.metadata["readiness_blockers"]) if isinstance(type_, CChar): metadata["c_char_policy"] = "implementation-defined signed 8-bit code unit" + dtype = semantic_name + if isinstance(type_, CInt) and semantic_name == "Int": + fact, fact_source = self._c_int_fact() + dtype = self._semantic_type_from_standard_fact(fact) or "Int" + metadata["c_primitive"] = "int" + metadata["c_type_fact"] = fact + metadata["c_type_fact_source"] = fact_source return SemanticType( name=semantic_name, - dtype=semantic_name, + dtype=dtype, metadata=metadata, origin=origin, ) @@ -649,13 +698,41 @@ def _union_type(self, union: CUnion, *, owner: str | None) -> SemanticType: return semantic_type def _enum_type(self, enum: CEnum) -> SemanticType: + enum = self._resolved_enum(enum) + underlying_type = self._enum_underlying_type(enum) return SemanticType( - name="Int32", - dtype="Int32", - metadata={"c_kind": "enum", "c_enum": enum.reference_name}, + name=self._enum_name(enum), + dtype=underlying_type.dtype, + coercions=[SemanticCoercion(source_type="Int")], + metadata={ + "c_kind": "enum", + "c_enum": enum.reference_name, + "c_enum_open": True, + "c_underlying_type": underlying_type.name, + "c_underlying_dtype": underlying_type.dtype, + }, origin=self._type_origin(enum, native_name=enum.reference_name), ) + def _enum_underlying_type(self, enum: CEnum) -> SemanticType: + fact = self.standard_type_facts.get(enum.reference_name) + if fact is not None and fact.get("available", True): + dtype = self._semantic_type_from_standard_fact(fact) or "Int" + name = "Int" if fact.get("underlying_c_type") == "int" else dtype + return SemanticType( + name=name, + dtype=dtype, + metadata={ + "c_enum": enum.reference_name, + "c_enum_type_fact": dict(fact), + "c_enum_type_fact_source": "compiler_probe", + }, + ) + underlying_type = self.visit_type(CInt()) + underlying_type.metadata["c_enum"] = enum.reference_name + underlying_type.metadata["c_enum_underlying_assumption"] = "int" + return underlying_type + def _pointer_type( self, pointee: SemanticType, @@ -731,43 +808,48 @@ def _array_type( ) return element - def _enum_constants(self, enums: list[CEnum]) -> list[SemanticArgument]: + def _enum_constants_for_enum(self, enum: CEnum) -> list[SemanticArgument]: variables: list[SemanticArgument] = [] - for enum in enums: - next_value: int | None = 0 - for enumerator in enum.constants: - value = enumerator.value - if value is None and next_value is not None: - value = str(next_value) - literal = self._integer_literal_value(value) - next_value = literal + 1 if literal is not None else None - variables.append( - SemanticArgument( - name=enumerator.name, - semantic_type=SemanticType( - name="Int32", - dtype="Int32", - constraints=[SemanticConstraint("Constant")], - metadata={"c_enum": enum.reference_name}, - origin=SemanticOrigin( - source_language="c", - native_name=enumerator.name, - native_scope=enum.reference_name, - source_kind="enum_constant", - source_type="enum", - source_location=self._location_dict(enumerator.source_location), - ), - ), - default_value=value, - origin=SemanticOrigin( - source_language="c", - native_name=enumerator.name, - native_scope=enum.reference_name, - source_kind="enum_constant", - source_location=self._location_dict(enumerator.source_location), - ), - ) + enum = self._resolved_enum(enum) + next_value: int | None = 0 + for enumerator in enum.constants: + value = enumerator.value + if value is None and next_value is not None: + value = str(next_value) + literal = self._integer_literal_value(value) + next_value = literal + 1 if literal is not None else None + semantic_type = self._enum_type(enum) + semantic_type.constraints.append(SemanticConstraint("Constant")) + semantic_type.metadata["semantic_enum"] = self._enum_name(enum) + semantic_type.origin = SemanticOrigin( + source_language="c", + native_name=enumerator.name, + native_scope=enum.reference_name, + source_kind="enum_constant", + source_type="enum", + source_location=self._location_dict(enumerator.source_location), + ) + metadata: dict[str, Any] = {} + if value is not None: + metadata["c_value_expression"] = value + pyi_value = self._pyi_integer_expression(value) + if pyi_value is not None: + metadata["pyi_default_value"] = pyi_value + variables.append( + SemanticArgument( + name=enumerator.name, + semantic_type=semantic_type, + default_value=value, + metadata=metadata, + origin=SemanticOrigin( + source_language="c", + native_name=enumerator.name, + native_scope=enum.reference_name, + source_kind="enum_constant", + source_location=self._location_dict(enumerator.source_location), + ), ) + ) return variables def _macro_constants(self, c_file: CFile) -> list[SemanticArgument]: @@ -837,6 +919,24 @@ def _integer_macro_expression(value: str, macro_types: dict[str, str]) -> bool: for node in ast.walk(expression) ) + @staticmethod + def _pyi_integer_expression(value: str | None) -> str | None: + if value is None or not _INTEGER_EXPRESSION_CHARS_RE.fullmatch(value): + return None + normalized = _C_INTEGER_LITERAL_SUFFIX_RE.sub(r"\1", value) + normalized = _C_OCTAL_LITERAL_RE.sub(r"0o\1", normalized) + try: + expression = ast.parse(normalized, mode="eval") + except SyntaxError: + return None + if not all( + isinstance(node, (*_INTEGER_EXPRESSION_AST_NODES, ast.Name, ast.Load)) + and not (isinstance(node, ast.Constant) and not isinstance(node.value, int)) + for node in ast.walk(expression) + ): + return None + return ast.unparse(expression.body) + def _file_metadata(self, c_file: CFile) -> dict[str, Any]: metadata: dict[str, Any] = { "source_language": "c", @@ -908,7 +1008,14 @@ def is_private_origin(origin: SemanticOrigin) -> bool: for variable in module.variables: if is_private_origin(variable.origin): variable.visibility = "private" + for enum in module.enums: + if is_private_origin(enum.origin): + enum.visibility = "private" + for enumerator in enum.enumerators: + enumerator.visibility = "private" for cls in module.classes: + if not isinstance(cls, SemanticClass): + continue if is_private_origin(cls.origin): cls.visibility = "private" cls.fields = [] @@ -918,6 +1025,8 @@ def is_private_origin(origin: SemanticOrigin) -> bool: def _externalize_private_classes(self, module: SemanticModule) -> None: external_classes: dict[str, str] = {} for cls in module.classes: + if not isinstance(cls, SemanticClass): + continue if cls.visibility != "private" or "Opaque" not in cls.base_classes: continue filename = self._source_filename(cls.origin.source_location) @@ -957,6 +1066,13 @@ def _classify_project_external_types( if owner is None: continue owners[self._identifier(struct.name)] = (owner.name, not struct.is_incomplete) + for enum in self._project_enum_declarations(project): + if enum.source_location is None: + continue + owner = modules_by_filename.get(enum.source_location.filename) + if owner is None: + continue + owners[self._enum_name(enum)] = (owner.name, True) for module in modules: external_names = { @@ -1009,7 +1125,7 @@ def _project_metadata(self, project: CProject) -> dict[str, Any]: "functions": len(project.functions), "structs": len(project.structs), "unions": len(project.unions), - "enums": len(project.enums), + "enums": len(self._project_enum_declarations(project)), "typedefs": len(project.typedefs), "macros": len(project.macros), "includes": len(project.includes), @@ -1092,6 +1208,12 @@ def _standard_semantic_type(self, name: str) -> SemanticType | None: metadata={"c_standard_type": name, "c_standard_type_fallback": True}, ) + def _c_int_fact(self) -> tuple[dict[str, Any], str]: + fact = self.standard_type_facts.get("int") + if fact is not None and fact.get("available", True): + return dict(fact), "compiler_probe" + return dict(_C_INT_FALLBACK_FACT), "fallback" + def _opaque_standard_type_classes(self) -> list[SemanticClass]: return [ SemanticClass( @@ -1228,6 +1350,32 @@ def _union_name(self, union: CUnion) -> str: alias = self._typedef_alias_for_type(union) return self._identifier(alias or union.anonymous_id or "anonymous_union") + def _enum_name(self, enum: CEnum) -> str: + if enum.name: + return self._identifier(enum.name) + alias = self._typedef_alias_for_type(enum) + return self._identifier(alias or enum.anonymous_id or "anonymous_enum") + + def _resolved_enum(self, enum: CEnum) -> CEnum: + if enum.name and enum.name in self.enums: + return self.enums[enum.name] + return enum + + @staticmethod + def _project_enum_declarations(project: CProject) -> list[CEnum]: + declarations = list(project.enums.values()) + anonymous_ids: set[str | int] = {enum.anonymous_id or id(enum) for enum in declarations if enum.name is None} + for c_file in project.files.values(): + for enum in c_file.enums: + if enum.name is not None: + continue + identity: str | int = enum.anonymous_id or id(enum) + if identity in anonymous_ids: + continue + anonymous_ids.add(identity) + declarations.append(enum) + return declarations + def _typedef_alias_for_type(self, target: CType) -> str | None: for typedef in self.typedefs.values(): if typedef.type is target: @@ -1283,7 +1431,7 @@ def _ambiguous_pointer_argument(semantic_type: SemanticType) -> bool: return False if storage.read_only: return False - return semantic_type.name in _NUMERIC_SEMANTIC_TYPES + return semantic_type.name in _NUMERIC_SEMANTIC_TYPES or semantic_type.metadata.get("c_kind") == "enum" def _add_incomplete_by_value_blocker(self, semantic_type: SemanticType, *, owner: str) -> None: storage = semantic_type.storage @@ -1406,6 +1554,14 @@ def c_struct_to_semantic_class( return CToIRConverter(standard_type_report=standard_type_report).visit_struct(struct) +def c_enum_to_semantic_enum( + enum: CEnum, + *, + standard_type_report: Any | None = None, +) -> SemanticEnum: + return CToIRConverter(standard_type_report=standard_type_report).visit_enum(enum) + + def c_file_to_semantic_module( parsed_file: CFile, *, @@ -1444,6 +1600,7 @@ def c_project_to_semantic_module( __all__ = ( "CToIRConverter", + "c_enum_to_semantic_enum", "c_file_to_semantic_module", "c_file_to_semantic_modules", "c_function_to_semantic_function", diff --git a/semantics/fortran2ir.py b/semantics/fortran2ir.py index ef599f154..f7da128a7 100644 --- a/semantics/fortran2ir.py +++ b/semantics/fortran2ir.py @@ -27,6 +27,7 @@ SemanticArrayContract, SemanticClass, SemanticConstraint, + SemanticEnum, SemanticFunction, SemanticImport, SemanticImportItem, @@ -1232,12 +1233,18 @@ def _resolve_semantic_module_compile_time_values( _resolve_semantic_argument_compile_time_values(var, compile_time_values) for func in module.functions: _resolve_semantic_function_compile_time_values(func, compile_time_values) - for cls in module.classes: - for field in cls.fields: + for declaration in module.classes: + if isinstance(declaration, SemanticEnum): + _resolve_semantic_type_compile_time_values(declaration.underlying_type, compile_time_values) + for enumerator in declaration.enumerators: + _resolve_semantic_argument_compile_time_values(enumerator, compile_time_values) + declaration.metadata = _resolve_semantic_value(declaration.metadata, compile_time_values) + continue + for field in declaration.fields: _resolve_semantic_argument_compile_time_values(field, compile_time_values) - for method in cls.methods: + for method in declaration.methods: _resolve_semantic_function_compile_time_values(method, compile_time_values) - cls.metadata = _resolve_semantic_value(cls.metadata, compile_time_values) + declaration.metadata = _resolve_semantic_value(declaration.metadata, compile_time_values) module.metadata = _resolve_semantic_value(module.metadata, compile_time_values) diff --git a/semantics/models.py b/semantics/models.py index 39ef2a147..5ba733fa2 100644 --- a/semantics/models.py +++ b/semantics/models.py @@ -432,7 +432,7 @@ def _canonical_expression_text(text: str, name_map: dict[str, str]) -> str: # ============================================================ -# Semantic Classes +# Semantic Classes And Enums # ============================================================ @@ -455,6 +455,23 @@ class SemanticClass: origin: SemanticOrigin = field(default_factory=SemanticOrigin, compare=False) +@dataclass +class SemanticEnum: + name: str + + native_name: str | None = None + + underlying_type: SemanticType = field(default_factory=lambda: SemanticType("Int")) + + enumerators: list[SemanticArgument] = field(default_factory=list) + + open: bool = True + + metadata: dict[str, Any] = field(default_factory=dict) + visibility: str = "public" + origin: SemanticOrigin = field(default_factory=SemanticOrigin, compare=False) + + # ============================================================ # Semantic Modules # ============================================================ @@ -478,7 +495,7 @@ class SemanticModule: functions: list[SemanticFunction] = field(default_factory=list) - classes: list[SemanticClass] = field(default_factory=list) + classes: list[SemanticClass | SemanticEnum] = field(default_factory=list) variables: list[SemanticArgument] = field(default_factory=list) imports: list[str | SemanticImport] = field(default_factory=list) @@ -487,6 +504,10 @@ class SemanticModule: origin: SemanticOrigin = field(default_factory=SemanticOrigin, compare=False) + @property + def enums(self) -> list[SemanticEnum]: + return [declaration for declaration in self.classes if isinstance(declaration, SemanticEnum)] + def _iter_semantic_type_tree(semantic_type: SemanticType | None): if semantic_type is None: @@ -503,10 +524,15 @@ def _iter_semantic_type_tree(semantic_type: SemanticType | None): def _iter_module_semantic_types(module: SemanticModule): for variable in module.variables: yield from _iter_semantic_type_tree(variable.semantic_type) - for cls in module.classes: - for semantic_field in cls.fields: + for declaration in module.classes: + if isinstance(declaration, SemanticEnum): + yield from _iter_semantic_type_tree(declaration.underlying_type) + for enumerator in declaration.enumerators: + yield from _iter_semantic_type_tree(enumerator.semantic_type) + continue + for semantic_field in declaration.fields: yield from _iter_semantic_type_tree(semantic_field.semantic_type) - for method in cls.methods: + for method in declaration.methods: for argument in method.arguments: yield from _iter_semantic_type_tree(argument.semantic_type) yield from _iter_semantic_type_tree(method.return_type) diff --git a/semantics/pyi_parser.py b/semantics/pyi_parser.py index 0adc59f36..158768287 100644 --- a/semantics/pyi_parser.py +++ b/semantics/pyi_parser.py @@ -12,6 +12,7 @@ SemanticArrayContract, SemanticClass, SemanticConstraint, + SemanticEnum, SemanticFunction, SemanticImport, SemanticImportItem, @@ -86,6 +87,7 @@ def __init__(self, *, module_name: str): def parse(self, tree: ast.Module) -> SemanticModule: _ModuleVisitor(self).visit(tree) + self._link_enum_constants() return self.module def import_from(self, node: ast.ImportFrom) -> SemanticImport: @@ -113,6 +115,33 @@ def class_def(self, node: ast.ClassDef, *, visibility: str) -> SemanticClass: visibility=visibility, ) + def enum_def(self, node: ast.ClassDef, *, visibility: str) -> SemanticEnum: + if len(node.bases) != 1 or not self.is_subscript_of(node.bases[0], "Enum"): + raise ValueError(f"Enum declaration expects exactly one underlying type: {_node_text(node)!r}") + if len(node.body) != 1 or not isinstance(node.body[0], ast.Pass): + raise ValueError(f"Enum declarations keep enumerators at module scope: {_node_text(node)!r}") + items = self.subscript_items(node.bases[0]) + if len(items) != 1: + raise ValueError(f"Enum declaration expects exactly one underlying type: {_node_text(node)!r}") + return SemanticEnum( + name=node.name, + native_name=node.name, + underlying_type=self.semantic_type(items[0]), + open=True, + visibility=visibility, + ) + + def _link_enum_constants(self) -> None: + by_name = {enum.name: enum for enum in self.module.enums} + for variable in self.module.variables: + enum = by_name.get(variable.semantic_type.name) + if enum is None or not any( + constraint.name == "Constant" for constraint in variable.semantic_type.constraints + ): + continue + variable.semantic_type.metadata["semantic_enum"] = enum.name + enum.enumerators.append(variable) + def function_def( self, node: ast.FunctionDef, @@ -166,7 +195,7 @@ def ann_assign(self, node: ast.AnnAssign, *, default_intent: str) -> SemanticArg intent=intent, optional=self.default_marks_optional(node.value), visibility=visibility, - default_value=self.literal_default_value(node.value), + default_value=self.assignment_default_value(node.value, semantic_type), ) def decorators(self, nodes: list[ast.expr], *, context: str) -> _Decorators: @@ -683,6 +712,14 @@ def literal_default_value(node: ast.expr | None) -> str | None: return None return str(ast.literal_eval(node)) + @staticmethod + def assignment_default_value(node: ast.expr | None, semantic_type: SemanticType) -> str | None: + if node is None or _PyiAstParser.default_marks_optional(node): + return None + if any(constraint.name == "Constant" for constraint in semantic_type.constraints): + return ast.unparse(node) + return _PyiAstParser.literal_default_value(node) + @staticmethod def qualified_name(node: ast.AST) -> tuple[str, ...] | None: if isinstance(node, ast.Name): @@ -913,7 +950,10 @@ def visit_ClassDef(self, node: ast.ClassDef) -> None: decorators = self.parser.decorators(node.decorator_list, context="class") if decorators.has_native_call: raise ValueError(f"Unsupported class decorator: {ast.unparse(node.decorator_list[-1])!r}") - self.parser.module.classes.append(self.parser.class_def(node, visibility=decorators.visibility)) + if len(node.bases) == 1 and self.parser.is_subscript_of(node.bases[0], "Enum"): + self.parser.module.classes.append(self.parser.enum_def(node, visibility=decorators.visibility)) + else: + self.parser.module.classes.append(self.parser.class_def(node, visibility=decorators.visibility)) def visit_FunctionDef(self, node: ast.FunctionDef) -> None: decorators = self.parser.decorators(node.decorator_list, context=".pyi") @@ -978,14 +1018,16 @@ def _imported_type_refs(module: SemanticModule) -> dict[str, tuple[str, str, str def _reconcile_external_type_refs(modules: list[SemanticModule]) -> list[SemanticModule]: - definitions = {(module.name, cls.name): cls for module in modules for cls in module.classes} + definitions = {(module.name, declaration.name): declaration for module in modules for declaration in module.classes} for module in modules: for semantic_type in _iter_module_semantic_types(module): ref = semantic_type.metadata.get(EXTERNAL_TYPE_REF_METADATA) if not isinstance(ref, dict): continue - cls = definitions.get((ref.get("origin_module"), ref.get("name"))) - wrapped = cls is not None and "Opaque" not in cls.base_classes + declaration = definitions.get((ref.get("origin_module"), ref.get("name"))) + wrapped = declaration is not None and ( + not isinstance(declaration, SemanticClass) or "Opaque" not in declaration.base_classes + ) ref["wrapped"] = wrapped ref["representation"] = "wrapped" if wrapped else "opaque" return modules diff --git a/semantics/pyi_printer.py b/semantics/pyi_printer.py index e21d12210..d8346ac35 100644 --- a/semantics/pyi_printer.py +++ b/semantics/pyi_printer.py @@ -14,6 +14,7 @@ SemanticArrayContract, SemanticClass, SemanticConstraint, + SemanticEnum, SemanticFunction, SemanticImport, SemanticImportItem, @@ -36,6 +37,8 @@ def emit(self, node) -> str: return self.emit_module(node) if isinstance(node, SemanticClass): return self.emit_class(node) + if isinstance(node, SemanticEnum): + return self.emit_enum(node) if isinstance(node, SemanticMethod): return self.emit_method(node) if isinstance(node, SemanticFunction): @@ -181,6 +184,10 @@ def _emit_typed_name( text = f"{name}: {type_text}" if arg.optional: text += " = ..." + elif self._is_enum_constant(arg): + enum_value = self._enum_default_value(arg) + if enum_value is not None: + text += f" = {enum_value}" return text @staticmethod @@ -194,6 +201,21 @@ def _annotated_type_text(type_text: str, metadata: list[str]) -> str: def _is_constant(semantic_type: SemanticType) -> bool: return any(constraint.name == "Constant" for constraint in semantic_type.constraints) + @staticmethod + def _is_enum_constant(arg: SemanticArgument) -> bool: + return PyiPrinter._is_constant(arg.semantic_type) and bool( + arg.semantic_type.metadata.get("semantic_enum") or arg.semantic_type.metadata.get("c_enum") + ) + + @staticmethod + def _enum_default_value(arg: SemanticArgument) -> str | None: + pyi_value = arg.metadata.get("pyi_default_value") + if isinstance(pyi_value, str): + return pyi_value + if arg.origin.source_language == "c": + return None + return arg.default_value + @staticmethod def _without_constant_constraint(semantic_type: SemanticType) -> SemanticType: if not PyiPrinter._is_constant(semantic_type): @@ -247,10 +269,15 @@ def emit_class(self, cls: SemanticClass) -> str: {body} """.strip() + def emit_enum(self, enum: SemanticEnum) -> str: + decorator = "@private\n" if self._is_private(enum) else "" + underlying = self.emit_semantic_type(enum.underlying_type) + return f"{decorator}class {enum.name}(Enum[{underlying}]):\n pass" + def emit_module(self, module: SemanticModule) -> str: sections: list[str] = [] self._append_imports(sections, module) - self._append_items(sections, module.classes, self.emit_class) + self._append_items(sections, module.classes, self.emit) self._append_items(sections, module.variables, self.emit_data_member) self._append_items(sections, module.functions, self.emit_function) return "\n".join(sections) @@ -493,7 +520,9 @@ def opaque_dependency_modules( ) -> list[SemanticModule]: source_modules = _module_list(modules) known_modules = _module_list(available_modules) if available_modules is not None else source_modules - known_classes = {(module.name, cls.name) for module in known_modules for cls in module.classes} + known_classes = { + (module.name, cls.name) for module in known_modules for cls in module.classes if isinstance(cls, SemanticClass) + } dependencies: dict[str, set[str]] = {} for module in source_modules: for semantic_type in _iter_module_semantic_types(module): diff --git a/semantics/readiness.py b/semantics/readiness.py index 81c9bbfa4..a0d67550d 100644 --- a/semantics/readiness.py +++ b/semantics/readiness.py @@ -8,6 +8,7 @@ EXTERNAL_TYPE_REF_METADATA, SemanticArgument, SemanticClass, + SemanticEnum, SemanticFunction, SemanticImport, SemanticMethod, @@ -29,6 +30,7 @@ "Complex128", "Float32", "Float64", + "Int", "Int8", "Int16", "Int32", @@ -129,6 +131,8 @@ def _public_api_counts(self) -> dict[str, int]: n_functions += sum(1 for func in module.functions if _is_public(func)) n_variables += sum(1 for var in module.variables if _is_public(var)) for cls in module.classes: + if not isinstance(cls, SemanticClass): + continue if not _is_public(cls): continue n_classes += 1 @@ -164,7 +168,19 @@ def _check_module(self, module: SemanticModule) -> None: unit_kind="variable", ) + for enum in module.enums: + if not _is_public(enum): + continue + self._check_enum( + enum, + module=module, + known_shape_symbols=set(module_constants), + constant_names=module_constant_names, + ) + for cls in module.classes: + if not isinstance(cls, SemanticClass): + continue if not _is_public(cls): continue self._check_class( @@ -187,6 +203,33 @@ def _check_module(self, module: SemanticModule) -> None: unit_kind="function", ) + def _check_enum( + self, + enum: SemanticEnum, + *, + module: SemanticModule, + known_shape_symbols: set[str], + constant_names: set[str], + ) -> None: + owner = f"{module.name}.{enum.name}" + self._check_metadata_blockers( + enum.metadata, + owner=owner, + item=enum.name, + unit=owner, + unit_kind="enum", + ) + self._check_type( + enum.underlying_type, + owner=owner, + item=enum.name, + module=module, + known_shape_symbols=known_shape_symbols, + constant_names=constant_names, + unit=owner, + unit_kind="enum", + ) + def _check_class( self, cls: SemanticClass, @@ -526,8 +569,8 @@ def __init__(self, modules: list[SemanticModule]): self.import_aliases_by_module: dict[str, set[str]] = {} for module in modules: - self.known_types.update(cls.name for cls in module.classes) - self.known_types.update(f"{module.name}.{cls.name}" for cls in module.classes) + self.known_types.update(declaration.name for declaration in module.classes) + self.known_types.update(f"{module.name}.{declaration.name}" for declaration in module.classes) imported_modules, import_aliases, imported_types = _import_index(module.imports) self.imported_modules_by_module[module.name] = imported_modules self.import_aliases_by_module[module.name] = import_aliases diff --git a/tests/parser/c/test_c_cli_skeleton.py b/tests/parser/c/test_c_cli_skeleton.py index a2c50f615..4a2f142f8 100644 --- a/tests/parser/c/test_c_cli_skeleton.py +++ b/tests/parser/c/test_c_cli_skeleton.py @@ -241,7 +241,10 @@ def test_cli_c_semantics_json_stdout_for_header(tmp_path: Path): assert semantic_modules[0]["name"] == "api" assert semantic_modules[0]["functions"][0]["name"] == "add" - assert semantic_modules[0]["functions"][0]["arguments"][0]["semantic_type"]["name"] == "Int32" + argument_type = semantic_modules[0]["functions"][0]["arguments"][0]["semantic_type"] + assert argument_type["name"] == "Int" + assert argument_type["dtype"] == "Int32" + assert argument_type["metadata"]["c_type_fact_source"] == "fallback" def test_cli_c_wrap_readiness_human_output_for_header(tmp_path: Path): @@ -309,7 +312,7 @@ def test_cli_c_pyi_human_output_for_header(tmp_path: Path): assert f"File: {header}" in res.stdout assert "def add(" in res.stdout - assert "a: Int32" in res.stdout + assert "a: Int" in res.stdout def test_cli_c_pyi_out_requires_explicit_language_and_writes_when_selected(tmp_path: Path): diff --git a/tests/parser/test_c_standard_type_probe.py b/tests/parser/test_c_standard_type_probe.py index 135f89057..062b78e22 100644 --- a/tests/parser/test_c_standard_type_probe.py +++ b/tests/parser/test_c_standard_type_probe.py @@ -33,6 +33,7 @@ def test_c_standard_type_probe_source_queries_standard_headers_without_layout_cl assert "#include " in source assert "#include " in source assert "#include " in source + assert 'X2PY_PRINT_ARITHMETIC("int"' in source assert 'X2PY_PRINT_ARITHMETIC("size_t"' in source assert 'X2PY_PRINT_ARITHMETIC("uint32_t"' in source assert 'X2PY_PRINT_ARITHMETIC("time_t"' in source @@ -121,6 +122,13 @@ def test_c_standard_type_probe_reports_semantic_facts_from_native_compiler(): compiler = _required_c_compiler() report = probe_c_standard_types(PreprocessingConfig(mode="compiler", compiler=compiler, std="c11")) + c_int = report.types["int"] + assert c_int["available"] is True + assert c_int["kind"] == "integer" + assert c_int["signed"] is True + assert c_int["underlying_c_type"] == "int" + assert c_int["bits"] >= 16 + size_t = report.types["size_t"] assert size_t["available"] is True assert size_t["kind"] == "integer" diff --git a/tests/property/test_semantic_properties.py b/tests/property/test_semantic_properties.py index 61527fd08..38618c12f 100644 --- a/tests/property/test_semantic_properties.py +++ b/tests/property/test_semantic_properties.py @@ -34,7 +34,7 @@ ("_Bool", "Bool"), ("double", "Float64"), ("float", "Float32"), - ("int", "Int32"), + ("int", "Int"), ] ) _FORTRAN_SCALAR_TYPES = st.sampled_from( @@ -50,7 +50,6 @@ ("_Bool", "logical", "Bool"), ("double", "real", "Float64"), ("float", "real(4)", "Float32"), - ("int", "integer", "Int32"), ] ) _SEMANTIC_SCALAR_TYPES = st.sampled_from(["Bool", "Float32", "Float64", "Int32"]) diff --git a/tests/pyi/fixtures/c/general/basic_array_update.pyi b/tests/pyi/fixtures/c/general/basic_array_update.pyi index e6232b326..ca95a354e 100644 --- a/tests/pyi/fixtures/c/general/basic_array_update.pyi +++ b/tests/pyi/fixtures/c/general/basic_array_update.pyi @@ -1,10 +1,10 @@ def add1( - n: Int32, + n: Int, x: Float64[1] ) -> None: ... def add1_strided( - n: Int32, + n: Int, x: Ptr(Float64), - incx: Int32 + incx: Int ) -> None: ... diff --git a/tests/pyi/fixtures/c/general/c_richer_features.pyi b/tests/pyi/fixtures/c/general/c_richer_features.pyi index 7e443dae7..205b8a6dd 100644 --- a/tests/pyi/fixtures/c/general/c_richer_features.pyi +++ b/tests/pyi/fixtures/c/general/c_richer_features.pyi @@ -1,3 +1,6 @@ +class x2py_status(Enum[Int]): + pass + class x2py_flags: ready: UInt32 mode: UInt32 @@ -7,33 +10,33 @@ class x2py_context(Opaque): pass class x2py_scalar: - i32: Int32 + i32: Int u64: UInt64 f64: Float64 -X2PY_STATUS_OK: Final[Int32] +X2PY_STATUS_OK: Final[x2py_status] = 0 -X2PY_STATUS_RETRY: Final[Int32] +X2PY_STATUS_RETRY: Final[x2py_status] = 1 -X2PY_STATUS_ERROR: Final[Int32] +X2PY_STATUS_ERROR: Final[x2py_status] = -1 -def x2py_slow_path() -> Int32: ... +def x2py_slow_path() -> Int: ... def x2py_sort( items: Ptr(Any), count: SizeT, item_size: SizeT, compare: CFunctionPointer -) -> Int32: ... +) -> Int: ... def x2py_register_callback( context: Ptr(x2py_context), callback: CFunctionPointer, userdata: Ptr(Any) -) -> Int32: ... +) -> Int: ... def x2py_status_message( - status: Int32 + status: x2py_status ) -> Ptr(Const(Int8)): ... def x2py_fill_matrix( diff --git a/tests/pyi/fixtures/c/general/constants.pyi b/tests/pyi/fixtures/c/general/constants.pyi index 504360951..51f480c6e 100644 --- a/tests/pyi/fixtures/c/general/constants.pyi +++ b/tests/pyi/fixtures/c/general/constants.pyi @@ -1,19 +1,22 @@ -COORD_X: Final[Int32] +class coordinate_axis(Enum[Int]): + pass -COORD_Y: Final[Int32] +COORD_X: Final[coordinate_axis] = 0 -COORD_Z: Final[Int32] +COORD_Y: Final[coordinate_axis] = 1 + +COORD_Z: Final[coordinate_axis] = 2 X2PY_GENERAL_NMAX: Final[Int32] X2PY_GENERAL_ORIGIN_RANK: Final[Int32] -nmax: Int32 +nmax: Int origin: Float64[3] def coordinate_axis_name( - axis: Int32 + axis: coordinate_axis ) -> Ptr(Const(Int8)): ... def coordinate_axis_count() -> SizeT: ... diff --git a/tests/pyi/fixtures/c/general/math_api.pyi b/tests/pyi/fixtures/c/general/math_api.pyi index e5a21be9c..47e2b5518 100644 --- a/tests/pyi/fixtures/c/general/math_api.pyi +++ b/tests/pyi/fixtures/c/general/math_api.pyi @@ -1,16 +1,16 @@ def norm2( - n: Int32, + n: Int, x: Const(Float64[1]) ) -> Float64: ... def scale( - n: Int32, + n: Int, alpha: Float64, x: Float64[1] ) -> None: ... def dot( - n: Int32, + n: Int, x: Ptr(Const(Float64)), y: Ptr(Const(Float64)) ) -> Float64: ... diff --git a/tests/pyi/fixtures/c/general/mesh.pyi b/tests/pyi/fixtures/c/general/mesh.pyi index 794dadd4c..b17789706 100644 --- a/tests/pyi/fixtures/c/general/mesh.pyi +++ b/tests/pyi/fixtures/c/general/mesh.pyi @@ -1,5 +1,5 @@ class node: - id: Int32 + id: Int xyz: Float64[3] class mesh: @@ -14,7 +14,7 @@ def node_move( def mesh_init( mesh: Ptr(mesh), nnodes: SizeT -) -> Int32: ... +) -> Int: ... def mesh_clear( mesh: Ptr(mesh) diff --git a/tests/pyi/fixtures/c/general/modern_math_physics.pyi b/tests/pyi/fixtures/c/general/modern_math_physics.pyi index 87ca73441..a0a9f89ac 100644 --- a/tests/pyi/fixtures/c/general/modern_math_physics.pyi +++ b/tests/pyi/fixtures/c/general/modern_math_physics.pyi @@ -1,18 +1,18 @@ class modern_particle: - id: Int32 + id: Int mass: Float64 position: Float64[3] class vector3: values: Float64[3] -modern_counter: Int32 +modern_counter: Int hidden_scale: private[Float64] def init_particle( p: Ptr(modern_particle), - pid: Int32, + pid: Int, mass: Float64, x: Float64, y: Float64, @@ -27,7 +27,7 @@ def kinetic_energy( ) -> Float64: ... def scale_vector( - n: Int32, + n: Int, v: Float64[1], alpha: Float64 ) -> None: ... diff --git a/tests/pyi/fixtures/c/general/name_reuse.pyi b/tests/pyi/fixtures/c/general/name_reuse.pyi index 4c7819ed4..6576e7643 100644 --- a/tests/pyi/fixtures/c/general/name_reuse.pyi +++ b/tests/pyi/fixtures/c/general/name_reuse.pyi @@ -1,7 +1,7 @@ class same_name: - payload: Int32 + payload: Int -same_name_i: Int32 +same_name_i: Int same_name_r: Float32 @@ -12,7 +12,7 @@ same_name_c: Complex128 same_name_s: Int8[8] def do_work_i( - same_name: Ptr(Int32) + same_name: Ptr(Int) ) -> None: ... def do_work_r( @@ -25,13 +25,13 @@ def do_work_l( ) -> None: ... def convert_to_complex( - same_name: Int32 + same_name: Int ) -> Complex128: ... def convert_to_string( same_name: Float32, shared: Int8[16] -) -> Int32: ... +) -> Int: ... def convert_to_logical( same_name: Ptr(Const(Int8)) diff --git a/tests/pyi/fixtures/c/general/particles.pyi b/tests/pyi/fixtures/c/general/particles.pyi index 6090f94ba..bfa5ae513 100644 --- a/tests/pyi/fixtures/c/general/particles.pyi +++ b/tests/pyi/fixtures/c/general/particles.pyi @@ -1,5 +1,5 @@ class particle: - id: Int32 + id: Int x: Float64[3] current_particle: private[particle] diff --git a/tests/pyi/fixtures/c/general/shape_exprs.pyi b/tests/pyi/fixtures/c/general/shape_exprs.pyi index 18842a92c..3fc8d7077 100644 --- a/tests/pyi/fixtures/c/general/shape_exprs.pyi +++ b/tests/pyi/fixtures/c/general/shape_exprs.pyi @@ -9,27 +9,27 @@ X2PY_EXPR_B: Final[Int32] X2PY_EXPR_C: Final[Int32] def fill_grid( - x: Int32[1, 4 + 2] + x: Int[1, 4 + 2] ) -> None: ... def update_plane( - n: Int32, + n: Int, x: Float32[1, n] ) -> None: ... def use_expr( - x: Int32[4 + 2], + x: Int[4 + 2], y: Float32[4 * 2] ) -> None: ... def all_exprs( - x1: Int32[8 + 3], - x2: Int32[8 - 3], - x3: Int32[3 * 2], - x4: Int32[8 / 2], - x5: Int32[1 << 3], - x6: Int32[(8 + 3) * 2 - 1], - x7: Int32[-(-8 + 3)], - x8: Int32[(8 + 3) * (2 + 1) - 1], - x9: Int32[(8 - 3) * (8 - 2)] + x1: Int[8 + 3], + x2: Int[8 - 3], + x3: Int[3 * 2], + x4: Int[8 / 2], + x5: Int[1 << 3], + x6: Int[(8 + 3) * 2 - 1], + x7: Int[-(-8 + 3)], + x8: Int[(8 + 3) * (2 + 1) - 1], + x9: Int[(8 - 3) * (8 - 2)] ) -> None: ... diff --git a/tests/pyi/test_pyi_to_ir.py b/tests/pyi/test_pyi_to_ir.py index 03f945f52..c66fb31d2 100644 --- a/tests/pyi/test_pyi_to_ir.py +++ b/tests/pyi/test_pyi_to_ir.py @@ -13,6 +13,7 @@ SemanticImport, SemanticImportItem, SemanticModule, + SemanticEnum, SemanticType, ) from semantics.pyi_parser import ( @@ -129,6 +130,35 @@ def touch( assert module.functions[0].arguments[0].intent == "inout" +def test_parse_pyi_text_round_trips_open_enum_with_unscoped_enumerators(): + source = """class status(Enum[Int]): + pass + +STATUS_OK: Final[status] = 0 +STATUS_NEXT: Final[status] = STATUS_OK + 1 + +def set_status( + value: status +) -> None: ... +""" + + module = parse_pyi_text(source, module_name="status_api") + + assert len(module.enums) == 1 + enum = module.enums[0] + assert isinstance(enum, SemanticEnum) + assert enum.name == "status" + assert enum.open is True + assert enum.underlying_type.name == "Int" + assert [item.name for item in enum.enumerators] == ["STATUS_OK", "STATUS_NEXT"] + assert module.variables[1].default_value == "STATUS_OK + 1" + assert module.functions[0].arguments[0].semantic_type.name == "status" + emitted = emit_module(module) + assert "class status(Enum[Int]):" in emitted + assert "STATUS_NEXT: Final[status] = STATUS_OK + 1" in emitted + assert parse_pyi_text(emitted, module_name="status_api") == module + + def test_parse_pyi_text_preserves_callable_signature_metadata(): module = parse_pyi_text( """ diff --git a/tests/semantics/test_c2ir.py b/tests/semantics/test_c2ir.py index b55819ed7..24ad61ad9 100644 --- a/tests/semantics/test_c2ir.py +++ b/tests/semantics/test_c2ir.py @@ -52,6 +52,7 @@ ) from semantics.c2ir import ( CToIRConverter, + c_enum_to_semantic_enum, c_file_to_semantic_module, c_file_to_semantic_modules, c_function_to_semantic_function, @@ -64,11 +65,13 @@ from semantics.models import ( SemanticArgument, SemanticClass, + SemanticEnum, SemanticModule, SemanticOrigin, SemanticStorageContract, SemanticType, ) +from semantics.pyi_parser import parse_pyi_text from semantics.readiness import assess_semantic_wrap_readiness from semantics.pyi_printer import emit_module, emit_module_stubs @@ -122,11 +125,13 @@ def test_c2ir_converts_scalar_function_signatures_and_preserves_native_order(): assert module.name == "api" assert [arg.name for arg in add.arguments] == ["a", "b"] - assert [arg.semantic_type.name for arg in add.arguments] == ["Int32", "Int32"] + assert [arg.semantic_type.name for arg in add.arguments] == ["Int", "Int"] + assert [arg.semantic_type.dtype for arg in add.arguments] == ["Int32", "Int32"] assert [arg.metadata for arg in add.arguments] == [{"native_position": 0}, {"native_position": 1}] assert add.native_name == "add" assert add.visibility == "public" - assert add.return_type.name == "Int32" + assert add.return_type.name == "Int" + assert add.return_type.dtype == "Int32" assert [mapping.native_position for mapping in add.projection] == [0, 1] assert scale.return_type.name == "Float64" assert scale.arguments[0].semantic_type.metadata == {} @@ -587,29 +592,23 @@ def test_c2ir_converts_enum_constants_and_simple_macro_constants(): source_kind="macro", ) status_ok = constants["STATUS_OK"] - assert asdict(status_ok.semantic_type) == { - "name": "Int32", - "rank": 0, - "dtype": "Int32", - "shape": [], - "constraints": [{"name": "Constant", "arguments": []}], - "coercions": [], - "ownership": {"ownership": "borrowed", "mutable": False, "aliasing": True}, - "metadata": {"c_enum": "enum status"}, - "storage": None, - "origin": _c_origin( - native_name="STATUS_OK", - native_scope="enum status", - source_kind="enum_constant", - source_type="enum", - source_location={ - "filename": "constants.h", - "line": 2, - "column": 1, - "source_line": "enum status { STATUS_OK = 0, STATUS_WARN, STATUS_ERROR = 10 };", - }, - ), + enum = module.enums[0] + assert isinstance(enum, SemanticEnum) + assert enum.name == "status" + assert enum.open is True + assert enum.metadata == { + "c_kind": "enum", + "c_open": True, + "c_underlying_type_assumption": "int", } + assert enum.underlying_type.name == "Int" + assert enum.underlying_type.dtype == "Int32" + assert [enumerator.name for enumerator in enum.enumerators] == ["STATUS_OK", "STATUS_WARN", "STATUS_ERROR"] + assert status_ok.semantic_type.name == "status" + assert status_ok.semantic_type.dtype == "Int32" + assert status_ok.semantic_type.metadata["semantic_enum"] == "status" + assert status_ok.semantic_type.metadata["c_underlying_type"] == "Int" + assert status_ok.semantic_type.coercions[0].source_type == "Int" assert asdict(status_ok.origin) == _c_origin( native_name="STATUS_OK", native_scope="enum status", @@ -623,6 +622,77 @@ def test_c2ir_converts_enum_constants_and_simple_macro_constants(): ) +def test_c2ir_names_anonymous_typedef_enums_and_keeps_enumerators_unscoped(): + source = "typedef enum { FLAG_NONE = 0, FLAG_READ = 1 } flag_t; flag_t get_flags(void);" + parsed = parse_c_file(source, filename="flags.h") + + module = c_file_to_semantic_module(parsed) + project_module = c_project_to_semantic_module(parse_c_project({"flags.h": source}), name="flags") + + assert [enum.name for enum in module.enums] == ["flag_t"] + assert [enum.name for enum in project_module.enums] == ["flag_t"] + assert [variable.name for variable in module.variables] == ["FLAG_NONE", "FLAG_READ"] + assert [variable.name for variable in project_module.variables] == ["FLAG_NONE", "FLAG_READ"] + assert [variable.semantic_type.name for variable in module.variables] == ["flag_t", "flag_t"] + assert _function(module, "get_flags").return_type.name == "flag_t" + assert _function(project_module, "get_flags").return_type.name == "flag_t" + + +def test_c2ir_enum_values_emit_only_python_compatible_expressions(): + parsed = parse_c_file( + "enum flags { FLAG_ONE = 1U, FLAG_OCTAL = 010, FLAG_SHIFT = FLAG_ONE << 1, FLAG_CHAR = 'A' };", + filename="flags.h", + ) + module = c_file_to_semantic_module(parsed) + + code = emit_module(module) + + assert "FLAG_ONE: Final[flags] = 1" in code + assert "FLAG_OCTAL: Final[flags] = 8" in code + assert "FLAG_SHIFT: Final[flags] = FLAG_ONE << 1" in code + assert "FLAG_CHAR: Final[flags]\n" in code + assert {variable.name: variable.default_value for variable in module.variables} == { + "FLAG_ONE": "1U", + "FLAG_OCTAL": "010", + "FLAG_SHIFT": "FLAG_ONE << 1", + "FLAG_CHAR": "'A'", + } + assert parse_pyi_text(code, module_name="flags").enums[0].name == "flags" + + +def test_c2ir_cross_header_enum_references_import_the_owner_enum(): + project = parse_c_project( + { + "types.h": "enum status { STATUS_OK = 0 };", + "api.h": "enum status get_status(void);", + } + ) + + modules = {module.name: module for module in c_project_to_semantic_modules(project)} + + assert modules["api"].enums == [] + assert [enum.name for enum in modules["types"].enums] == ["status"] + assert _function(modules["api"], "get_status").return_type.metadata["external_type_ref"] == { + "name": "status", + "local_name": "status", + "origin_module": "types", + "wrapped": True, + "representation": "wrapped", + } + + anonymous_project = parse_c_project( + { + "types.h": "typedef enum { FLAG_NONE = 0 } flag_t;", + "api.h": "flag_t get_flags(void);", + } + ) + anonymous_modules = {module.name: module for module in c_project_to_semantic_modules(anonymous_project)} + assert ( + _function(anonymous_modules["api"], "get_flags").return_type.metadata["external_type_ref"]["origin_module"] + == "types" + ) + + def test_c2ir_converts_integer_expression_macro_constants_when_resolvable(): parsed = parse_c_file( """ @@ -765,6 +835,71 @@ def test_c2ir_uses_standard_type_probe_facts_when_supplied(): assert _function(module, "count").return_type.name == "UInt32" +def test_c2ir_preserves_c_int_identity_and_stores_compiler_probed_precision(): + converter = CToIRConverter( + standard_type_report={ + "types": { + "int": { + "available": True, + "kind": "integer", + "signed": True, + "bits": 16, + "underlying_c_type": "int", + } + } + } + ) + + semantic_type = converter.visit_type(CInt()) + + assert semantic_type.name == "Int" + assert semantic_type.dtype == "Int16" + assert semantic_type.metadata == { + "c_primitive": "int", + "c_type_fact": { + "available": True, + "kind": "integer", + "signed": True, + "bits": 16, + "underlying_c_type": "int", + }, + "c_type_fact_source": "compiler_probe", + } + + +def test_c2ir_uses_enum_specific_underlying_type_facts_when_supplied(): + parsed = parse_c_file( + "enum status { STATUS_OK = 0, STATUS_ERROR = 255 }; enum status get_status(void);", + filename="status.h", + ) + module = CToIRConverter( + standard_type_report={ + "types": { + "enum status": { + "available": True, + "kind": "integer", + "signed": False, + "bits": 8, + "underlying_c_type": "unsigned char", + } + } + } + ).visit_file(parsed) + + enum = module.enums[0] + return_type = _function(module, "get_status").return_type + assert enum.underlying_type.name == "UInt8" + assert enum.underlying_type.dtype == "UInt8" + assert enum.underlying_type.metadata["c_enum_type_fact_source"] == "compiler_probe" + assert enum.metadata == { + "c_kind": "enum", + "c_open": True, + "c_underlying_type_fact_source": "compiler_probe", + } + assert return_type.name == "status" + assert return_type.dtype == "UInt8" + + def test_c2ir_uses_standard_type_probe_opaque_handle_facts(): parsed = parse_c_file("void close_file(FILE *stream);\n", filename="stdio_api.h") converter = CToIRConverter( @@ -843,11 +978,13 @@ def test_c_compatibility_helpers_forward_standard_type_reports(): CStruct(name="measurement", members=[CVariable(name="value", type=measured_type)]), standard_type_report=report, ) + enum = c_enum_to_semantic_enum(CEnum(name="measurement_status"), standard_type_report=report) assert argument.semantic_type.name == "UInt32" assert converted_type.name == "UInt32" assert converted_function.return_type.name == "UInt32" assert cls.fields[0].semantic_type.name == "UInt32" + assert enum.name == "measurement_status" assert _function( c_file_to_semantic_module(parsed_file, standard_type_report=report), "measure" ).return_type.name == ("UInt32") @@ -863,33 +1000,33 @@ def test_c_compatibility_helpers_forward_standard_type_reports(): @pytest.mark.parametrize( - ("ctype", "expected"), + ("ctype", "expected_name", "expected_dtype"), [ - (CBool(), "Bool"), - (CChar(), "Int8"), - (CSignedChar(), "Int8"), - (CUnsignedChar(), "UInt8"), - (CShort(), "Int16"), - (CUnsignedShort(), "UInt16"), - (CInt(), "Int32"), - (CUnsignedInt(), "UInt32"), - (CLong(), "Int64"), - (CUnsignedLong(), "UInt64"), - (CLongLong(), "Int64"), - (CUnsignedLongLong(), "UInt64"), - (CFloat(), "Float32"), - (CDouble(), "Float64"), - (CLongDouble(), "Float128"), - (CFloatComplex(), "Complex64"), - (CDoubleComplex(), "Complex128"), - (CLongDoubleComplex(), "Complex256"), + (CBool(), "Bool", "Bool"), + (CChar(), "Int8", "Int8"), + (CSignedChar(), "Int8", "Int8"), + (CUnsignedChar(), "UInt8", "UInt8"), + (CShort(), "Int16", "Int16"), + (CUnsignedShort(), "UInt16", "UInt16"), + (CInt(), "Int", "Int32"), + (CUnsignedInt(), "UInt32", "UInt32"), + (CLong(), "Int64", "Int64"), + (CUnsignedLong(), "UInt64", "UInt64"), + (CLongLong(), "Int64", "Int64"), + (CUnsignedLongLong(), "UInt64", "UInt64"), + (CFloat(), "Float32", "Float32"), + (CDouble(), "Float64", "Float64"), + (CLongDouble(), "Float128", "Float128"), + (CFloatComplex(), "Complex64", "Complex64"), + (CDoubleComplex(), "Complex128", "Complex128"), + (CLongDoubleComplex(), "Complex256", "Complex256"), ], ) -def test_c_primitive_precisions_map_to_semantic_types(ctype, expected): +def test_c_primitive_precisions_map_to_semantic_types(ctype, expected_name, expected_dtype): semantic_type = CToIRConverter().visit_type(ctype) - assert semantic_type.name == expected - assert semantic_type.dtype == expected + assert semantic_type.name == expected_name + assert semantic_type.dtype == expected_dtype @pytest.mark.parametrize( @@ -931,11 +1068,13 @@ def test_c2ir_visitor_and_project_compatibility_entrypoints_cover_supported_node assert converter.visit(first.structs[0]).name == "point" assert converter.visit(CUnion(name="loose_union")).name == "loose_union" assert converter.visit(first.variables[0]).name == "value" - assert converter.visit(CInt()).name == "Int32" + assert converter.visit(CInt()).name == "Int" enum_type = converter.visit(CEnum(name="status")) - assert enum_type.name == "Int32" + assert enum_type.name == "status" assert enum_type.dtype == "Int32" - assert enum_type.metadata == {"c_kind": "enum", "c_enum": "enum status"} + assert enum_type.metadata["c_kind"] == "enum" + assert enum_type.metadata["c_enum"] == "enum status" + assert enum_type.metadata["c_underlying_type"] == "Int" assert enum_type.origin.native_name == "enum status" assert enum_type.origin.metadata["c_type"] == "CEnum" with pytest.raises(TypeError) as error: @@ -959,12 +1098,12 @@ def test_c2ir_visitor_and_project_compatibility_entrypoints_cover_supported_node typedefs={"contextual_t": CTypedef(name="contextual_t", type=CInt())}, unions={"context_union": contextual_union}, ) - assert _function(contextual_module, "contextual").return_type.name == "Int32" + assert _function(contextual_module, "contextual").return_type.name == "Int" assert _function(contextual_module, "use_context_union").arguments[0].semantic_type.metadata["incomplete"] is False assert [cls.name for cls in contextual_module.classes] == ["context_union"] assert c_file_to_semantic_module(first).name == "a" - assert c_type_to_semantic_type(CInt()).name == "Int32" + assert c_type_to_semantic_type(CInt()).name == "Int" assert c_parameter_to_semantic_argument(CParameter(name=None, type=CInt()), position=2).name == "arg2" default_argument = c_parameter_to_semantic_argument(CParameter(name=None, type=CInt())) assert default_argument.name == "arg0" @@ -1017,9 +1156,9 @@ def test_c2ir_visitor_and_project_compatibility_entrypoints_cover_supported_node typedefs={"global_count_t": CTypedef(name="global_count_t", type=CInt())}, ) reference_modules = converter.visit_project(reference_project) - assert _function(reference_modules[0], "global_count").return_type.name == "Int32" + assert _function(reference_modules[0], "global_count").return_type.name == "Int" reference_merged = converter.visit_project_module(reference_project) - assert _function(reference_merged, "global_count").return_type.name == "Int32" + assert _function(reference_merged, "global_count").return_type.name == "Int" record = CStruct(name="global_record", members=[CVariable(name="value", type=CInt())]) choice = CUnion(name="global_choice", members=[CVariable(name="value", type=CInt())]) registry_function = CFunction( @@ -1108,15 +1247,15 @@ def test_c2ir_converts_qualifiers_callbacks_bitfields_and_unspecified_functions( source_kind="function_pointer", source_type="void (*)(int)", ) - assert field.semantic_type.metadata == { - "readiness_blockers": [ - _blocker( - "c_bitfield_unsupported", - "C bitfields require explicit semantic policy before wrapping.", - {"owner": "bits", "field": "bits", "bit_width": "3"}, - ) - ] - } + assert field.semantic_type.metadata["c_primitive"] == "int" + assert field.semantic_type.metadata["c_type_fact"]["bits"] == 32 + assert field.semantic_type.metadata["readiness_blockers"] == [ + _blocker( + "c_bitfield_unsupported", + "C bitfields require explicit semantic policy before wrapping.", + {"owner": "bits", "field": "bits", "bit_width": "3"}, + ) + ] assert field.intent == "in" assert field.visibility == "public" assert mutable_pointer_variable.intent == "inout" @@ -1341,7 +1480,7 @@ def test_c2ir_models_pointer_to_arrays_unknown_extents_unions_and_anonymous_alia "variant_t": CTypedef(name="variant_t", type=anon_union), } - assert direct_array.name == "Int32" + assert direct_array.name == "Int" assert direct_array.rank == 1 assert direct_array.shape == ["2"] assert direct_array.storage.array.shape == ["2"] @@ -1365,7 +1504,7 @@ def test_c2ir_models_pointer_to_arrays_unknown_extents_unions_and_anonymous_alia {"owner": "matrix", "type": "double (*)[]"}, ) ] - assert pointer_matrix.name == "Int32" + assert pointer_matrix.name == "Int" assert pointer_matrix.shape == ["2", "3"] assert union_type.name == "choice" assert union_type.dtype == "choice" diff --git a/tests/semantics/test_c_semantic_readiness.py b/tests/semantics/test_c_semantic_readiness.py index 6a85d3058..189bc1deb 100644 --- a/tests/semantics/test_c_semantic_readiness.py +++ b/tests/semantics/test_c_semantic_readiness.py @@ -102,6 +102,28 @@ def test_c_semantic_readiness_reports_pointer_ownership_ambiguity(): assert any(blocker["code"] == "c_pointer_ownership_ambiguous" for blocker in report["wrappability_blockers"]) +def test_c_semantic_readiness_accepts_enum_values_and_blocks_mutable_enum_pointers(): + from c_parser import parse_c_file + from semantics.c2ir import c_file_to_semantic_modules + from semantics.readiness import assess_semantic_wrap_readiness + + parsed = parse_c_file( + """ +enum status { STATUS_OK = 0 }; +enum status get_status(void); +void update_status(enum status *status); +""", + filename="status.h", + ) + modules = c_file_to_semantic_modules(parsed) + report = assess_semantic_wrap_readiness(modules, source="status.h") + + assert report["wrappable"] is False + blockers = {blocker["code"]: blocker for blocker in report["wrappability_blockers"]} + assert "unresolved_semantic_types" not in blockers + assert blockers["c_pointer_ownership_ambiguous"]["items"][0]["owner"] == "update_status.status" + + def test_c_semantic_readiness_aggregates_file_and_function_blockers(): from c_parser import parse_c_file from semantics.c2ir import c_file_to_semantic_modules diff --git a/tests/semantics/test_fortran2ir.py b/tests/semantics/test_fortran2ir.py index 1e3916e4a..acb176da2 100644 --- a/tests/semantics/test_fortran2ir.py +++ b/tests/semantics/test_fortran2ir.py @@ -731,6 +731,30 @@ def test_resolve_semantic_compile_time_values_rewrites_shapes_and_constraints(): assert resolved.variables[0].semantic_type.storage.array.shape == ["1:8"] +def test_resolve_semantic_compile_time_values_handles_enum_declarations(): + enumerator = SemanticArgument( + name="STATUS_LIMIT", + semantic_type=SemanticType("status"), + default_value="n", + ) + module = SemanticModule( + name="status_mod", + classes=[ + semantic_models.SemanticEnum( + name="status", + underlying_type=SemanticType("Int", metadata={"bits": "n"}), + enumerators=[enumerator], + ) + ], + variables=[enumerator], + ) + + resolved = resolve_semantic_compile_time_values(module, {"n": 16}) + + assert resolved.enums[0].underlying_type.metadata == {"bits": "16"} + assert resolved.enums[0].enumerators[0].default_value == "16" + + def test_resolve_semantic_compile_time_values_handles_nested_modules(): module = SemanticModule( name="nested_mod", diff --git a/x2py/__init__.py b/x2py/__init__.py index 9a0fc6174..87309ebed 100644 --- a/x2py/__init__.py +++ b/x2py/__init__.py @@ -27,6 +27,7 @@ ) from semantics.c2ir import ( CToIRConverter, + c_enum_to_semantic_enum, c_file_to_semantic_module, c_file_to_semantic_modules, c_function_to_semantic_function, @@ -80,6 +81,7 @@ def __getattr__(name: str): "assess_pyi_wrap_readiness", "assess_semantic_wrap_readiness", "build_fortran_type_probe_source", + "c_enum_to_semantic_enum", "c_file_to_semantic_module", "c_file_to_semantic_modules", "c_function_to_semantic_function", diff --git a/x2py/c_type_probe.py b/x2py/c_type_probe.py index f5a77fcec..1a78cb481 100644 --- a/x2py/c_type_probe.py +++ b/x2py/c_type_probe.py @@ -104,6 +104,8 @@ def build_c_standard_type_probe_source() -> str: int main(void) { printf("{\"types\":{"); + X2PY_PRINT_ARITHMETIC("int", "", int); + printf(","); X2PY_PRINT_ARITHMETIC("size_t", "stddef.h", size_t); printf(","); #ifdef UINT32_MAX