Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ examples. Contribution and pull-request requirements remain in
long-term architecture and runtime model.
- [Wrapper `.pyi` format](semantics/pyi_format.md): editable semantic
interface syntax and conversion behavior.
- [C to semantic IR mapping](semantics/c2ir_mapping.md): implemented C
semantic conversion subset and blocker policy.
- [Self-contained C semantic `.pyi` specification](semantics/c_pyi_self_contained_specification.md):
staged C wrapper interface design.

Expand All @@ -29,8 +31,8 @@ examples. Contribution and pull-request requirements remain in

### C

- [C parser reference](c_parser/c_parser_reference.md): implemented parse-only
frontend behavior and testing workflow.
- [C parser reference](c_parser/c_parser_reference.md): implemented parser
behavior, semantic handoff, and testing workflow.
- [C parser architecture plan](c_parser/c_parser_architecture.md): design and
integration decisions.
- [C parser CLI workflow plan](c_parser/c_parser_cli_workflow.md): command
Expand Down
28 changes: 17 additions & 11 deletions docs/c_parser/c_parser_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The x2py CLI also has a shared C/Fortran preprocessing option surface and can
run an exact compiler/preprocessor executable for C compiler mode. Compiler
and preprocessed C inputs preserve `#line`/GCC linemarker source locations for
parsed declarations and diagnostics. A separate compiler-derived standard-type
probe supplies target ABI facts needed by later C semantic conversion.
probe supplies target ABI facts consumed by C semantic conversion.

This document records the target architecture for the C parser frontend in
x2py. The initial skeleton has grown into a partial parser, and the remaining
Expand Down Expand Up @@ -91,12 +91,18 @@ Implemented now:
preserving compiler/runner/source provenance for semantic conversion. It
carries target-relevant include, macro, undefine, and compiler-argument flags
and records the requested project standard as provenance.
- `--language c --semantics`, `--language c --pyi`, and C wrap-readiness are
rejected until semantic conversion exists.
- `semantics.c2ir` converts the supported C parser subset into semantic IR,
including scalar functions, pointer storage contracts, declared arrays,
structs/opaque structs, enum and numeric macro constants, local typedef
chains, target standard-type probe facts, and C-specific readiness blockers.
- `--language c --semantics`, `--language c --wrap-readiness`, and starter
exact-contract `--language c --pyi` output are enabled for the supported C
semantic subset.
- Focused partial CLI/API, declaration/function, diagnostic color, project
include/index, raw lexer/directive, project golden, error golden, preprocessed
linemarker remapping, and JSON schema tests are active. Remaining
parser-suite skips are limited to the pinned corpus roadmap.
linemarker remapping, JSON schema, and cJSON partial-parse regression tests
are active. A separately pinned/provenanced corpus remains deferred work,
not a skipped test.
- `tests/data/c/` contains general fixtures modeled after the Fortran general
fixture themes, additional C-specific API shapes, fatal diagnostic inputs,
and real-world cJSON/jsmn/tinyexpr/linmath/NanoSVG/stb inputs whose partial
Expand All @@ -110,7 +116,7 @@ Deferred:
- compiler attributes and alignment specifiers
- broader compiler-family validation for preprocessing; parsed declarations
already retain preprocessed origin and mapped source identity
- C semantic readiness, semantic IR conversion, and `.pyi` output
- broader C callback/ownership policy beyond exact starter `.pyi` stubs

Documentation rule: any future C parser implementation change must update all
affected docs under `docs/c_parser/` in the same change. This applies to model,
Expand Down Expand Up @@ -328,8 +334,8 @@ class CParser:

These entrypoints are exposed from both `c_parser` and `x2py.__init__`, using
the same top-level file/project invocation pattern already provided for
Fortran. C semantic conversion remains unavailable despite the parse API
export.
Fortran. C semantic conversion is exposed separately through
`semantics.c2ir` and top-level `x2py` compatibility helpers.

## Core Model Families

Expand Down Expand Up @@ -723,9 +729,9 @@ Python APIs.

## `.pyi` Integration

Generated `.pyi` stubs for C should come after parser models and semantic IR
conversion are stable. Readiness, if added for C, should follow the semantic
layer pattern already used by the project.
Generated `.pyi` stubs for C are emitted from semantic IR for the supported
exact-contract subset. Readiness follows the semantic-layer pattern already
used by the project.

Likely stub patterns:

Expand Down
39 changes: 21 additions & 18 deletions docs/c_parser/c_parser_cli_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ represent parenthesized pointer/array precedence through concrete
parameters expose both declared and C-adjusted effective type facts. The CLI
also exposes shared C/Fortran preprocessing flags and can run exact
compiler/preprocessor executables for compiler mode. Target-specific standard
header type facts for later C semantics are available through the separate
`python -m x2py.c_type_probe` command.
header type facts for C semantics are available through the separate
`python -m x2py.c_type_probe` command. C semantic IR, readiness, and starter
exact-contract `.pyi` output are available through the shared `x2py` CLI.

This document records the implemented C parse command shape, output schema,
and diagnostic contract, plus deferred CLI behavior.
Expand All @@ -28,6 +29,9 @@ python -m x2py path/to/api.h --language c --parse
python -m x2py path/to/api.h --language c --parse --json
python -m x2py path/to/api.h --language c --parse --out report.json
python -m x2py path/to/api.h --language c --parse --preprocess compiler --compiler clang-18 -I include -D API_EXPORT= --std c11
python -m x2py path/to/api.h --language c --semantics
python -m x2py path/to/api.h --language c --wrap-readiness
python -m x2py path/to/api.h --language c --pyi
```

The C parser accepts explicit `.c`, `.h`, and direct `.i` files, plus
Expand Down Expand Up @@ -86,17 +90,15 @@ as `int i; int i;` are also merged. Duplicate definitions and incompatible
top-level redeclarations are reported as diagnostics. Local declarations inside
function bodies are not parsed.

Unsupported C stages:
Unsupported C display controls:

```bash
python -m x2py path/to/api.h --language c --semantics
python -m x2py path/to/api.h --language c --pyi
python -m x2py path/to/api.h --language c --wrap-readiness
python -m x2py path/to/api.h --language c --parse --show-vars
python -m x2py path/to/api.h --language c --parse --print-limit 20
```

These commands return clear argparse errors until C semantic IR conversion and
`.pyi` generation are implemented. Fortran-only parse display flags such as
`--show-vars` and `--print-limit` are rejected in C mode.
Fortran-only parse display flags such as `--show-vars` and `--print-limit`
return clear argparse errors in C mode until C-specific display controls exist.

## Current CLI Baseline

Expand Down Expand Up @@ -402,9 +404,9 @@ JSON output for a file without raw directives:
}
```

The parser should not claim C files are wrappable. If C readiness is added
later, it should follow the semantics-owned readiness boundary used elsewhere
in x2py, not become parser JSON.
The parser should not claim C files are wrappable. C readiness follows the
semantics-owned readiness boundary used elsewhere in x2py and does not become
parser JSON.

For raw directives, the same JSON shape is used, but `includes`, `macros`,
`raw_directives`, `macro_dependencies`, and `diagnostics` may contain populated
Expand Down Expand Up @@ -597,10 +599,9 @@ The active CLI/parser tests cover the current partial subset:
by focused C tests.
- `--show-vars` and `--print-limit` are rejected in C mode until C-specific
display controls exist.
- `--semantics` with `--language c` is rejected until C semantic conversion is
implemented.
- `--pyi` with `--language c` is rejected until C `.pyi` emission is
implemented.
- `--semantics`, `--wrap-readiness`, and `--pyi` with `--language c` use
`semantics.c2ir`, the semantic readiness checker, and the shared `.pyi`
emitter for the supported exact-contract subset.

## Integration Order

Expand Down Expand Up @@ -648,8 +649,10 @@ Completed order:
including recursively mapped preprocessed provenance.
21. Added `_Atomic(type)` type-specifier parsing on the shared declarator path
and executable parser-developer walkthrough tests.
22. Added compiler-derived standard-header ABI probing for future C semantic
mapping without hard-coded host type aliases.
22. Added compiler-derived standard-header ABI probing for C semantic mapping
without hard-coded host type aliases.
23. Added C semantic IR, readiness, and starter exact-contract `.pyi` output
through `x2py --language c`.

Next implementation work should continue with fixture-driven compiler
extension policy and broader project conflict policy
Expand Down
35 changes: 21 additions & 14 deletions docs/c_parser/c_parser_reference.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# C Parser Reference

Status: current reference for the parse-only C frontend. The `c_parser`
Status: current reference for the partial C frontend. The `c_parser`
package, typed parser models, explicit C CLI parse path, raw directive
metadata, compiler-assisted preprocessing, source-location remapping, project
indexes, parser goldens, and C standard-type probe are implemented. C semantic
readiness, semantic IR conversion, and `.pyi` generation remain future work and
are intentionally rejected by the CLI.
indexes, parser goldens, C standard-type probe, first semantic IR conversion
subset, semantic readiness path, and starter exact-contract C `.pyi`
generation are implemented.

## Purpose

Expand Down Expand Up @@ -52,8 +52,11 @@ Implemented:
the `c_parser` package entrypoints
- `CParseError` with compiler-style diagnostic formatting
- explicit `x2py --language c --parse` output
- explicit `x2py --language c --semantics` and
`x2py --language c --wrap-readiness` output
- starter exact-contract `x2py --language c --pyi` output for the supported C
semantic subset
- C JSON partial output and `--out` behavior
- rejection of C `--semantics`, `--pyi`, and `--wrap-readiness`
- raw lexer records with comment stripping, line-continuation folding, and
lightweight token source locations
- top-level source splitting that tracks braces, parentheses, brackets, and
Expand Down Expand Up @@ -123,14 +126,19 @@ Implemented:
`tests/data/c/errors/parser/`, and partial-parser regression inputs under
`tests/data/c/json/`, `tests/data/c/tinyexpr/`, `tests/data/c/linmath/`,
`tests/data/c/nanosvg/`, and top-level C inputs from `tests/data/c/stb/`
- `semantics.c2ir` conversion for the first identity subset: scalar
functions, const/mutable pointer storage contracts, declared arrays,
structs/opaque structs, enums, numeric macro constants, local typedef
chains, standard-type probe facts, and explicit semantic readiness blockers

Still deferred:

- callback policy metadata beyond parser-side callback candidates
- broad compiler-extension declarators
- broader typedef/tag conflict policy beyond the implemented basic project
resolution
- semantic readiness, semantic IR conversion, and `.pyi` generation
- richer C ownership/callback projection policy beyond exact starter `.pyi`
stubs

## Supported C Subset

Expand Down Expand Up @@ -292,9 +300,8 @@ target-relevant flags from the matching entry to the probe explicitly.
For cross targets, provide a runner, for example `--runner=qemu-aarch64
--runner=-L --runner=/opt/aarch64-sysroot`.

The eventual C semantic converter should accept this report as target context.
The parser model remains source-faithful and does not embed host ABI
assumptions.
The C semantic converter accepts this report as target context. The parser
model remains source-faithful and does not embed host ABI assumptions.

## Public API

Expand Down Expand Up @@ -603,16 +610,16 @@ Test families should mirror the Fortran parser:
- typedef tests
- macro/constant tests
- include/project tests
- semantic readiness tests once C semantic conversion exists
- C semantic readiness tests
- CLI tests
- semantic conversion tests
- `.pyi` generation/parser tests
- fixture/golden parser tests
- error fixture/golden tests
- corpus parse-only tests

The C test area contains active partial-parser/raw-metadata tests plus narrowly
scoped roadmap skips under `tests/parser/c/`. The active tests cover
The C test area contains active partial-parser/raw-metadata tests, including
parse-only cJSON regression coverage under `tests/parser/c/`. The active tests cover
public entrypoints, empty model serialization, CLI discovery, JSON/output-file
behavior, unsupported C stages, comment stripping, line-continuation folding,
top-level splitting, include collection, simple macro collection, macro-shaped
Expand All @@ -625,8 +632,8 @@ prototypes/definitions, function-definition start/end locations, JSON golden
serialization, fatal diagnostic goldens, and project-level callback typedef
resolution. The `json` regression inputs
intentionally retain recoverable diagnostics from unsupported constructs; they
do not claim complete library parsing. Remaining parser-suite skips cover the
pinned/provenanced corpus target. Golden comparison tests rewrite their baselines when
do not claim complete library parsing. A separately pinned/provenanced corpus
target remains deferred without disabling parser tests. Golden comparison tests rewrite their baselines when
`C_PARSER_UPDATE_GOLDENS=1` is set. Future implementation branches should
activate only the tests for the capability they implement.

Expand Down
64 changes: 64 additions & 0 deletions docs/semantics/c2ir_mapping.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# C To Semantic IR Mapping

Status: first C semantic conversion subset implemented in `semantics/c2ir.py`.
The converter consumes `c_parser` models and emits the same language-neutral
semantic IR used by Fortran and edited `.pyi` files.

## Supported Identity Subset

- C translation unit -> one `SemanticModule` named from the source file stem.
- C function -> `SemanticFunction`, preserving native name and parameter order.
- C parameter -> `SemanticArgument`.
- `void` return -> `None`.
- `_Bool` -> `Bool`.
- `char` -> `Int8` with `c_char_policy` metadata; `signed char` -> `Int8`;
`unsigned char` -> `UInt8`.
- `short`, `int`, `long`, and `long long` map to fixed signed integer names
using the current Linux-oriented defaults: `Int16`, `Int32`, `Int64`,
`Int64`.
- Unsigned integer spellings map to `UInt16`, `UInt32`, `UInt64`, and
`UInt64`; fixed-width typedef spellings such as `uint32_t` map to the
matching `UInt*` fallback.
- `float` -> `Float32`; `double` -> `Float64`.
- `float _Complex` -> `Complex64`; `double _Complex` -> `Complex128`.
- Local typedef chains are resolved when their parser model definitions are
available.
- `size_t` maps to `SizeT` without a target probe; supplied
`x2py.c_type_probe` facts override standard typedefs with width-specific
`Int*`, `UInt*`, or `Float*` semantic names.
- Opaque standard-type probe facts such as `FILE` create named opaque semantic
classes when referenced by converted declarations.
- Object-like numeric macros and enum constants become `Final`-style semantic
variables through the `Constant` constraint.
- Struct definitions become `SemanticClass` entries. Incomplete structs become
opaque classes and may be used through direct `Ptr(...)` identity contracts.
- Declared C arrays, including adjusted array parameters, become semantic array
storage contracts with C order for rank greater than one.
- Pointers become explicit `SemanticStorageContract` pointer/reference
metadata. `const` on the pointee makes the storage read-only, and `restrict`
is preserved as aliasing metadata.

## Conservative Blockers

The converter does not silently invent wrapper policy. It attaches
`readiness_blockers` metadata that the semantic readiness checker reports:

- unresolved typedef or unknown type references;
- macro-dependent declarations in raw C input;
- variadic functions;
- function pointer/callback signatures without edited `.pyi` `Callable`
policy;
- mutable numeric or `void *` pointer parameters without ownership,
scalar-reference, or array policy;
- arrays with unknown extents;
- incomplete structs used by value;
- unions used in semantic signatures;
- `long double`, `volatile`, `_Atomic`, bitfields, and unsupported declarator
compositions.

The current C semantic path supports `--language c --semantics`,
`--language c --wrap-readiness`, and starter exact-contract
`--language c --pyi` output for this supported subset. Generated stubs remain
conservative: ambiguous ownership, callback, ABI-extension, and Pythonic
projection policy stays out of the generated `.pyi` until supplied by the
semantic model or an edited interface.
4 changes: 2 additions & 2 deletions docs/semantics/c_pyi_self_contained_specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,8 +196,8 @@ constraint is written.
not part of the canonical public array annotation unless they produce an
actual storage constraint. In particular, Fortran dummy bounds are
established by native association rather than supplied as Python array
metadata. This does not add C semantic conversion support; C conversion
remains deferred.
metadata. The implemented C conversion subset is described in
[C to semantic IR mapping](c2ir_mapping.md).

Stride-aware dimensions use a slice step marker:

Expand Down
26 changes: 15 additions & 11 deletions docs/semantics/pyi_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ language-neutral: Fortran and future C inputs use the same type, storage,
pointer, array, layout and metadata notation. Source language differences are
represented by contracts and metadata, not by separate syntax families.

This document describes the behavior implemented for the current Fortran path
and the shared notation it establishes for later C semantic conversion. C
semantic conversion and C `.pyi` generation remain deferred.
This document describes the behavior implemented for the current Fortran and C
semantic conversion paths.

## Canonical Type And Storage Contract

Expand Down Expand Up @@ -39,6 +38,10 @@ entries express range or stride contracts (`Float64[1:n]`,
`Float64[::Strided]`, `Float64[:, 0:n:m]`). `Strided` means the runtime stride
is part of the accepted storage contract.

Generic semantic constraints are not represented as type subscriptions.
Constants use `Final[T]`; other constraints and non-dimensional array metadata
use `Annotated[T[...], Constraint, ...]`.

`Annotated[...]` carries non-dimensional metadata:

- `ORDER_F` for a Fortran-oriented multidimensional contract.
Expand Down Expand Up @@ -411,20 +414,21 @@ visible Python values
The projection mechanism is language-neutral. It can later adapt exact
Fortran or C contracts through the same notation and runtime concepts, but
this milestone does not implement automatic Pythonic generation, current
exact-reference adaptation, coercion/contract execution or C semantic
conversion/output.
exact-reference adaptation, coercion/contract execution or C wrapper lowering.
The C frontend can generate starter exact-contract `.pyi` output for the
implemented semantic subset.

## Deferred C Work

The shared model is capable of representing future C functions, variables,
The shared model represents the current C semantic conversion subset for
functions, variables,
fields, constants, scalar references, pointers, arrays with known contracts,
origin metadata, mutability and ownership facts. This task does not implement:
origin metadata, mutability and ownership facts. The C frontend can generate
starter exact-contract stubs from that model. Remaining C work includes:

- `semantics/c2ir.py`;
- C semantic conversion;
- C `.pyi` generation;
- C wrapper lowering;
- C ownership, callback or pointer policy inference.
- C ownership, callback or pointer policy inference beyond facts already
present in exact contracts.

Future C conversion should use the same notation: by-value scalars as bare
types, unrefined pointers as `Ptr(T)` or `Ptr(Const(T))`, and array notation
Expand Down
Loading
Loading