Skip to content

docs #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
saidctb merged 2 commits into from
Jun 4, 2026
Merged

docs #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1b7145c
docs
saidctb Jun 4, 2026
2e9065f
docs
saidctb Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,27 @@ native source

`Wrappable: yes` means the semantic contract has no known readiness blockers.
x2py does not currently generate or compile a runtime wrapper.
The [generated target datatype mapping example](docs/semantics.md#generated-linux-x86_64-mapping-example)
shows how the GitHub Actions C and Fortran scalar types map to NumPy dtypes.

### Fortran

Recognizable Fortran files do not require an explicit language. Parse the
checked basic-subroutine fixture:

Input (`tests/data/fortran/general/basic_subroutine.f90`):

<!-- x2py-doc-source: tests/data/fortran/general/basic_subroutine.f90 -->
```fortran
module m1
contains
subroutine add1(n, x)
integer, intent(in) :: n
real(kind=8), intent(inout), dimension(n) :: x
end subroutine add1
end module m1
```

<!-- x2py-doc-test: exact -->
```bash
python3 -m x2py tests/data/fortran/general/basic_subroutine.f90 --parse
Expand Down Expand Up @@ -105,6 +120,21 @@ python3 -m x2py solver.pyi --wrap-readiness
C inputs require explicit C mode. These commands parse the checked C API
fixture, inspect semantic IR, generate its `.pyi`, and check readiness:

Input (`tests/data/c/general/math_api.h`):

<!-- x2py-doc-source: tests/data/c/general/math_api.h -->
```c
#ifndef X2PY_GENERAL_MATH_API_H
#define X2PY_GENERAL_MATH_API_H

double norm2(int n, const double x[static 1]);
void scale(int n, double alpha, double x[static 1]);
double dot(int n, const double *restrict x, const double *restrict y);
void fill_identity3(double a[static 3][3]);

#endif
```

<!-- x2py-doc-test: exact -->
```bash
python3 -m x2py tests/data/c/general/math_api.h --language c --parse
Expand Down Expand Up @@ -167,6 +197,10 @@ python3 -m x2py include/api.h --language c --parse \
--compiler-arg=--sysroot=/opt/sdk
```

Compiler-backed semantic, `.pyi`, and readiness stages also measure and cache
target datatype facts. C probing covers primitive ABI widths and signedness;
Fortran probing resolves kind expressions and measures intrinsic storage.

C projects can use a compilation database:

```bash
Expand Down
71 changes: 50 additions & 21 deletions docs/c_parser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,9 @@ Implemented:
for direct `.i` input where linemarkers provide it
- optional `preprocessing_recipe` JSON on `CFile` output for compiler streams
generated by the shared x2py CLI
- compiler-derived standard-library ABI probing for `size_t`, `uint32_t`,
`time_t`, and opaque `FILE` handles through `x2py.c_type_probe`
- compiler-derived target ABI probing for every modeled arithmetic primitive,
`size_t`, `uint32_t`, `time_t`, and opaque `FILE` handles through
`x2py.c_type_probe`, with reusable memory and persistent caches
- C directory/file-list discovery for `.c`, `.h`, and direct `.i` inputs in
explicit C mode, while leaving Fortran directory scanning unchanged
- include resolution for quoted includes relative to the current file and
Expand Down Expand Up @@ -265,27 +266,31 @@ entry. Parsed declarations from compiler or direct `.i` input keep mapped
source locations; direct `.i` files also expose `preprocessed_source_path` and
mapped `original_source_paths` where available.

## Standard Type ABI Probe
## C Type ABI Probe

Types introduced by standard headers are not portable primitive aliases.
`size_t`, `uint32_t`, and `time_t` may depend on the compiler target, and
`FILE` should remain an opaque library handle rather than exposing private
library layout. Raw parsing therefore preserves unresolved typedef-name uses
instead of hard-coding an ABI.
C primitive spellings and types introduced by standard headers are target
facts. Plain `char` signedness, `long` width, `long double` representation,
`size_t`, and `time_t` can vary with compiler target and flags. `FILE` should
remain an opaque library handle rather than exposing private library layout.
Raw parsing therefore remains source-faithful instead of embedding an ABI.

For C semantic conversion, `x2py.c_type_probe` compiles and runs a small
C11 query program under an exact compiler and emits target-specific JSON:
For direct compiler-backed C semantic, `.pyi`, and readiness stages, the shared
CLI automatically compiles and runs a small C11 query under the selected
compiler. The standalone command emits the same target-specific report:

```bash
python -m x2py.c_type_probe --compiler /usr/bin/gcc-13 --std c11
python3 -m x2py.c_type_probe --compiler /usr/bin/gcc-13 --std c11
```

The report records arithmetic category, underlying C spelling, bit width, and
alignment for builtin C `int`, `size_t`, available `uint32_t`, and `time_t`; it
records opaque handle and pointer ABI facts for `FILE`. It also retains the
generated C source and exact compile/run commands. Semantic conversion keeps
the name `Int` for builtin C `int` and stores the measured concrete dtype and
probe fact separately.
alignment for all modeled primitive integer, real, and complex types plus
`size_t`, available `uint32_t`, and `time_t`. It records plain `char`
signedness, real mantissa precision and exponent range, and opaque handle and
pointer ABI facts for `FILE`. It also retains the generated C source and exact
compile/run commands. Semantic conversion keeps the name `Int` for builtin C
`int`, stores its measured concrete dtype separately, and maps other primitives
to the measured target width. Unsupported measured widths produce an explicit
semantic readiness blocker.

The probe must be run with the same target profile as the source being parsed.
It carries `-I`, `-D`, `-U`, and `--compiler-arg` options into the compile
Expand All @@ -295,12 +300,36 @@ is retained as provenance; the generated query is compiled as C11 because it
uses C11 `_Generic` and `_Alignof`. If a standard-selection flag affects the
target profile and is compatible with the probe source, pass it through
`--compiler-arg` so it is part of the actual compile command.
The probe does not consume `compile_commands.json` directly; if parser
preprocessing uses a compile database, pass the selected compiler and
target-relevant flags from the matching entry to the probe explicitly.

For cross targets, provide a runner, for example `--runner=qemu-aarch64
--runner=-L --runner=/opt/aarch64-sysroot`.
Automatic results are cached in memory and persistently. The cache key includes
the probe schema/source, resolved compiler binary identity, target flags,
includes, defines, undefines, requested standard, working directory,
target-related compiler environment, and runner executable/arguments. The
default persistent location is `$XDG_CACHE_HOME/x2py/c_type_probe` or
`~/.cache/x2py/c_type_probe`; `X2PY_CACHE_DIR`,
`--c-type-probe-cache-dir`, and standalone `--cache-dir` override it. Use
`--refresh-c-type-probe` on the shared CLI or standalone `--refresh` after an
external target/sysroot change that does not alter the cache key.

The probe does not consume `compile_commands.json` or custom preprocessing
templates directly because one project may contain different target recipes.
Generate a report with the selected compiler and target-relevant flags, then
reuse it during semantic conversion:

```bash
python3 -m x2py.c_type_probe --compiler clang \
--compiler-arg=--target=aarch64-linux-gnu \
--compiler-arg=--sysroot=/opt/aarch64-sysroot \
--runner=qemu-aarch64 --runner=-L --runner=/opt/aarch64-sysroot \
> build/aarch64-c-types.json

python3 -m x2py src/api.c --language c --semantics \
--compile-commands build/compile_commands.json \
--c-type-report build/aarch64-c-types.json
```

For direct shared-CLI cross-target probing, repeat
`--c-type-probe-runner=...` for the runner command and arguments.

The C semantic converter accepts this report as target context. The parser
model remains source-faithful and does not embed host ABI assumptions.
Expand Down
55 changes: 47 additions & 8 deletions docs/developper_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,12 @@ When adding a user example:

`tests/tools/test_documentation_examples.py` executes explicitly marked
`bash` CLI examples and `python` API snippets from `README.md` and Markdown
files under `docs/`. Bash examples must be `python3 -m x2py` commands; the
test replaces `python3` with the active test interpreter and runs them without
a shell. It rejects shell operators, output-writing options, and options that
select custom executables or preprocessing command templates. Python snippets
run with the active test interpreter.
files under `docs/`. Bash examples must be `python3 -m x2py` or
`python3 -m x2py.type_mapping_report` commands; the test replaces `python3`
with the active test interpreter and runs them without a shell. It rejects
shell operators, output-writing options, and options that select custom
executables or preprocessing command templates. Python snippets run with the
active test interpreter.

Mark a command that only needs to exit successfully:

Expand Down Expand Up @@ -116,6 +117,28 @@ mark placeholder commands, snippets that modify the checkout,
environment-dependent compiler recipes, or intentionally failing diagnostic
examples.

When a command reads a checked fixture, include its source input in the user
documentation and verify the displayed source against the fixture:

````markdown
<!-- x2py-doc-source: tests/data/fortran/general/basic_subroutine.f90 -->
```fortran
module m1
...
end module m1
```
````

Append a target profile to an exact marker only for compiler-generated output
that is intentionally architecture-specific:

```markdown
<!-- x2py-doc-test: exact linux-x86_64 -->
```

Off-target checks are skipped. The matching profile must still run the command
and compare its complete output.

Run the documentation checks directly:

```bash
Expand Down Expand Up @@ -157,8 +180,9 @@ implementation files.
| C parse output | `c_parser/parser.py`, `c_parser/models.py`, `c_parser/lexer.py` | `tests/parser/c/test_c_declarations_and_declarators.py`, `tests/parser/c/test_c_fixture_suite.py`, `tests/parser/c/test_c_error_fixture_suite.py` |
| CLI stage selection and output | `x2py/cli.py`, `fortran_parser/cli.py` | `tests/parser/test_cli.py` |
| Compiler preprocessing | `x2py/preprocessing.py` | `tests/parser/test_preprocessing_cli.py`, `tests/parser/test_preprocessor_and_execution_boundaries.py`, `tests/parser/c/test_c_lexer_preprocessor.py` |
| C standard type probing | `x2py/c_type_probe.py` | `tests/parser/test_c_standard_type_probe.py` |
| Fortran type probing | `x2py/fortran_type_probe.py` | `tests/parser/test_fortran_type_probe.py` |
| C target ABI probing and cache | `x2py/c_type_probe.py` | `tests/parser/test_c_standard_type_probe.py` |
| Fortran target type probing and cache | `x2py/fortran_type_probe.py` | `tests/parser/test_fortran_type_probe.py` |
| Generated target datatype mapping examples | `x2py/type_mapping_report.py` | `tests/tools/test_type_mapping_report.py`, `tests/tools/test_documentation_examples.py` |
| Fortran to semantic IR | `semantics/fortran2ir.py`, `semantics/models.py` | `tests/semantics/test_fortran2ir.py` |
| C to semantic IR | `semantics/c2ir.py`, `semantics/models.py` | `tests/semantics/test_c2ir.py`, `tests/semantics/test_c_semantic_readiness.py` |
| `.pyi` printing | `semantics/pyi_printer.py` | `tests/semantics/test_pyi_printer.py`, `tests/semantics/test_pyi_printer_modern_example.py` |
Expand Down Expand Up @@ -199,9 +223,12 @@ When changing `.pyi` syntax:
User-visible datatype names are semantic names, not raw parser spellings.
Mapping happens during parser-to-IR conversion:

- Fortran intrinsic/kind mapping lives in `semantics/fortran2ir.py`.
- Fortran intrinsic/kind mapping and compiler storage-fact application live in
`semantics/fortran2ir.py`.
- C primitive, typedef, and probe-aware mapping lives in `semantics/c2ir.py`.
- The shared dtype names and storage contracts live in `semantics/models.py`.
- Compiler-measured mapping snapshots are generated by
`x2py/type_mapping_report.py`.

When changing datatype mapping:

Expand All @@ -213,6 +240,18 @@ When changing datatype mapping:
4. Update [semantics.md](semantics.md), plus
[tutorial.md](tutorial.md) or [examples.md](examples.md) when the visible
user workflow or examples change.
5. Regenerate and update the exact target mapping snapshots in
[semantics.md](semantics.md). The executable documentation test must match
the complete output of:

```bash
python3 -m x2py.type_mapping_report --language c
python3 -m x2py.type_mapping_report --language fortran
```

For Fortran, keep both modern and legacy spellings in the generated report.
Legacy numeric `type*N` forms carry fixed total storage; compiler-dependent
default, kind, `DOUBLE PRECISION`, and `DOUBLE COMPLEX` forms use probe facts.

### Readiness Internals

Expand Down
114 changes: 114 additions & 0 deletions docs/examples.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,120 @@ The most useful small, checked examples are:
| Generated Fortran semantic interface | `tests/pyi/fixtures/general/modern_pyi_example.pyi` |
| Generated C semantic interface | `tests/pyi/fixtures/c/general/math_api.pyi` |

The core native inputs are included here so the command examples are
self-contained.

### Basic Fortran Input

<!-- x2py-doc-source: tests/data/fortran/general/basic_subroutine.f90 -->
```fortran
module m1
contains
subroutine add1(n, x)
integer, intent(in) :: n
real(kind=8), intent(inout), dimension(n) :: x
end subroutine add1
end module m1
```

### Basic C Input

<!-- x2py-doc-source: tests/data/c/general/math_api.h -->
```c
#ifndef X2PY_GENERAL_MATH_API_H
#define X2PY_GENERAL_MATH_API_H

double norm2(int n, const double x[static 1]);
void scale(int n, double alpha, double x[static 1]);
double dot(int n, const double *restrict x, const double *restrict y);
void fill_identity3(double a[static 3][3]);

#endif
```

### Rich Fortran Input

<details>
<summary>Show <code>tests/data/fortran/general/modern_pyi_example.f90</code></summary>

<!-- x2py-doc-source: tests/data/fortran/general/modern_pyi_example.f90 -->
```fortran
module modern_math_physics
implicit none
private
public :: particle, vector3, counter, init_particle, kinetic_energy, scale_vector, dot3, fill_identity3, normalize_particle

integer :: counter
real(8) :: hidden_scale

type :: particle
integer :: id
real(8) :: mass
real(8), dimension(3) :: position
end type particle

type :: vector3
real(8), dimension(3) :: values
end type vector3

type :: hidden_state
integer :: code
end type hidden_state

contains

subroutine init_particle(p, pid, mass, x, y, z)
type(particle), intent(out) :: p
integer, intent(in) :: pid
real(8), intent(in) :: mass, x, y, z
p%id = pid
p%mass = mass
p%position = [x, y, z]
end subroutine init_particle

function kinetic_energy(p, vx, vy, vz) result(e)
type(particle), intent(in) :: p
real(8), intent(in) :: vx, vy, vz
real(8) :: e
e = 0.5d0 * p%mass * (vx*vx + vy*vy + vz*vz)
end function kinetic_energy

subroutine scale_vector(v, alpha)
real(8), dimension(:), intent(inout) :: v
real(8), intent(in) :: alpha
v = alpha * v
end subroutine scale_vector

function dot3(a, b) result(s)
real(8), dimension(3), intent(in) :: a, b
real(8) :: s
s = a(1)*b(1) + a(2)*b(2) + a(3)*b(3)
end function dot3

subroutine fill_identity3(a)
real(8), dimension(3,3), intent(out) :: a
a = 0.0d0
a(1,1) = 1.0d0
a(2,2) = 1.0d0
a(3,3) = 1.0d0
end subroutine fill_identity3

subroutine normalize_particle(p)
type(particle), intent(inout) :: p
real(8) :: n
n = sqrt(dot3(p%position, p%position))
if (n > 0.0d0) p%position = p%position / n
end subroutine normalize_particle

subroutine hidden_proc(x)
integer, intent(in) :: x
end subroutine hidden_proc

end module modern_math_physics
```

</details>

## CLI Stage Examples

### Parse
Expand Down
Loading
Loading