Merge remote-tracking branch 'origin/master' into claude/pythonbpf-state-replication-g23gij

# Conflicts:
#	tests/test_config.toml

Author: claude

.claude/skills/ir-first-feature/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: ir-first-feature
+description: PythonBPF's development loop for implementing a new compiler feature — write a minimal C eBPF reference, compile it to LLVM IR and read that as the spec, stop for a human syntax decision, then implement against the reference. Use whenever adding or extending a PythonBPF language feature (new statement/expression support, map types, globals, helpers, program constructs).
+---
+
+# The IR-first feature loop
+
+PythonBPF targets LLVM IR via llvmlite. For any new feature, clang's output for the
+equivalent C is the specification — not documentation, not intuition. Follow the loop
+in order; do not skip steps because the feature "looks simple".
+
+## 1. Write the C reference
+
+A minimal `.bpf.c` in `tests/c-form/` exercising **only** the target feature. Small
+enough that every line of the resulting IR is attributable to the feature. Prefer no
+includes (define `SEC` and the `__u*` typedefs by hand) so nothing else pollutes the IR.
+Cover each variant of the feature in one file (e.g. for globals: zero-init, initialized,
+const, const volatile).
+
+## 2. Compile and read the IR — this is the spec
+
+```bash
+clang -target bpf -O2 -g -emit-llvm -S feature.bpf.c -o feature.ll
+llc -march=bpf -filetype=obj feature.ll -o feature.o
+bpftool btf dump file feature.o        # what must come out the far end
+```
+
+Read `feature.ll` and answer, in writing: What top-level symbols/globals appear? What
+do loads/stores/calls look like in the body? What `!DI*` debug metadata exists, and
+what BTF does llc manufacture from it? What did -O2 fold away, and does that folding
+carry semantics (it did for `const` globals)?
+
+Version discipline: llvmlite ≥0.49 emits LLVM 21/22-era attribute spellings
+(`captures(none)`, not `nocapture`). Use a clang/llc generation that accepts them, and
+compare against what `pythonbpf` + the CI's LLVM actually use.
+
+## 3. Diff against current PythonBPF output
+
+Compile the nearest thing PythonBPF can already express and diff the `.ll`s. The delta
+is the actual work item — often smaller than expected (machinery like section placement
+and BTF generation frequently comes free from llc).
+
+## 4. HARD STOP — syntax is a human decision
+
+Present 2–3 Pythonic syntax candidates with trade-offs (declaration site, usage site,
+failure modes, precedents from FastAPI/typing/Triton-style DSLs). **Wait for a human to
+choose. Never proceed on your own judgment, and never treat silence as consent.** The
+maintainers own the language surface.
+
+## 5. Implement against the reference
+
+Emit IR through the existing passes (`globals_pass`, `expr_pass`, `assign_pass`,
+`allocation_pass`, `debuginfo/`). Verify by **diffing your emitted `.ll` against the
+clang reference for the same shapes** — "it compiles and llc accepts it" is not the
+bar; llc accepts plenty of subtly wrong IR.
+
+### House style: lower, don't desugar
+
+Handlers walk the AST the user wrote and emit IR directly. **Do not synthesize new AST
+nodes mid-compilation and feed them back through other handlers** — no
+`ast.Assign(ast.BinOp(...))` conjured to make `x += v` reuse the assignment path.
+
+The temptation is legitimate, so know the argument you are declining. Desugaring is a
+standard compiler move (CPython itself lowers `x += v` this way), it guarantees semantic
+agreement with the composed form, it is the smallest diff, and any later fix to the
+composed path applies automatically. Those are real benefits.
+
+They lose in this codebase for structural reasons: the passes communicate through the
+source tree. Allocation runs before codegen and walks `Assign` — a synthetic `Assign`
+created during codegen is invisible to it, so the two passes silently disagree about
+what the function contains (it happened to be harmless for augmented assignment only
+because that statement never needs a fresh slot; that is luck, not design). Synthetic
+nodes carry no source location, so diagnostics point nowhere. And `ast.dump` in the logs
+shows statements the user never typed, which turns every debugging session into an
+archaeology exercise.
+
+The resolution is to move sharing down a level: **equivalence should come from shared
+value-level helpers, not shared AST.** When two constructs must agree, extract the common
+logic into a helper both call — the way binary-op evaluation and augmented assignment
+both use `apply_binop` for the operator table and `get_operand_value` for operands —
+and let each handler resolve its own target and emit its own store. Two handlers calling
+one helper is the idiom; one handler manufacturing input for another is not.
+
+The operator tables themselves — binary operators, comparisons, and the supported
+unary/boolean operators — live in exactly one place, `expr/operators.py`. A new operator
+is added there first; if it is not in that file, the compiler does not support it.
+
+## 6. Test at the right tier
+
+- Works now → `tests/passing_tests/<category>/`.
+- Documents a gap → `tests/kernel_selftest_equivalent/` with a strict xfail in
+  `tests/test_config.toml` (level `"ir"`, `"llc"`, or `"verifier"`).
+- Wrong-input behaviour → `tests/failing_tests/` with a config entry.
+- Kernel verifier level runs in CI; locally it needs the user's sudo — ask, don't
+  assume.
+
+## House guardrails (always)
+
+- **Never read/cat/grep `vmlinux.py` or `vmlinux.h`** — generated, enormous, will
+  exhaust context. Probe with one-liners:
+  `.venv/bin/python -c "import vmlinux; print(vmlinux.struct_x._fields_[:3])"`
+- Ask the dev which Python binary to use, and remember that for future. The original authors use `.venv/bin/python` as their system python has no llvmlite.
+- Atomic commits, `Core:`/`Tests:` subject prefixes, one logical change each.

README.md

@@ -48,7 +48,7 @@ Dependencies:
 
 * `bpftool`
 * `clang`
-* Python ≥ 3.8
+* Python ≥ 3.10
 
 Install via pip:
 

docs/index.md

@@ -69,6 +69,7 @@ user-guide/maps
 user-guide/structs
 user-guide/compilation
 user-guide/helpers
+user-guide/integers
 ```
 
 ```{toctree}

docs/user-guide/index.md

@@ -44,6 +44,9 @@ PythonBPF uses Python's `ctypes` module for type definitions:
 * `c_void_p` - Void pointers
 * `str(N)` - Fixed-length strings (e.g., `str(16)` for 16-byte string)
 
+Integers follow C's rules for width, sign, conversion and arithmetic; see
+{doc}`integers` for the details and the places where this differs from Python.
+
 ## Example Structure
 
 A typical PythonBPF program follows this structure:

docs/user-guide/integers.md

@@ -0,0 +1,153 @@
+# Integer Semantics
+
+PythonBPF programs are Python syntax, but the integers in them behave as C integers: the
+program runs in the kernel as BPF bytecode, where every value is a fixed-width machine
+word. This page describes the rules the compiler applies. They are C's rules, applied to
+the `ctypes` types you declare, so a program's arithmetic matches what the equivalent C
+program compiled with clang would compute.
+
+```{note}
+This is one of the few places where PythonBPF deliberately differs from Python. Python
+integers have arbitrary precision and no unsigned types; BPF has neither. The
+[divergences from Python](#divergences-from-python) are listed at the end of this page.
+```
+
+## Types
+
+An integer's type is the `ctypes` type it was declared with, and the type carries both
+a width and a sign:
+
+| Signed | Unsigned | Width |
+|---|---|---|
+| `c_int8` | `c_uint8` | 8 |
+| `c_int16` | `c_uint16` | 16 |
+| `c_int32` | `c_uint32` | 32 |
+| `c_int64` | `c_uint64` | 64 |
+
+Every declaration site uses these types: local variables initialised with a constructor
+call, `@bpfglobal` variables, `@struct` fields, map keys and values, and fields read from
+`vmlinux` structures. Helper functions return the type of the kernel's signature, so
+`pid()` and `ktime()` are unsigned while `probe_read`-style helpers return a signed
+`long`.
+
+```python
+count = c_uint32(0)        # a 32-bit unsigned local
+delta = c_int64(-1)        # a 64-bit signed local
+```
+
+A local assigned without a constructor takes its type from the expression:
+
+```python
+now = ktime()              # c_uint64, the helper's return type
+total = count + 1          # the type of the addition (see below), held in a 64-bit slot
+```
+
+Undeclared locals are always 64 bits wide; the inferred type only decides their sign.
+Declare the local with a constructor when a narrower width matters.
+
+### Literals
+
+A literal has the type a C compiler gives it: `int` (32-bit signed) if the value fits,
+`long long` (64-bit signed) otherwise. This matters for mixed arithmetic: in
+`count / -2` with `count` a `c_uint32`, the literal `-2` is a 32-bit `int`, so the
+division happens in `c_uint32` exactly as it would in C.
+
+## Assignment and conversion
+
+Assigning a value to a variable of a different integer type converts it, and the
+variable's declared type is what the stored value *is* afterwards:
+
+* **Widening preserves the value.** The conversion looks at the *source*'s sign: an
+  unsigned source is zero-extended, a signed source is sign-extended. So a `c_uint32`
+  holding `0xFFFFFFFF` stored into a `c_int64` gives `4294967295`, and a `c_int32`
+  holding `-1` stored into a `c_uint64` gives `0xFFFFFFFFFFFFFFFF`. This is what C and
+  `ctypes` both do.
+* **Narrowing truncates.** Only the low bits survive.
+* **Same width reinterprets.** A `c_uint32` `0xFFFFFFFF` stored into a `c_int32` reads
+  as `-1`.
+
+The same rules apply to explicit conversions written as constructor calls
+(`c_int64(count)`), to struct field stores and to `return`.
+
+## Arithmetic
+
+Each binary operation is typed on its own, from its two operands, following C's usual
+arithmetic conversions:
+
+1. Operands narrower than 32 bits are promoted to `c_int32`.
+2. If both operands have the same sign, the result has the wider width and that sign.
+3. If the signs differ, the unsigned type wins when it is at least as wide as the signed
+   one; otherwise the signed type wins.
+
+The operation is then performed in that type, and its result has that type. The variable
+receiving the result plays no part until the final store. Two consequences worth knowing:
+
+* **Intermediate results wrap at their own width.** `c_uint32(0x80000000) * c_uint32(2)`
+  is a `c_uint32` multiplication, so it wraps to `0` before being stored, even if the
+  destination is a `c_uint64`. Widen an operand first if you want a 64-bit product.
+* **Mixed signs go unsigned.** `c_uint32(10) / c_int32(-2)` is an unsigned division by
+  `0xFFFFFFFE`, giving `0`, not `-5`.
+
+The sign of the operation's type selects the instruction for the operations where it
+matters:
+
+| Operator | Signed type | Unsigned type |
+|---|---|---|
+| `/`, `//` | truncating signed division | unsigned division |
+| `%` | remainder with the dividend's sign | unsigned remainder |
+| `>>` | arithmetic shift (sign bit shifts in) | logical shift (zeros shift in) |
+| `<`, `<=`, `>`, `>=` | signed comparison | unsigned comparison |
+
+`+`, `-`, `*`, `<<`, `&`, `|`, `^`, `==` and `!=` produce the same bits for either sign.
+
+Unary minus on an unsigned value follows C too: `-x` is `2^N - x` in the value's type.
+
+## Comparisons
+
+A comparison converts both operands with the same usual arithmetic conversions and then
+compares in the resulting type. `c_uint64(10) > c_int64(-1)` is therefore an unsigned
+comparison in which `-1` is the largest possible value, and the result is false. The
+result of a comparison is `1` or `0`, as in C.
+
+## A verifier gotcha: packet pointer fields
+
+A few context fields are declared as 32-bit integers but are pointers as far as the
+kernel verifier is concerned: `data`, `data_end` and `data_meta` on `xdp_md`, and `data`
+and `data_end` on `__sk_buff`. Because they are `c_uint32`, arithmetic on them directly
+is a 32-bit operation, exactly as in C, and the verifier rejects 32-bit arithmetic on a
+pointer:
+
+```
+R0 32-bit pointer arithmetic prohibited
+```
+
+C programs cast these fields through `(void *)(long)` before using them for the same
+reason. Until PythonBPF does this for you, copy the field into a local first, which is a
+64-bit slot, or cast it with `c_void_p`:
+
+```python
+data = ctx.data            # 64-bit local
+end = ctx.data_end
+if data + 34 < end:        # 64-bit pointer arithmetic, accepted
+    ...
+```
+
+```{note}
+This is a known gap. The plan is to give these fields pointer rank automatically so that
+no cast or copy is needed; this section will go away when that lands.
+```
+
+## Divergences from Python
+
+Because the semantics are C's, some Python behaviour does not carry over:
+
+* `/` is integer division; there is no floating-point result.
+* `//` and `/` are the same operation, and both truncate toward zero: `-7 // 2` is `-3`,
+  where Python gives `-4`.
+* `%` takes the sign of the dividend: `-7 % 2` is `-1`, where Python gives `1`.
+* Integers have a fixed width and wrap on overflow; there is no arbitrary precision.
+* Unsigned types exist, and mixing them with signed values follows C's conversions
+  rather than Python's mathematical integers.
+
+The test programs under `tests/passing_tests/signedness/` show each rule with its
+expected value, and `tests/c-form/signedness.bpf.c` is the equivalent C program.

pyproject.toml

@@ -15,8 +15,6 @@ classifiers = [
     "Intended Audience :: Developers",
     "Operating System :: POSIX :: Linux",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",