Promote graph context and I/O hardening from dev

.github/workflows/README.md

@@ -1,5 +1,19 @@
 # GitHub Actions Workflows
 
+## Release candidates on TestPyPI
+
+Push an annotated tag such as `v0.2.0rc1` after committing the matching
+`__version__ = "0.2.0rc1"` to `jvspatial/version.py`. The
+`publish-testpypi.yml` workflow verifies the tag/version match, builds
+and validates the package, and uploads it to TestPyPI. Use a new `rcN`
+for each rebuild because published files cannot be replaced.
+
+Configure TestPyPI Trusted Publishing for owner `TrueSelph`, repository
+`jvspatial`, and workflow `publish-testpypi.yml`. The candidate workflow
+does not create GitHub releases or publish to production PyPI. Stable
+publishing remains in `publish.yml` and requires a stable `X.Y.Z`
+version on `main`.
+
 ## PyPI Publishing Workflow
 
 The `publish.yml` workflow automatically publishes the `jvspatial` package to PyPI when code is merged to the `main` branch.
@@ -90,4 +104,3 @@ The workflow uses the version specified in `jvspatial/version.py`. To release a
 - **Publishing fails:** Check the workflow logs for specific error messages
 - **Version conflicts:** Ensure the version in `pyproject.toml` is higher than the current PyPI version
 - **Trusted publishing not working:** Verify the publisher configuration on PyPI matches your repository exactly
-

.github/workflows/benchmarks.yml

@@ -21,6 +21,7 @@ on:
   push:
     branches:
       - main
+      - dev
     paths:
       - 'jvspatial/**'
       - 'tests/benchmarks/**'
@@ -67,19 +68,42 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -e '.[dev,test]'
+          pip install -e '.[dev,test,postgres]'
 
-      - name: Run benchmarks (PR or main push)
+      - name: Run benchmarks (PR or main/dev push)
         env:
           JVSPATIAL_POSTGRES_TEST_DSN: postgresql://jvspatial:jvspatial@localhost:5432/jvspatial
         run: |
           mkdir -p .benchmarks/current
           pytest tests/benchmarks \
             --benchmark-only \
+            --junitxml=.benchmarks/current/tests.xml \
             --benchmark-json=.benchmarks/current/result.json \
             --benchmark-min-rounds=5 \
             --benchmark-warmup=on
 
+      - name: Stamp benchmark methodology
+        run: |
+          python - <<'PYTHON'
+          import json
+          from pathlib import Path
+          path = Path('.benchmarks/current/result.json')
+          report = json.loads(path.read_text())
+          report['jvspatial_methodology'] = 'warm-operation-v2'
+          path.write_text(json.dumps(report))
+          PYTHON
+
+      - name: Require executed PostgreSQL benchmark coverage
+        run: |
+          python - <<'PYTHON'
+          from pathlib import Path
+          import xml.etree.ElementTree as ET
+          cases = [case for case in ET.parse(Path('.benchmarks/current/tests.xml')).iter('testcase')
+                   if 'test_postgres_benchmarks' in case.attrib.get('classname', '')]
+          assert len(cases) == 2, 'Expected both PostgreSQL benchmarks'
+          assert all(case.find('skipped') is None for case in cases), 'PostgreSQL benchmarks skipped'
+          PYTHON
+
       - name: Upload current results
         uses: actions/upload-artifact@v7
         with:
@@ -113,7 +137,7 @@ jobs:
       - name: Download baseline (best-effort)
         id: baseline
         continue-on-error: true
-        uses: dawidd6/action-download-artifact@v25
+        uses: dawidd6/action-download-artifact@v27
         with:
           workflow: benchmarks.yml
           branch: main
@@ -145,6 +169,11 @@ jobs:
           cur = json.loads(Path('.benchmarks/current/result.json').read_text())
           base = json.loads(Path('.benchmarks/baseline/result.json').read_text())
 
+          if cur.get('jvspatial_methodology') != base.get('jvspatial_methodology'):
+              Path('comment.md').write_text('Benchmark methodology changed. Capture a new main baseline before comparing timings.\n')
+              print('::notice::Incompatible benchmark baseline; timing comparison skipped')
+              sys.exit(0)
+
           threshold = float(os.environ.get('REGRESSION_PCT', '25'))
 
           # Build {name: mean_seconds}

.github/workflows/pre-commit.yaml

@@ -5,6 +5,7 @@ on:
   push:
     branches:
       - main
+      - dev
 
 jobs:
   pre-commit-jvspatial:

.github/workflows/publish-testpypi.yml

@@ -0,0 +1,68 @@
+name: Publish release candidate to TestPyPI
+
+on:
+  push:
+    tags:
+      - 'v*rc*'
+
+permissions:
+  contents: read
+
+jobs:
+  publish-testpypi:
+    name: Build and publish release candidate to TestPyPI
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write # PyPI Trusted Publishing (OIDC)
+
+    steps:
+      - name: Check out tagged candidate
+        uses: actions/checkout@v7
+
+      - name: Set up Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: '3.11'
+
+      - name: Verify candidate version matches tag
+        env:
+          GITHUB_REF_NAME: ${{ github.ref_name }}
+        run: |
+          VERSION=$(sed -n 's/^__version__ = "\(.*\)"/\1/p' jvspatial/version.py)
+          if [ -z "$VERSION" ]; then
+            echo "Error: could not read __version__ from jvspatial/version.py"
+            exit 1
+          fi
+          EXPECTED_TAG="v${VERSION}"
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$'; then
+            echo "Error: expected a release candidate version (for example 0.2.0rc1), got $VERSION"
+            exit 1
+          fi
+          if [ "$GITHUB_REF_NAME" != "$EXPECTED_TAG" ]; then
+            echo "Error: tag $GITHUB_REF_NAME does not match version.py ($EXPECTED_TAG)"
+            exit 1
+          fi
+          echo "Publishing candidate $VERSION from $GITHUB_REF_NAME"
+
+      - name: Install build tools
+        run: python -m pip install --upgrade build twine
+
+      - name: Build package
+        run: python -m build
+
+      - name: Validate package
+        run: twine check dist/*
+
+      - name: Publish to TestPyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/
+          repository-url: https://test.pypi.org/legacy/
+
+      - name: Save build artifacts
+        uses: actions/upload-artifact@v7
+        with:
+          name: jvspatial-${{ github.ref_name }}
+          path: dist/
+          retention-days: 30

.github/workflows/security.yml

@@ -13,6 +13,7 @@ on:
   push:
     branches:
       - main
+      - dev
     paths:
       - 'pyproject.toml'
   schedule:

.github/workflows/test-jvspatial.yaml

@@ -5,10 +5,39 @@ on:
   push:
     branches:
       - main
+      - dev
 
 jobs:
   test:
     runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_USER: jvspatial
+          POSTGRES_PASSWORD: jvspatial  # pragma: allowlist secret
+          POSTGRES_DB: jvspatial
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U jvspatial"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+      mongodb:
+        image: mongo:8.0
+        ports:
+          - 27017:27017
+        options: >-
+          --health-cmd "mongosh --quiet --eval 'db.adminCommand({ping:1})'"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+    env:
+      JVSPATIAL_POSTGRES_TEST_DSN: postgresql://jvspatial:jvspatial@localhost:5432/jvspatial  # pragma: allowlist secret
+      JVSPATIAL_PROJECTION_TEST_DSN: postgresql://jvspatial:jvspatial@localhost:5432/jvspatial  # pragma: allowlist secret
+      JVSPATIAL_MONGODB_TEST_URI: mongodb://localhost:27017
+      JVSPATIAL_PROJECTION_MONGO_URI: mongodb://localhost:27017
     strategy:
       fail-fast: false
       matrix:
@@ -22,16 +51,21 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
 
+    - name: Install content-detection dependency
+      run: sudo apt-get update && sudo apt-get install -y libmagic1
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
 
     - name: Install jvspatial package with test, lambda, and postgres dependencies
-      # postgres/pgvector pull in asyncpg so the Postgres *unit* tests run in CI
-      # (translator pushdown, pool auto-tuning, vector encoding). Live-DB
-      # integration tests still skip without JVSPATIAL_POSTGRES_TEST_DSN.
+      # Exercise translators and live isolated PostgreSQL/MongoDB integration
+      # tests, including projections, on every supported matrix Python version.
       run: pip install -e '.[test,lambda,postgres,pgvector]'
 
+    - name: Verify content-based MIME detection is active
+      run: python -c "from jvspatial.storage.security.validator import HAS_MAGIC; assert HAS_MAGIC"
+
     # Coverage on every matrix Python version spikes memory (~1.6k tests); the runner often OOM-kills
     # the process (shown as "The operation was canceled"). Run full suite everywhere; enforce
     # coverage threshold once on 3.12.

AGENTS.md

@@ -92,12 +92,33 @@ OAuth hosts in production must set `JVSPATIAL_OAUTH_KEY_ENCRYPTION_KEY` from a s
 
 The in-memory matcher and PostgreSQL pushdown treat both missing and JSON `null` as absent for `$exists`; MongoDB retains its native field-presence semantics. `graph_transaction` isolates request identity state and invalidates touched entities in the parent cache after commit or rollback. Index setup must be scoped to the database instance so one store cannot suppress another store's indexes. Preserve these contracts when changing query translation, caches, or transactions.
 
+Composite keyset continuation includes every sort value and direction plus ID. Never apply LIMIT before residual filtering or unsupported ordering. A configured fallback scan budget raises on exhaustion rather than returning a partial result. Bulk cache entries require confirmed outcomes; ambiguous writes are invalidated. Preserve JSON path locks/atomic durability and streaming handle cleanup on cancellation.
+
 File-backed SQLite closes its prior connection when rebinding to another event loop; dropping it without `close()` leaks the `aiosqlite` worker thread and can keep the process alive.
 
+### 12. Rich graph context
+
+`ProjectionContextConfig` declarations do not grant access. Intersect profile,
+request, scope and current per-record field grants. Relationship selectors and
+same-record evidence use only selected, authorized, untruncated captured values.
+Emit referenced relationship definitions only; denied intermediaries cannot
+bridge a capture. Preserve format 1 defaults and opt-in format 2. Recorded
+provenance and traversal completeness do not certify truth or decision
+sufficiency. Model execution and application policy stay downstream.
+
 ---
 
 ## Common gotchas
 
+Graph projection is an opt-in read-only extraction/rendering operation for
+language-model context. Preserve explicit host scope, field allowlists, access
+before frontier expansion, closed edge endpoints, resource/detail completeness,
+and immutable payloads. `native` paging limits transferred rows, not physical DB
+scan work. Captures use best-effort reads and new expansions require current
+scope/access again; never present request fingerprints as atomic graph revisions
+or captures as permission caches. Raw projection reads bypass record caches.
+See [graph-projection.md](docs/md/graph-projection.md) and SPEC §6.9.
+
 | Symptom | Cause | Fix |
 |---|---|---|
 | `TypeError: object NoneType can't be used in 'await' expression` | Forgot `await` on async DB call | Add `await` |

CHANGELOG.md

@@ -7,6 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Added a tag-triggered TestPyPI workflow for publishing release candidates independently of stable PyPI releases.
+
+- Optional rich graph-context format 2 via `ProjectionContextConfig`,
+  `ProjectionField`, `ProjectionEvidence`, and `ProjectionRelation`: per-type
+  detail profiles, recorded same-record provenance, qualified relationship rules,
+  explicit missing/null/truncated states, and authorized coverage metadata.
+  Legacy format 1 defaults are preserved; enrichment adds no database operations
+  versus equivalent selected fields. Includes a rooted operational example and
+  paired capture/rendering performance probes.
+
+- Read-only `GraphContext.project_graph` context for language-model inference,
+  with relationship, hierarchy, and ontology views; application-declared type
+  meanings; deterministic evidence paths; and payload-based text/Mermaid rendering.
+- Public `ProjectionRequest`, `ProjectionScope`, `ProjectionAccess`,
+  `ProjectionSemantics`, and immutable `GraphProjection` capture APIs. Explicit
+  scope and field allowlists, batch access decisions, candidate/frontier/detail/
+  byte/time budgets, completeness markers, and fresh authorized expansion requests
+  preserve visibility and distinguish captured results from historical snapshots.
+- Database projection-node batching and incident-edge keyset paging, with native
+  paging on SQLite/PostgreSQL/MongoDB and disclosed compatibility scans elsewhere.
+  Cache wrappers bypass stale record caches for projection reads. Host diagnostics
+  report work without placing inaccessible cardinalities in model context.
+- A runnable dummy supply-chain graph, realistic cross-backend tests, and a
+  reproducible projection/renderer/CRUD scaling recorder.
+- Added opt-in SQL residual scan limits, ID-only paging, optional pager/graph exact counts, and generic BFS priority prefixes.
+- Added native DynamoDB scan pages with query-bound continuation and an opt-in PyMongo Async driver (`driver="pymongo"`; PyMongo >=4.13).
+
+### Fixed
+
+- Restored descending single-field index declarations through GraphContext,
+  repaired existing PostgreSQL B-tree key ordering with serialized transactional
+  index creation across workers, and retained canonical
+  MongoDB/DynamoDB single-field names and schemas alongside SQL sort companions.
+- Local node `$exists` criteria now match the in-memory null/absence contract.
+- Isolated default contexts in GraphContext tests and replaced incomplete PNG
+  fixtures with real content; CI requires libmagic and exercises content-based
+  MIME rejection for disguised unknown binaries.
+
+- Repeated cancellation during file/S3 stream opening, reading, or closing waits for the worker before releasing the reader.
+
+- Source-tree example imports work under both the `pytest` console entry point
+  and `python -m pytest`, including graph projection tests and benchmarks in CI.
+- PostgreSQL benchmark fixtures create and clean their isolated schemas before
+  graph seeding, allowing the live benchmark lane to run on a fresh database.
+- Composite and descending iterators now retain every sort key and ID, including null/missing runs. PostgreSQL sort values retain numeric types and iteration releases its connection before consumer yields.
+- DynamoDB always evaluates residual predicates, including across pages and limits. Exhausted unprocessed batch reads raise instead of appearing absent; batch writes report accurate failed IDs.
+- Cache wrappers forward native iterators and invalidate unsuccessful or uncertain bulk writes.
+- Local/S3 file streams yield incremental chunks with backpressure and close readers on cancellation or early close.
+- SQLite ID filters/sorts use the physical primary key. SQL annotated indexes gain companion indexes matching nullable compound sort expressions and directions.
+
+### Changed
+
+- `find_one` requests one match. JSON scans read in bounded batches, count incrementally and iterate once in ID order. SQL residual scans preserve safe conjunct pushdown and count without retaining full results.
+- `GraphContext.save_batch` uses bounded per-collection native detailed writes with normal context-save preparation and input-order results. Backend exceptions now propagate after cache invalidation; confirmed partial successes remain reportable. Override entity `save()` hooks explicitly when needed; context operations do not invoke them.
+- Graph expansion bounds transferred incident documents and batches frontier node reads/degrees. Context monitoring retains at most 1000 samples per category; percentiles cover that window, while operation counts, means, byte totals and cache counters remain lifetime totals. Iterator logs add active duration separate from consumer time.
+- Query benchmarks exclude seeding/connection setup, require executed PostgreSQL coverage in CI, and skip incompatible baseline methodologies.
+
 ## [0.1.1] - 2026-10-03
 
 ### Fixed

PRD.md

@@ -65,6 +65,7 @@ People building higher-level frameworks that ship a graph model to their own use
 | 6 | **Security defaults that fail closed** | JWT secret required, constant-time secret comparisons, content-based file MIME validation, restrictive CORS, strict CSP on app routes, optional `/docs` gating. | SPEC §15 |
 | 7 | **FastAPI integration without lock-in** | `@endpoint` exposes functions and walkers as routes. The underlying FastAPI app is accessible for middleware, custom routers, and lifecycle hooks. | `jvspatial/api/server.py` |
 | 8 | **Observable by construction** | Structured DB op logs, `MetricsRecorder` protocol, GraphContext `PerformanceMonitor`, OpenTelemetry adapter. | `jvspatial/observability/`, `jvspatial/db/_observable.py` |
+| 9 | **Relationship context for model inference** | Opt-in read-only graph projections render bounded relationship graphs, hierarchies, and captured ontology views with configurable detail profiles, declared relationship meanings, recorded evidence and supporting paths. Applications own inference, scope, and user access. | `jvspatial/core/graph_projection.py` |
 
 ---
 
@@ -153,6 +154,18 @@ The library distinguishes three tiers; [docs/md/stability.md](docs/md/stability.
 
 The PRD ratifies this tiering. Promotion of a feature from experimental to stable, or demotion of an internal helper to public, is a product decision and must be reflected here as well as in `docs/md/stability.md`.
 
+The stable graph projection surface includes `GraphContext.project_graph`,
+`ProjectionRequest`, `ProjectionScope`, `ProjectionAccess`, `ProjectionSemantics`,
+`ProjectionContextConfig`, `ProjectionField`, `ProjectionEvidence`,
+`ProjectionRelation`, and `GraphProjection`. Rich format 2 is explicitly opt-in;
+existing callers retain format 1. Evidence is copied as recorded, and selected
+relationship declarations do not establish truth or decision sufficiency.
+It is a bounded extraction/rendering operation, not a
+general graph query language. It invokes no model provider. Snapshot captures
+advertise best-effort reads; retained historical generations and atomic snapshot
+maintenance are not part of this release. Backend-side paging limits transferred
+candidates without promising bounded physical scan work.
+
 ---
 
 ## 8. Decision Boundaries