Design cacheable per-bundle Needs exports and global graph composition

[AlexanderLanin]

Context

The public needs_json target currently runs one Sphinx build over the complete
mounted documentation tree. A change in one docs_bundle() therefore
invalidates a large global action and makes unchanged bundles rebuild their
Needs data.

We should agree on a long-term architecture before extending the current bundle
implementation further. The intended direction is to make a bundle the unit of
ownership and caching for the documentation, implementation links, and
eventually test metadata of a component, feature, or unit.

This issue proposes an architecture for discussion. It does not prescribe the
exact implementation yet.

Goals

• Produce cacheable Needs data per source-bearing docs_bundle().
• Do not invalidate unrelated bundle exports when one bundle changes.
• Keep final docname paths compatible with the mounted documentation tree.
• Preserve global Sphinx-Needs validation instead of replacing it with a
  hand-written JSON merger.
• Keep the existing comprehensive docs_check for checks that require the
  complete source tree or doctrees.
• Avoid mandatory Needs dependencies between bundles.

Why bundles should not depend on every referenced bundle

Needs links naturally cross architectural levels and may run in both
directions. Making these references Bazel dependencies would likely create
cycles between units, components, and features. It would also mean that a
small documentation edit at a higher level invalidates lower-level bundle
exports.

The proposed architecture is therefore a fan-in model:

bundle A sources -> bundle A needs.json --\
bundle B sources -> bundle B needs.json ----> lightweight global Needs build
bundle C sources -> bundle C needs.json --/

Explicit bundle dependencies may remain an option for independently published
subsets, but should not be required by the base model.

Proposed architecture

1. Standalone bundle exports

Every source-bearing docs_bundle() produces a standalone needs.json using
the Sphinx Needs builder.

The action consumes only:

• the bundle's own documentation sources;
• its entry_doc;
• the fixed docs-as-code extension stack and built-in metamodel;
• its direct source-link data;
• bundle-local test metadata when available.

It does not consume parent or sibling sources, final mount_at placement, or
the complete composed bundle.

The export uses a generated minimal Sphinx configuration. Need types, fields,
and link definitions are already part of the docs-as-code toolchain and do not
need to become parameters of every bundle.

2. Lightweight global Sphinx-Needs build

The top-level needs_json target uses a synthetic Sphinx project without the
mounted RST sources. It imports all reachable bundle exports through the
existing needs_external_needs mechanism.

Sphinx-Needs initially represents such imports as external Needs and removes
local source information. Before normal Needs post-processing, a docs-as-code
handler would relocalize only the mounted bundle imports:

• restore intrinsic fields from the original bundle export;
• rebase docname using the bundle's final mount_at;
• restore source location data;
• set is_external to False;
• remove temporary external URL metadata;
• leave backlinks, dead-link flags, and constraint results unset.

Projects imported through docs(data=[...]) remain truly external and are not
relocalized.

3. Let Sphinx-Needs finalize the graph

After relocalization, normal Sphinx-Needs post-processing runs over the complete
graph. In the currently pinned Sphinx-Needs version this includes:

• link and link-condition resolution;
• resetting and regenerating backlinks;
• dead-link calculation;
• constraint processing;
• final Needs sealing and export.

This is preferable to directly merging finalized JSON and manually reproducing
Sphinx-Needs semantics.

4. Keep the full documentation check

The lightweight global Needs build becomes authoritative for the Needs graph,
but cannot replace checks that require mounted source documents.

docs_check remains responsible for:

• RST/MyST parsing;
• roles and document references;
• toctrees and navigation;
• extension behavior requiring doctrees;
• rendering and assets.

Expected cache behavior

For a source change in bundle A:

• bundle A export rebuilds;
• bundle B and C exports remain cached;
• the lightweight global Needs build rebuilds;
• the full docs_check rebuilds only when requested.

For a placement-only change, all bundle exports remain cached and only the
global composition rebuilds.

Semantic assumptions and questions

needextend

Our metamodel restricts needextend to the current document. It can therefore
be applied completely inside the standalone bundle build and cannot mutate a
Need owned by another bundle.

Dynamic functions and variants

Bundle exports contain values after local Sphinx-Needs processing. A dynamic
function that queries Needs outside the bundle may therefore produce a
different result from the monolithic build.

Before adopting the architecture, we need to inventory current usage and
either:

• prove that used functions depend only on local Need/document/configuration
  data;
• reject graph-dependent functions for bundle exports; or
• fall back to exporting pre-postprocessing fragments if this is a real
  requirement.

External import round trip

We need to establish exactly which fields are removed by the external loader
and which must be restored. Globally derived fields must not be restored,
because Sphinx-Needs should recalculate them.

The implementation must also preserve link conditions and Need parts across
the export/import round trip.

Custom metamodels

The normal Needs dialect is built into docs-as-code. We still need to decide
how docs(metamodel=...) behaves:

• validation-only customization can remain a global input;
• customization that changes directives, fields, or links may require a
  compatible bundle build or a monolithic fallback.

Validation model

Bundle builds have only a partial graph. They may suppress narrowly classified
warnings for unresolved cross-bundle links, while parse errors and local checks
remain fatal.

The global Needs build must make Needs-level warnings fatal and run:

• duplicate-ID validation;
• outgoing-link and link-condition validation;
• backlink and dead-link calculation;
• constraints;
• SCORE graph checks;
• Needs-based metrics.

The full docs_check remains the final integration check for all
document-level behavior.

Suggested implementation phases

1. Characterize semantics

   • Inventory dynamic functions, variants, constraints, link conditions, and
     Need parts.
   • Add equivalence fixtures for the features we use.
   • Decide custom-metamodel behavior.

2. Add standalone exports

   • Produce one cacheable needs.json per source bundle.
   • Feed direct source links into the bundle action.
   • Propagate exports and source runtime paths through DocsBundleInfo.
   • Verify that unrelated bundle actions stay cached.

3. Add global composition

   • Import bundle exports as External Needs.
   • Relocalize mounted bundle Needs before post-processing.
   • Run the normal Sphinx-Needs finalization and SCORE graph checks.
   • Preserve the historical public output path.

4. Prove equivalence

   • Compare the new output and diagnostics with the monolithic build.
   • Cover duplicate IDs, missing links, constraints, Need parts, true external
     Needs, source links, and metrics.

5. Switch consumers

   • Make the composed graph the public needs_json.
   • Move metrics_json and traceability_gate only after graph equivalence is
     established.
   • Keep docs_check as the comprehensive integration check.

Alternatives considered

Direct JSON merge

Simple, but derived graph state would have to be manually removed or
recalculated. This risks weakening validation and duplicating Sphinx-Needs
semantics.

Raw pre-postprocessing fragments

Semantically stronger for graph-dependent dynamic functions, but requires a
new stable serialization/import boundary in Sphinx-Needs. This remains the
fallback if finalized bundle exports cannot be composed correctly.

Explicit dependencies between bundles

Provides earlier link checking, but introduces architectural coupling, cache
invalidation, and likely dependency cycles. Keep as an optional model, not the
default.

Decision requested

Do we agree on the following direction?

1. docs_bundle() is the cache unit for local Needs extraction.
2. Bundle exports do not depend on other mounted bundles.
3. The global graph is produced by importing, relocalizing, and normally
   post-processing all bundle exports in a lightweight Sphinx build.
4. docs_check remains the comprehensive document-level integration check.
5. We prototype and verify semantic equivalence before switching public
   consumers.