Skip to content

design: agent-visible source access (D107), originals on the navigation path (D108) - #361

Open
fazpu wants to merge 1 commit into
mainfrom
design/source-open-agent-visual
Open

design: agent-visible source access (D107), originals on the navigation path (D108)#361
fazpu wants to merge 1 commit into
mainfrom
design/source-open-agent-visual

Conversation

@fazpu

@fazpu fazpu commented Sep 3, 2026

Copy link
Copy Markdown
Member

Problem

Two gaps, both about the same thing: the system derives text from media well and makes the media itself awkward to reach.

D65 bound how a media source is derived and pointed at, and left the last step — an unmounted agent actually perceiving it — unbuilt. media_design.md §4 named "a source_open primitive" once as an alternative; §8 and retrieval_design.md §3/§7 all selected hydrate depth=bytes, whose bound result is a seekable byte segment. Nothing ever required the result to be something a model can consume.

D51 mounted originals but kept them off the navigation path, reachable only by resolving an explicit pointer. That was right when a harness could do nothing with a video but read its transcript.

D107 — source_open serves perception

source_open(version_id, representation_id?, locator?, accept?), a direct retrieval primitive (D87's four assured operations unchanged) sharing hydrate depth=bytes's resolution path, authorization, and audit. Returns the D49 envelope at evidence grain with perceptual content in the client's native channels, plus a content_manifest[] pairing each block to its role, hash, original/agent_rendition origin, transforms, locator, and untrusted label.

  • No caller-selected view mode. The server decides original-versus-rendition; the result declares which. A caller flag would let an agent pull an unbounded original into its own context.
  • Formats negotiated by caller declaration — MCP has no client capability announcing accepted tool-result MIME types, so accept states it; an empty intersection with the served set is a typed boundary.
  • The guarantee is scoped honestly. Server-side delivery only. MCP leaves host-to-model forwarding undefined, so the audit records that content of a stated kind, size, and hash reached a named principal — never that a model perceived it. End-to-end perception is proved by evaluation (§10 spike 12b). This is weaker than the framing the supporting analysis started with, deliberately.
  • Reads stay side-effect-free. Standard renditions are conversion-time assets; a locator with no stored asset uses the ephemeral transform §4 already binds for clips.
  • Image routing becomes lane emphasis. Both lanes run on every image, the classifier setting budget — a screenshot, chart, slide, and whiteboard photo each carry readable symbols and visual structure, so an exclusive switch turns misclassification into permanently missing evidence.
  • content_manifest[], not parts[] — D87 removed Envelope.parts as envelope-of-envelopes composition. This names one response's own content blocks and composes nothing.

D108 — originals sit on the navigation path

Every original is reachable at <doc_id>/<content_hash>/original.<ext>, one level above the representation directory, so all representations of the same bytes share one copy and a re-conversion never duplicates it. D51's off-navigation-path clause is withdrawn; everything else in D51 stands.

D51 already reached this conclusion and stopped one step short. Its own Context says that for whole-file media "the original is the artifact" and "a transcript is precisely the lossy rendering a multimodal agent needs to bypass"; storage_routing.py encodes the same judgement in code. What it kept off-path was navigation, which it treats as distinct from reachability ("promotion ≠ reachability"). Models that read video, audio, and images directly are now ordinary, so the barrier costs more than it protects — it asks an agent to resolve a pointer to reach the one artifact it handles best, on a filesystem the same design tells it to prefer.

Scope is every original, not only media: one rule is easier to teach a cold agent, and the cost that motivated the split is a storage-class cost, already solved by mime-routed classes.

The read record follows D51's own basis and is unchanged where the object store provides it — a gcsfuse read is a GCS read under Cloud Audit Logs. AuditedRawReader remains the audited path for API/CLI byte fetches. A deployment backing raw with a plain local filesystem does not record browse-path reads; stated here rather than left to surprise an operator.

For a mounted agent, D108 removes the need for source_open entirely — it opens the file. source_open remains the only path for an unmounted agent (most agents against a managed deployment) and the locator-scoped path for large media.

Scope

decisions.md (D107, D108; amendment notes on D51, D65, D38), media_design.md (§1, §2, §4, §4a, §4b, §8, §9, §10), retrieval_design.md (§3, §5, §7, §8), e0_files_design.md (router), media_converter_routes_bound.yaml. No code.

Validation

  • eval-banana validate83 checks valid.
  • git diff --check clean.
  • Codex review, three rounds against this repo's CLAUDE.md rules (10, 9, then 9 findings; each accepted and fixed or recorded below).

Limitations — reviewer decisions

  1. Cross-surface payload encoding is unspecified. Retrieval §7 promises API/CLI/SDK parity; how HTTP/SDK/CLI carry multiple binary parts is not bound here.
  2. agent_view_ready reads "safe-decode admission at ingest", which has no persisted authority or schema yet.
  3. content_manifest[] sits close to what D87 removed. The rationale is in §4a and D107; a reviewer may want D87 amended explicitly.
  4. locomo_benchmark_design.md §6 deliberately untouched. Adding a public operation to that answer catalog forces a new protocol version and breaks score comparability — the benchmark owner's call. The catalog stays true until source_open is implemented and exposed.
  5. D108 needs implementation follow-through: LocalMountPublisher is currently called without raw_root/artifacts_root (selfhost.py:1124-1129), so those views are empty placeholders and raw_uri resolves to nothing on the mount today. The mounts.py module docstring's guardrail 1 also becomes stale.
  6. Managed-offering claims about how originals are accessed are a D5 claim-governance matter in the cloud repo.

Related in-flight work

Decision numbers have collided twice while this sat open (D103→D106→D107). Please assign final numbers at merge.

…ation path (D108)

D65 bound how a media source is derived and pointed at, and left the last
step -- an unmounted agent actually perceiving it -- unbuilt. D51 mounted
originals but kept them off the navigation path. Both gaps are about the
same thing: the system derives text from media well and makes the media
itself awkward to reach.

D107 binds `source_open(version_id, representation_id?, locator?, accept?)`
as a direct retrieval primitive sharing hydrate's resolution path,
authorization, and audit. It returns the D49 envelope at evidence grain with
perceptual content in the client's native channels and a content_manifest[]
pairing each block to its role, hash, original/agent_rendition origin,
transforms, locator, and untrusted label. There is no caller-selected view
mode: the server decides original-versus-rendition and the result declares
which. Formats are negotiated by caller declaration, because no protocol
declares them -- MCP has no client capability announcing accepted
tool-result MIME types.

The guarantee is scoped honestly: it covers server-side delivery only. MCP
leaves host-to-model forwarding undefined, so the audit records that content
of a stated kind, size, and hash reached a named principal, never that a
model perceived it. End-to-end perception is proved by evaluation, bound as
media_design section 10 spike 12b. D107 also replaces D65's exclusive
picture-versus-document discriminator with lane emphasis, and adds four
independent readiness dimensions.

D108 withdraws D51's off-navigation-path clause. Every original is reachable
at `<doc_id>/<content_hash>/original.<ext>`, one level above the
representation directory, so all representations of the same bytes share one
copy and a re-conversion never duplicates it. D51 already reached this
conclusion for media and stopped short: its own Context says the original IS
the artifact and a transcript is the lossy rendering a multimodal agent needs
to bypass, and storage_routing.py encodes that in code. The barrier it kept
was navigational, and models that read video, audio, and images directly are
now ordinary.

The read record follows D51's own basis and is unchanged where the object
store provides it -- a gcsfuse read is a GCS read under Cloud Audit Logs.
AuditedRawReader remains the audited path for API/CLI byte fetches. A
deployment backing raw with a plain local filesystem does not record
browse-path reads; that is stated rather than left to surprise an operator.

For a mounted agent D108 removes the need for source_open entirely. It
remains the only path for an unmounted agent, and the locator-scoped path
for large media.

Also: e0_files_design's router is corrected to match lane emphasis, and the
media_converter_routes_bound acceptance check is amended so a correct
lane-emphasis implementation passes rather than fails it.

Validation: `eval-banana validate` -- 83 checks valid. `git diff --check`
clean. Reviewed by codex over three rounds against this repository's
CLAUDE.md rules.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NXUunrARvBWBteHSUDCYR
@fazpu
fazpu force-pushed the design/source-open-agent-visual branch from 31c5608 to 45c0973 Compare September 3, 2026 13:13
@fazpu fazpu changed the title design: agent-visible source access — source_open serves perception (D106) design: agent-visible source access (D107), originals on the navigation path (D108) Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant