Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -4548,12 +4548,12 @@ principal, so D55's identical-bytes no-op can never let a later submitter
rewrite it. The pair travels in `X-Ingest-Principal-*` **headers, never the
query string**, because `external_ref` is erasable PII and a URL is copied
into access logs, proxies and traces. It is honoured **only** when the
composing profile sets `trusted_principal_source`; the deployment-wide
bearer identifies a deployment, not a caller. Untrusted attribution is
**ignored, not rejected** — nothing forged is recorded either way, so a
refusal would buy no safety while letting a metadata concern fail a real
upload. `external_ref` is constrained to printable ASCII because the
header transport cannot carry more, making that an explicit 422 rather
composing profile sets `trusted_principal_source` and, where API authentication
is configured, the presenting credential carries full `write` authority.
Untrusted attribution is **ignored, not rejected** — nothing forged is recorded
either way, so a refusal would buy no safety while letting a metadata concern
fail a real upload. `external_ref` is constrained to printable ASCII because
the header transport cannot carry more, making that an explicit 422 rather
than an encoding crash. Deleting a principal
nulls attribution (`ON DELETE SET NULL`) and never destroys the version.
`IngestedVersion` is unchanged and the `ingested_by` keyword is omitted
Expand All @@ -4571,6 +4571,12 @@ Alembic's autocommit boundary and impossible to pre-build by hand before
the column existed. The later "documents by principal" operation adds the
index in its own revision.

**Amendment (2026-09-03, ingest attribution authority).** Where API
authentication is configured, trusted attribution additionally requires full
`write` authority. The unscoped shared secret remains unrestricted; narrow
signed scopes cannot assert an immutable principal and their attribution
headers are ignored.

**Why.** D50 makes content-level authorization and per-user scoping
library non-goals; that is about **authorization**. Attribution is a
provenance fact, changes no read path, and is the one thing only the
Expand All @@ -4581,12 +4587,13 @@ this decision exists to prevent.
**Consequences.** Erasure here is **row deletion, not a D74-grade
forget**: there is no portable manifest, barrier, residual verification or
restore replay for principals, so this must not be offered as an erasure
guarantee until the follow-up person-grain forget target lands. Within a
trusted perimeter the principal remains caller-asserted; deriving it would
need a per-caller perimeter the engine does not have. Operators must apply
the migration and start the new runtime **before** any client forwards
attribution — an older engine accepts the upload, creates an unattributed
version, and D55 then prevents a retry from repairing it.
guarantee until the follow-up person-grain forget target lands. A full-write
caller within a trusted perimeter still asserts the principal. Narrow
credentials never turn their own subject into attribution implicitly, because
that would conflate the credential holder with the immutable creator claim.
Operators must apply the migration and start the new runtime **before** any
client forwards attribution — an older engine accepts the upload, creates an
unattributed version, and D55 then prevents a retry from repairing it.

**Rejected.** No principal; an opaque string without a kind; inferring a
person from a credential; the principal in the query string; accepting
Expand Down
6 changes: 3 additions & 3 deletions openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -3669,7 +3669,7 @@
},
"/ingest": {
"post": {
"description": "Push one file through E0, optionally as a stable lineage version.\n\nAttribution travels in **headers, never the query string**: the\nreference is erasable PII and a URL is copied verbatim into access\nlogs, proxies and traces, where a later principal deletion cannot\nreach it.\n\nThe pair is honoured only when the composing profile declares its\nperimeter trusted (``trusted_principal_source``). The deployment-wide\nbearer identifies a deployment, not a caller, so elsewhere any client\ncould assert it was a person. Untrusted attribution is **ignored, not\nrejected**: nothing forged is recorded either way, and refusing would\nlet a metadata concern fail an otherwise valid ingest.",
"description": "Push one file through E0, optionally as a stable lineage version.\n\nAttribution travels in **headers, never the query string**: the\nreference is erasable PII and a URL is copied verbatim into access\nlogs, proxies and traces, where a later principal deletion cannot\nreach it.\n\nThe pair is honoured only when the composing profile declares its\nnetwork perimeter trusted (``trusted_principal_source``) and, where API\nauthentication is configured, the presenting credential carries full\nWRITE authority. The unscoped shared secret is unrestricted; a narrow\nsigned ``ingest`` credential may add a document but cannot nominate its\nprincipal. Untrusted attribution is **ignored, not rejected**: nothing\nforged is recorded either way, and refusing would let a metadata\nconcern fail an otherwise valid ingest.",
"operationId": "ingest_document_ingest_post",
"parameters": [
{
Expand Down Expand Up @@ -3790,7 +3790,7 @@
}
},
{
"description": "One of: user | api_credential | service. Sent with X-Ingest-Principal-Ref. Ignored unless the deployment declares a trusted principal source; malformed values are 422 only on a trusted deployment.",
"description": "One of: user | api_credential | service. Sent with X-Ingest-Principal-Ref. Ignored unless the deployment declares a trusted principal source and, where API authentication is configured, the presenting credential carries full write authority; malformed values are 422 only for a trusted assertion.",
"in": "header",
"name": "X-Ingest-Principal-Kind",
"required": false,
Expand All @@ -3803,7 +3803,7 @@
"type": "null"
}
],
"description": "One of: user | api_credential | service. Sent with X-Ingest-Principal-Ref. Ignored unless the deployment declares a trusted principal source; malformed values are 422 only on a trusted deployment.",
"description": "One of: user | api_credential | service. Sent with X-Ingest-Principal-Ref. Ignored unless the deployment declares a trusted principal source and, where API authentication is configured, the presenting credential carries full write authority; malformed values are 422 only for a trusted assertion.",
"title": "X-Ingest-Principal-Kind"
}
},
Expand Down
38 changes: 22 additions & 16 deletions plan/designs/ingest_principal_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,25 +55,31 @@ co-uploader list and no re-attribution path.

The composing profile surfaces this as `trusted_principal_source`
(env **`REMEMBERSTACK_SELFHOST_TRUSTED_PRINCIPAL_SOURCE`** — the self-host
settings class carries the `REMEMBERSTACK_SELFHOST_` prefix), default off. Enable it **only**
where the deployment is reachable solely by a control plane that has
already authenticated the actor it names — a managed data plane behind a
private transit gateway. On a publicly reachable deployment it makes
attribution meaningless rather than merely permissive.

Attribution is honoured only when the composing profile declares
`trusted_principal_source=True`. The deployment-wide bearer identifies a
*deployment*, not a caller, so elsewhere any client could assert it was a
person. Untrusted attribution is **ignored, not rejected**: nothing forged
is recorded either way, so refusing buys no safety while adding a failure
mode where a merely misconfigured deployment rejects real documents.
settings class carries the `REMEMBERSTACK_SELFHOST_` prefix), default off.
Enable it **only** where the deployment's network perimeter has already
authenticated the actor it names, such as a managed data plane behind a private
transit gateway for privileged callers. When the API auth perimeter is
configured, the request must additionally carry full `write` authority before
these headers are believed. The unscoped shared secret is unrestricted and
therefore qualifies; a narrow signed browser `ingest` credential can add a
document but cannot name its immutable principal. Without API auth, enabling
the setting trusts the network perimeter by itself. On an ordinarily reachable
deployment it makes attribution meaningless rather than merely permissive.

Full `write` is the line because such a credential can already write anything
to this deployment, so naming a principal grants it no authority it lacks. A
narrow credential's issuer deliberately withheld everything else, and
attribution must not be the one fact it can still forge. Untrusted attribution
is **ignored, not rejected**: nothing forged is recorded either way, so refusing
buys no safety while adding a failure mode where a merely misconfigured
deployment rejects real documents.
**Metadata must never break ingest.** Default is off, so the self-host
posture is unchanged.

This bounds the trust rather than eliminating it: within a trusted
perimeter the caller is still asserting the principal. Deriving identity
from a per-caller credential would require a per-caller perimeter, which
the engine does not have and this slice does not add.
This bounds the trust rather than eliminating it: a full-write caller within a
trusted perimeter is still asserting the principal. Narrow credentials never
turn their own subject into attribution implicitly; doing so would conflate
the credential holder with the immutable creator claim.

### 2.4 Transport: headers, never the query string

Expand Down
27 changes: 15 additions & 12 deletions src/rememberstack/profiles/selfhost.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,21 @@ class SelfHostSettings(BaseSettings):
managed host sets its published bound here so it is enforced where the
body is actually received."""
trusted_principal_source: bool = False
"""Whether `X-Ingest-Principal-*` on `POST /ingest` is believed (D101).

Env: `REMEMBERSTACK_SELFHOST_TRUSTED_PRINCIPAL_SOURCE` (this settings
class carries the `REMEMBERSTACK_SELFHOST_` prefix).

Off by default. Enable only where the deployment is reachable through a
network perimeter that authenticates the asserted actor, such as a managed
data plane behind a private transit gateway for privileged callers. When
API auth is configured, only full ``write`` authority may supply
attribution: the unscoped shared secret is unrestricted and qualifies,
while a narrow signed credential's attribution headers are ignored even
when this flag is on; the upload itself still succeeds. Without API auth,
this setting trusts the network perimeter by itself. Enabling it on an
ordinarily reachable deployment makes attribution meaningless, not merely
permissive."""
#: Browser origins allowed to call this deployment (D59).
#:
#: Empty by default, which is the self-host answer: nothing is advertised
Expand All @@ -128,18 +143,6 @@ class SelfHostSettings(BaseSettings):
#:
#: Comma-separated in the environment; each must be an exact https origin.
browser_origins: str = ""
"""Whether `X-Ingest-Principal-*` on `POST /ingest` is believed (D101).

Env: `REMEMBERSTACK_SELFHOST_TRUSTED_PRINCIPAL_SOURCE` (this settings
class carries the `REMEMBERSTACK_SELFHOST_` prefix).

Off by default. The perimeter credential authenticates a *deployment*,
not a caller, so on an ordinary or public perimeter any client could
assert it was a person; attribution there is ignored. Set this **only**
when the deployment is reachable solely by a control plane that has
already authenticated the actor it names — e.g. a managed data plane
behind a private transit gateway. Enabling it on a publicly reachable
deployment makes attribution meaningless, not merely permissive."""
conversion_routes: Annotated[dict[str, str], NoDecode] = Field(
default_factory=lambda: dict(STOCK_CONVERSION_ROUTE_NAMES)
)
Expand Down
57 changes: 38 additions & 19 deletions src/rememberstack/surfaces/http_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
from rememberstack.model import SpendLeaseRefused
from rememberstack.model import SpendLeaseUnavailable
from rememberstack.model import ToolDescriptor
from rememberstack.model.auth import PerimeterScope
from rememberstack.ports.auth import AuthPerimeterPort
from rememberstack.surfaces.graph_queries import GraphBusyError
from rememberstack.surfaces.graph_queries import GraphHydrationError
Expand Down Expand Up @@ -354,12 +355,15 @@ def build_api(
control plane for ingest/search/operations POST (D46). Each capability
is explicitly composed; absent services do not pretend to exist.

`trusted_principal_source` declares that this deployment's perimeter is
reached only by a caller entitled to state who ingested a document (a
managed control plane). It is **off by default**: the deployment-wide
bearer identifies a deployment, not a caller, so elsewhere an asserted
`X-Ingest-Principal-*` pair is **ignored, never rejected** — metadata
must not be able to fail an otherwise valid ingest.
`trusted_principal_source` declares that this deployment's network perimeter
is reached only by callers entitled to state who ingested a document. It is
**off by default**, and a configured auth perimeter additionally requires
full WRITE authority before believing the headers. The unscoped shared
secret is unrestricted; a narrow signed browser ingest credential may add
a document, but its attribution headers are ignored and its own subject
never becomes attribution. Elsewhere an asserted `X-Ingest-Principal-*`
pair is **ignored, never rejected** — metadata must not be able to fail an
otherwise valid ingest.

`ingest_body_max_bytes` bounds `POST /ingest` request bodies before they
are buffered (413 over the cap; 411 when no Content-Length is declared).
Expand Down Expand Up @@ -514,6 +518,7 @@ def hydrate_relation(relation_id: UUID) -> Envelope:
deployment_id=deployment_id,
max_body_bytes=ingest_body_max_bytes,
trusted_principal_source=trusted_principal_source,
attribution_requires_write=auth is not None,
)
if ingest_body_max_bytes is not None:
app.add_middleware(_IngestBodyLimit, max_bytes=ingest_body_max_bytes)
Expand Down Expand Up @@ -1133,8 +1138,10 @@ def _parse_ingest_principal(
) -> IngestPrincipal | None:
"""Validate a trusted attribution pair, or raise a 422 explaining why.

Only reached on a trusted perimeter, so a malformed pair here is a real
client error worth reporting rather than metadata that should be dropped.
Only reached on a trusted network perimeter and, where API authentication
is configured, one presenting full WRITE authority. A malformed pair here
is therefore a real client error worth reporting rather than metadata that
should be dropped.
"""
if kind is None and ref is None:
return None
Expand Down Expand Up @@ -1171,12 +1178,14 @@ def _mount_ingest(
ingest: IngestPort,
deployment_id: UUID,
max_body_bytes: int | None,
attribution_requires_write: bool,
trusted_principal_source: bool = False,
) -> None:
"""Add the D62 lineage-aware push surface over the E0 ingest gate."""

@app.post("/ingest", response_model=IngestedVersion)
def ingest_document(
request: Request,
content: Annotated[bytes, Body(media_type="application/octet-stream")],
filename: Annotated[str, Query(min_length=1)],
mime: Annotated[str, Query(min_length=1)],
Expand All @@ -1197,8 +1206,10 @@ def ingest_document(
description=(
"One of: user | api_credential | service. Sent with"
" X-Ingest-Principal-Ref. Ignored unless the deployment"
" declares a trusted principal source; malformed values"
" are 422 only on a trusted deployment."
" declares a trusted principal source and, where API"
" authentication is configured, the presenting credential"
" carries full write authority; malformed values are 422"
" only for a trusted assertion."
),
),
] = None,
Expand All @@ -1221,11 +1232,13 @@ def ingest_document(
reach it.

The pair is honoured only when the composing profile declares its
perimeter trusted (``trusted_principal_source``). The deployment-wide
bearer identifies a deployment, not a caller, so elsewhere any client
could assert it was a person. Untrusted attribution is **ignored, not
rejected**: nothing forged is recorded either way, and refusing would
let a metadata concern fail an otherwise valid ingest.
network perimeter trusted (``trusted_principal_source``) and, where API
authentication is configured, the presenting credential carries full
WRITE authority. The unscoped shared secret is unrestricted; a narrow
signed ``ingest`` credential may add a document but cannot nominate its
principal. Untrusted attribution is **ignored, not rejected**: nothing
forged is recorded either way, and refusing would let a metadata
concern fail an otherwise valid ingest.
"""
if max_body_bytes is not None and len(content) > max_body_bytes:
# the ASGI guard already refused honest requests; this backstop
Expand All @@ -1237,10 +1250,16 @@ def ingest_document(
detail="source_kind and source_ref must be supplied together",
)
# The headers are taken RAW and are not validated at request binding.
# An untrusted deployment must discard them without inspection: if
# malformed attribution could 422, a metadata concern would still fail
# an otherwise valid upload, which is exactly what ignoring is for.
if not trusted_principal_source:
# An untrusted source must discard them without inspection: if malformed
# attribution could 422, a metadata concern would still fail an
# otherwise valid upload, which is exactly what ignoring is for. A
# configured perimeter publishes its context before this handler; only
# full WRITE authority may make the immutable attribution assertion.
context = getattr(request.state, "perimeter_context", None)
may_assert_principal = not attribution_requires_write or (
context is not None and context.scope.covers(required=PerimeterScope.WRITE)
)
if not trusted_principal_source or not may_assert_principal:
principal_kind, principal_ref = None, None
ingested_by = _parse_ingest_principal(kind=principal_kind, ref=principal_ref)
# An old structural IngestPort has no `ingested_by` keyword; passing it
Expand Down
Loading
Loading