Pin only verified state and add explicit HEAD recovery

docs/design.md

@@ -122,7 +122,18 @@ administrative CAS is never automatically rebased.
 - Packs are decrypted segment by segment directly into `git index-pack`
   oldest-first.
 - After import, every advertised ref must resolve to a complete local Git
-  object graph before pins or remote-tracking refs move.
+  object graph before the continuity floor or remote-tracking refs move.
+- Authentication alone never advances the continuity floor. The floor advances
+  after a successful import and connectivity check, after the client's own
+  publication wins compare-and-swap, or during listing when every advertised
+  ref already resolves to a complete local object graph. The last case matters
+  because Git skips the helper's fetch command when nothing needs to be
+  transferred, as after a membership change, a deletion, or a tag pointing at
+  an existing object; without it, such a state would never be pinned and a
+  later replay of the previous state would be accepted.
+- Per-repository/remote client-state updates hold an advisory file lock across
+  validation and read-modify-write. The state replacement remains durable and
+  atomic, and its manifest and policy generations never move backward.
 - The helper advertises `refs/heads/*` and `refs/tags/*`; every other ref
   namespace is rejected before publication.
 - Branch updates require fast-forward ancestry unless force is explicit. Tags
@@ -213,7 +224,23 @@ motivates future garbage collection.
 
 Returning clients pin repository root, manifest ID/generation, and policy
 generation. They reject rollback, non-descendant state, policy rollback, and
-root substitution.
+root substitution. A signed manifest seen by list is not yet a pin: helper
+state advances after all required packs are imported and every advertised ref
+passes the complete-object-graph check, or after the client's own publication
+successfully wins compare-and-swap. A missing or unusable pack therefore cannot
+raise the continuity floor.
+
+Each local repository/remote pair has one stable lock file beside its client
+state. Helpers hold the exclusive lock while checking the current floor and
+performing an import or publication update. State writes merge monotonically
+and use the existing flushed temporary-file, rename, and parent-directory
+sync path. Administrator pins use the same serialized, monotonic state update
+rules after initialization or a successful policy transition.
+
+State files written by earlier builds are not lowered automatically. Their
+head pin may have come from listing, but it is indistinguishable from a valid
+pin made after the client's own publication; lowering it could weaken rollback
+detection.
 
 Storage alone still cannot prevent:
 
@@ -227,6 +254,131 @@ Old manifest headers retain device envelopes. Later compromise of a device
 private key can retroactively expose every generation addressed to that device.
 There is no forward secrecy for stored history.
 
+## Recovering from an unverifiable HEAD
+
+Anyone with storage write access can point `HEAD` at something conforming
+clients reject: bytes that fail to parse, a manifest signed by a reader-only
+device, or a manifest that fails any other check. Clients already refuse such
+a state and stop. Recovery lets an honest writer continue afterwards without
+trusting anything the rejected state claims. It is an explicit operation,
+`git-e2ee recover`; nothing recovers automatically.
+
+### Principle
+
+Recovery never derives a base, refs, policy, keys, or pack inventory from the
+rejected state. The default base is the client's continuity floor: the newest
+state it accepted after a complete connectivity check or its own successful
+publication. A fully validated descendant of the floor may be chosen
+explicitly. Anything else storage offers, including a rejected manifest's
+`previous` pointer, is only a discovery hint and is validated independently.
+
+The danger is an update this client never saw. If a legitimate state `L`
+followed the floor and storage later replaced it, continuing from the floor
+would drop `L`, and if `L` revoked a device, would deliver new keys to that
+device again. The client cannot rule this out from the floor alone, so
+recovery shows the user what it found and never chooses silently.
+
+### Classifying the head
+
+| Class | Meaning | Recovery allowed |
+|---|---|---|
+| valid | authenticated, authorized, continuous with the floor | not needed |
+| invalid | demonstrably bad: unparseable, bad signature, signer not authorized under the authenticated policy, broken audit, commitment, or predecessor link | yes |
+| unverifiable | this device cannot open any envelope | never: a valid revocation looks exactly like this to the revoked device |
+| unsupported | unknown format version or transition type, even if the parser reports it as a parse error | never: an old client must not erase a newer client's state |
+| unavailable | objects missing or a storage error, including failure to read a historical object | never |
+| discontinuous | authenticated but older than, or diverging from, the floor | never: this is detected rollback or equivocation and is reported as such |
+
+These restrictions bind `recover` as well; it cannot override them. A signer ID
+read from a sealed header is reported as the claimed signer until its signature
+verifies against a device record in the authenticated policy.
+
+### `git-e2ee recover`
+
+1. **Discover, read-only.** Classify the head. On the carrier backend, walk the
+   outer commit history back to the floor's manifest within a fixed budget and
+   list every authenticated manifest that descends from the floor, with its
+   generation, outer commit, verified signer, and whether candidates conflict.
+   A replay of an older manifest found in outer history is reported, never
+   treated as newer. Exhausting the budget is an error, not a choice.
+2. **Refuse ambiguity.** If authenticated descendants conflict, report a fork
+   and stop.
+3. **Select explicitly.** The default offer is the newest validated descendant
+   of the floor, or the floor itself if none exists. Choosing an older base
+   than one found is a deliberate history-discarding choice and requires a
+   separate confirmation. On the directory backend, which has no `HEAD`
+   history, only the floor can be offered, with the confirmation: "Updates
+   after this floor may be omitted. This may also omit revocations and
+   disclose subsequently published content to devices excluded by a newer
+   policy."
+4. **Publish.** The device must be an active writer under the base's policy.
+   Build `R` with `R.previous = base`, `R.generation = base.generation + 1`,
+   the predecessor link to the base key, refs and policy from the base, and
+   compare-and-swap from the observed storage token, which is the current outer
+   tip on the carrier. The verified base and the CAS token are separate values.
+   On the carrier the new outer commit is a child of the current tip, so the
+   push is still a fast-forward. Rebuild the `e2ee/` tree from the ciphertext
+   objects the base chain requires, restored byte for byte from the carrier's
+   history or the local cache and checked against their content IDs; if any is
+   missing, fail as unavailable. A lost CAS fails; the operator must start a
+   new `recover` invocation and review its fresh discovery report.
+
+Git does not pass a push of unchanged refs to the helper, so a plain `git push`
+cannot perform recovery; `recover` is the only entry point. It must never
+publish during `list for-push`.
+
+Fresh clones have no floor. Unless they hold an authenticated invitation
+checkpoint, `recover` states that freshness and fork identity are unverified.
+
+### Possible automatic recovery later
+
+Automatic recovery is safe only if the client can prove nothing legitimate
+came between its floor and the junk. Comparing the parent's inner `HEAD` value
+with the floor is not enough: storage writers can append a legitimate `L`, then
+a replay of the floor, then junk, and the junk's parent would still match. A
+future automatic mode would have to anchor to the exact outer commit at which
+this client established its floor, never move that anchor on a replay, and
+require the current tip to be a single-parent child of it.
+
+### Implementation requirements
+
+- Validate every accepted ancestor under the policy in force at that point of
+  the chain; never import policy authority from a rejected manifest.
+- Bound discovery by outer commits, candidates, bytes read, and cryptographic
+  operations, deduplicating repeated IDs.
+- Record rejected heads as bounded diagnostic evidence next to the client
+  state, separate from the floor. Do not blacklist unverifiable or unavailable
+  states.
+- Never lower the floor. A successful recovery advances it to `R`.
+
+### Current implementation
+
+`git-e2ee recover` accepts `--remote <carrier-url>` or `--storage <directory>`,
+plus `--key`, `--repo`, and `--remote-name`. The repository defaults to `.`
+and the remote name defaults to `origin`; the continuity floor is read from
+`.git/git-remote-e2ee/<remote-name>/state.json`. Without `--publish`, the command
+only reports. Publishing an older offer requires `--discard-newer`. Directory
+publication requires `--accept-stale-floor` and prints the warning above.
+
+Discovery scans the available carrier history and refuses any history that
+contains a merge commit, since legitimate publications are always
+single-parent and a merge could hide a legitimate state behind its second
+parent. It works within
+budgets of 2,048 outer commits, 256 distinct heads, 64 MiB of manifest bytes,
+and 1,000,000 estimated cryptographic operations. It scans past replays of the
+floor so a legitimate descendant hidden behind a replay can still be offered.
+An exhausted budget fails closed. A CAS loss is returned to the operator;
+there is no automatic retry with an old selection. This implements the
+restart-at-discovery requirement as a fresh command invocation, matching the
+requirement that a losing recovery fail without retrying blindly.
+
+Before publication, required manifests and packs are restored into the
+disposable carrier checkout, unneeded ciphertext objects are removed from that
+checkout's `e2ee/` tree, and all base-chain packs are imported into a temporary
+bare Git repository to verify complete ref connectivity. The temporary
+repository is deleted when the command exits; the persistent carrier cache
+continues to contain ciphertext Git objects only.
+
 ## Anonymous recipients (v5)
 
 v4 let storage see each device's per-repository public keys, its roles, and

docs/spec.md

@@ -23,7 +23,8 @@ It is designed to satisfy all of the following:
    envelopes scale with the padded reader count; a policy transition also
    carries its encrypted policy body.
 5. A newly admitted reader receives full history by default.
-6. Concurrent publications have exactly one visible winner.
+6. Concurrent publications have exactly one visible winner, provided storage
+   honors the compare-and-swap contract.
 
 The protocol provides confidentiality, authenticity, integrity, and continuity
 from state pinned by the same client. It does not provide global freshness or
@@ -292,8 +293,10 @@ bytes, and verify SHA-256 of the complete stored stream against the descriptor
 ID. A malformed header, size mismatch, counter overflow, short read, reordered
 or duplicated segment, authentication failure, trailing byte, or hash mismatch
 is fatal. Plaintext may stream into `git index-pack` before the final object hash
-is known, but no ref or continuity pin may move until all cryptographic,
-content-address, process, and Git-connectivity checks succeed.
+is known, but no imported-state pin or remote ref may move until all
+cryptographic, content-address, process, and Git-connectivity checks succeed.
+Listing an authenticated manifest does not advance the continuity floor. A
+publisher advances its own floor only after its compare-and-swap succeeds.
 
 Immutable writes MUST be staged within the storage backend and become visible
 under their ciphertext ID only after the complete streamed hash matches that
@@ -402,6 +405,24 @@ refs, pinning the new head, or publishing a successor. A signed ref advance
 whose required pack delta is absent is invalid even when every cryptographic
 check succeeds.
 
+Authenticating a manifest alone MUST NOT advance the continuity floor. A
+helper client's head, generation, repository root, and policy-generation floor
+advances only when the complete-ref connectivity check succeeds for that
+manifest's refs, either after pack import or, during listing, because every
+advertised ref is already connected locally; or after that client successfully
+publishes its own successor with compare-and-swap. A client MUST NOT rely on
+receiving a fetch command to pin a state, because Git omits it when no objects
+need to be transferred. A failed or incomplete
+import leaves the previous floor intact, so an authenticated but unusable
+publication cannot make a later valid successor look like rollback.
+
+Client-state updates for one repository and remote MUST serialize their
+read-modify-write operation across helper processes. The merge MUST preserve
+the greatest recorded manifest and policy generations and MUST reject a
+same-generation head-ID conflict. The durable state-file replacement remains
+atomic; the lock is held while an import or publication checks and advances
+the floor.
+
 A client MAY satisfy that requirement incrementally. After a successful full
 or incremental connectivity check, it records the exact checked ref tips as a
 verified frontier. On a later fetch it first requires every frontier tip to
@@ -417,14 +438,65 @@ client state from another wire version is not accepted. The incremental rule ass
 client's existing object database has not been corrupted outside this
 protocol; explicit full verification remains available to check local storage.
 
+### Explicit recovery
+
+`git-e2ee recover` MUST default to a read-only report. It MUST classify the
+observed storage head as valid, invalid, unverifiable, unsupported, unavailable,
+or discontinuous. Recovery publication is permitted only for an invalid head.
+Unverifiable, unsupported, unavailable, and discontinuous heads MUST never be
+overridden. In particular, a device that cannot open the current envelope
+MUST NOT recover from an older floor, because the head may be a valid
+revocation. A signer ID extracted before successful signature verification
+MUST be labelled as a claim, not as the signer.
+
+On carrier Git, discovery MUST inspect bounded outer history and MUST refuse
+to proceed if that history contains any merge commit: every legitimate
+publication is a single-parent fast-forward, and a merge can hide a legitimate
+state behind a non-first parent. Discovery MUST deduplicate manifest IDs and
+validate candidates against the authenticated policy and predecessor chain. The byte, outer-commit, candidate, and
+cryptographic-operation budgets MUST be fixed. Exhaustion MUST fail closed.
+Discovery MUST report authenticated descendants and replays. If authenticated
+descendants conflict, recovery MUST refuse to choose a fork. A replayed older
+manifest MUST NOT be treated as a newer candidate. Directory storage has no
+outer history and MUST offer only the pinned floor.
+
+The default base MUST be the newest validated descendant of the floor, or the
+floor when no descendant is available. Selecting an older offered base MUST
+require an explicit history-discard confirmation. Directory publication MUST
+require explicit acknowledgement that updates and revocations after the floor
+may be omitted and that subsequently published content may be disclosed to
+devices excluded by a newer policy.
+
+Before publication, the device MUST be an active writer under the selected
+base's policy. Recovery MUST publish a normal signed writer successor whose
+previous ID and generation are the selected base and its generation plus one,
+whose refs and policy are copied from that base, and whose predecessor link
+opens to the base key. It MUST compare-and-swap from the observed storage token
+while keeping that token separate from the selected manifest base. Required
+base-chain ciphertext MUST be available and checked against content IDs. A
+carrier publication MUST restore required historical ciphertext byte-for-byte,
+publish a child of the observed outer tip, and use a normal fast-forward push.
+Before publishing, the selected refs MUST resolve to complete local Git object
+graphs. A lost CAS MUST fail without a blind retry.
+
+Recovery MUST record invalid rejected heads as bounded diagnostic evidence
+separate from client continuity state. That evidence MUST NOT lower the floor
+or make an unverifiable or unavailable head ineligible for a later valid
+operation. A successful recovery advances the floor only after its own CAS
+succeeds. Recovery MUST NOT run from remote-helper `list` or `list for-push`,
+and an ordinary `git push` MUST NOT trigger it.
+
 Force pushes append a normal pack delta and new ref state. Older packs remain
 in historical manifests and may become unreachable in the inner Git graph.
 
 ## 10. Concurrency
 
 The server does not inspect encrypted inner commit parents. Two writers read
 the same opaque `HEAD`, create immutable candidates, and attempt the same CAS.
-Exactly one wins.
+Against storage that honors the CAS contract, exactly one wins. Malicious or
+noncompliant storage can report outcomes inconsistently; that is outside the
+guarantees, and a noncooperating writer to directory storage can also cause an
+ABA change of the compared `HEAD` value.
 
 - Filesystem storage uses locking. `HEAD` is published by rename inside the
   storage root. Immutable objects are hard-linked from `.staging` into
@@ -441,9 +513,14 @@ Unreachable losing objects require later GC.
 
 ## 11. Continuity and invitations
 
-Returning clients pin at least repository root, manifest ID and generation,
-and policy ID and generation. They reject rollback, a non-descendant manifest,
-policy rollback/sideways movement, root substitution, and generation gaps.
+Returning clients pin repository root, manifest ID and generation, and policy
+generation. The policy ID is not stored separately: the pinned manifest ID
+binds it through the manifest's sealed header. Helper continuity pins advance
+only after the complete-ref connectivity check succeeds (after import, or
+during listing when the advertised refs are already connected locally) or
+after a successful own publication, never from authentication alone. They reject rollback, a non-descendant manifest, policy
+rollback/sideways movement, root substitution, and generation gaps. Local
+client-state updates are serialized and monotonic across concurrent helpers.
 
 A fresh device should receive an administrator-authorized invitation checkpoint
 through an authenticated channel containing: