feat(schemes): typed recursion schemes as composable optics — the zoo (para/apo/histo/futu, zygo/mutu, fused refolds, M drivers), Graft[Affine] build seam, fused cross - #24
Open
kryptt wants to merge 61 commits into
Conversation
Contributor
|
🚀 Cloudflare Pages preview for https://77621bfa.cats-eo-docs.pages.dev Branch alias: https://feat-typed-recursion-schemes.cats-eo-docs.pages.dev Built from commit |
Contributor
Author
U6 benchmark done (folded into this PR)Wired
Decision: the Table + rationale: Remaining follow-ups: the typed-heap-machine driver (for B/op parity), deriving |
Contributor
Author
Profiling-driven optimization: typed driver now ~1.1–2.2× droste (was ~8–16×)The U6 profiling flagged the B/op (
~7× allocation cut. Bonus wins from dropping
This resolves the U6 decision: the heap-machine driver ships as v1 — no |
This was referenced Jun 9, 2026
kryptt
force-pushed
the
feat/typed-recursion-schemes
branch
from
June 11, 2026 23:41
310be0d to
70497cc
Compare
kryptt
changed the title
kryptt
commented
Jun 12, 2026
kryptt
left a comment
kryptt
left a comment
Contributor
Author
There was a problem hiding this comment.
Still needs some polish
| */ | ||
|
|
||
| /** Fold-scheme citizen: Getter-shaped, carrying the node-supplied algebra for fusion. */ | ||
| final class Cata[F[_], S, A] private[schemes] ( |
Contributor
Author
There was a problem hiding this comment.
This should be it's own file in a 'zoo' package
Contributor
Author
There was a problem hiding this comment.
Done in 19d409c — Cata now lives in its own file schemes/zoo/Cata.scala (new dev.constructive.eo.schemes.zoo package).
| /** Generic effectful-fold citizen: `run: S => M[A]`. What `hyloM` and the fused | ||
| * `AnaM.andThen(CataM)` return. | ||
| */ | ||
| sealed class FoldM[M[_], S, A] private[schemes] (val run: S => M[A]) |
Contributor
Author
There was a problem hiding this comment.
Move to the zoo package and make it public
Contributor
Author
There was a problem hiding this comment.
Done in 19d409c — FoldM is schemes/zoo/FoldM.scala, now public with an open constructor (class FoldM[M[_], S, A](val run: S => M[A])): users can wrap their own S => M[A] as a citizen. CataM/AnaM still extend it; the no-longer-sealed trade-off is noted in its scaladoc.
| def from(d: Forget[M][X, Unit]): Unit = () | ||
|
|
||
| /** Effectful fold-scheme citizen: carries its algebra for fusion. */ | ||
| final class CataM[M[_], F[_], S, A] private[schemes] ( |
Contributor
Author
There was a problem hiding this comment.
Done in 19d409c — schemes/zoo/CataM.scala.
| /** Effectful unfold-scheme citizen: `run: Seed => M[S]`, carrying the coalgebra + instances for | ||
| * fusion. | ||
| */ | ||
| final class AnaM[M[_], F[_], Seed, S] private[schemes] ( |
Contributor
Author
There was a problem hiding this comment.
Done in 19d409c — schemes/zoo/AnaM.scala.
| * the typed [[foldLayered]] machine. | ||
| */ | ||
| private def unfoldFold[N, R](expand: N => PSVec[N], combine: (N, PSVec[R]) => R): N => R = | ||
| private val EmptyAnyRefs: Array[AnyRef] = new Array[AnyRef](0) |
Contributor
Author
There was a problem hiding this comment.
What is this? Who uses this? Is this threadsafe?
Contributor
Author
There was a problem hiding this comment.
Answered in 19d409c, in EmptyAnyRefs' scaladoc (now in Machines.scala): it's the shared zero-length leaf sentinel — childrenArr returns it for every leaf layer so the engines don't allocate a fresh empty array per leaf. Thread-safe by construction: it's never written (every store loop bounds at arr.length, which is 0), so it's an immutable constant shared across all concurrent runs. The new SchemesConcurrencySpec exercises exactly that sharing.
kryptt
added a commit
that referenced
this pull request
Jun 12, 2026
…PR review) Addresses every inline thread on PR #24: - One entity per file in the new schemes.zoo package: Attr, Coattr, Gather, Scatter, Cata, Ana, FoldM, CataM, AnaM (Citizens/CitizensM/ Decorations.scala dissolved). FoldM is now PUBLIC with an open constructor — users can wrap their own S => M[A] as a citizen; CataM/AnaM still extend it (no longer sealed, noted in scaladoc). - Schemes.scala keeps only the factory methods; ALL machinery moved to private[schemes] object Machines (OnStackLimit, EmptyAnyRefs, childrenArr, AscendToken, rebuildLayer(+Paired), foldLayered/Or/M, fusedPairedFold(+M)). Machines' header documents the thread-safety model: every machine allocates its mutable state per invocation (per FORCE on the M path), the only shared values are immutable sentinels. EmptyAnyRefs' scaladoc answers what/who/why-threadsafe directly (the PR question): a zero-length leaf sentinel used by childrenArr in every engine; read-only by construction (store loops bound at arr.length). - The thread-safety claim is now TESTED, both places the review asked: SchemesConcurrencySpec (16 concurrent tasks mixing cata/hylo/ana/ para/histo/futu/cataM[Eval] over shared optics and fixtures) and PlatedConcurrencySpec in tests/ (concurrent Plated transform/universe/ rewrite over a shared structure). Fixed in review of the restructure itself: the concurrency spec's coalgebra guarded at n <= 0, making seed 1 its own child — a non-terminating unfold the engines correctly turn into heap exhaustion. Guard is n <= 1; the suite runs in the default 1GB test heap. All 511 tests green; mdoc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kryptt
changed the title
kryptt
commented
Jun 12, 2026
| // =========================================================================================== | ||
|
|
||
| /** Fold-side decoration optic: gather-only (build-only member). `from` = gather. */ | ||
| type Gather[F[_], W, A] = |
Contributor
Author
There was a problem hiding this comment.
this should be a trait or a class, and avoid the annonymous class creation down below
Contributor
Author
There was a problem hiding this comment.
Done in cf18e41 — Gather is an abstract class extending Optic directly: a decoration implements ONE named method, gather(layer, a): W, and the BiAffine to/from plumbing lives once in the class. The named values are named final classes (Gather.Id, Gather.Histo) — no anonymous instances anywhere (fixtures, benches, and the docs zygo all shrank to one-method extensions).
|
|
||
| /** Unfold-side decoration optic: scatter (`to`, an affine match) + pointed unit (`from` on Step). | ||
| */ | ||
| type Scatter[F[_], W, A] = |
Contributor
Author
There was a problem hiding this comment.
same comment from Gather
Contributor
Author
There was a problem hiding this comment.
Done in cf18e41 — same shape: abstract class with scatter(w): Either[F[W], A] + unit(a): W, named final classes Scatter.Id/Scatter.Futu.
| * execute zero iterations when `arr` is [[EmptyAnyRefs]]. No element is ever written into it. | ||
| * An immutable zero-length array is safe to share across any number of concurrent callers. | ||
| */ | ||
| private[schemes] val EmptyAnyRefs: Array[AnyRef] = new Array[AnyRef](0) |
Contributor
Author
There was a problem hiding this comment.
this needs to be renamed
Contributor
Author
There was a problem hiding this comment.
Renamed in cf18e41: EmptyAnyRefs → NoChildren (the semantic name — a leaf layer's children), scaladoc kept.
| * its combiner closure (stored in the frame on the heap path). Same on-stack/heap hybrid and | ||
| * stack-safety as [[unfoldFold]]. | ||
| def cata[F[_], S, W, A]( | ||
| decor: Gather[F, W, A] |
Contributor
Author
There was a problem hiding this comment.
Done in cf18e41 — decor → gather (and scatter on the ana side).
| def cata[F[_], S, W, A]( | ||
| decor: Gather[F, W, A] | ||
| )(galg: (S, F[W]) => A)(using F: Traverse[F], P: Project[F, S]): Getter[S, A] = | ||
| if decor.asInstanceOf[AnyRef] eq Gather.cata[F, A] then |
Contributor
Author
There was a problem hiding this comment.
lets find a more elegant way to detect this that avoids the asInstanceOf calls
Contributor
Author
There was a problem hiding this comment.
Resolved in cf18e41 by DELETING the dispatch rather than beautifying it: the plain cata(alg) overload now implements directly on the machine (it IS the fast path — nothing to detect), and the generic cata(gather)(galg) always runs the generic route, calling gather.gather(layer, a) directly. Zero asInstanceOf at the seam, zero eq checks, no singletons — and a perf bonus: with the per-node Step wrapper gone the generic route is now byte-identical to the fast path (361,321 vs 361,385 B/op; previously equal only via escape analysis). Agreement stays law-pinned: cata(Gather.cata)(alg) == cata(alg).
| def ana[F[_], A, W, S]( | ||
| decor: Scatter[F, W, A] | ||
| )(gcoalg: A => F[W])(using F: Traverse[F], E: Embed[F, S]): Review[S, A] = | ||
| if decor.asInstanceOf[AnyRef] eq Scatter.ana[F, A] then |
Contributor
Author
There was a problem hiding this comment.
same here, scala3 has more elegant ways of enabling this behaviour
Contributor
Author
There was a problem hiding this comment.
Same resolution in cf18e41 — ana(coalg) direct, ana(scatter)(gcoalg) always generic via scatter.scatter/scatter.unit, zero casts, law-pinned agreement.
| All three run on a single stack-safe, post-order machine (heap-stacked, not JVM-call-stacked), | ||
| so they are safe to depths a hand-written recursion would overflow. The examples below use the | ||
| circe `Plated[Json]` from `cats-eo-circe` as a concrete recursive `S`. | ||
| > An earlier `PSVec`-based untyped path (`cata`/`ana`/`hylo` driven by `Plated`) was |
Contributor
Author
There was a problem hiding this comment.
this can be delegated to the bottom of the file
Contributor
Author
There was a problem hiding this comment.
Done in cf18e41 — the removal note now sits at the bottom of the page under a divider.
| */ | ||
| class FoldM[M[_], S, A](val run: S => M[A]) | ||
| extends Optic[S, Unit, A, Unit, Forget[M]]: | ||
| type X = Nothing |
Contributor
Author
There was a problem hiding this comment.
I see X = Nothing here and in AnaM, CataM while we have Attr and CoAttr that pretty much are exactly the extra info needed on every to / from step. Surely we can better integrate these two ideas.. I do see ATtr, and CoAttr come in for histo and futu morphism. Can you think deeply about this and see if we're actually missing a very beautiful connection between optics and recursion schemes?
Contributor
Author
There was a problem hiding this comment.
You're seeing something real — I've thought it through and written it up as docs/brainstorms/2026-06-12-existential-x-is-the-decoration.md (committed on the branch). The short version:
X = Nothing is a choice, and it's the forgetful one. In eo's encoding X is "whatever to must retain for from to rebuild". The citizens pin X = Nothing because Direct/Forget are the forgetful carriers — a Cata-as-Getter keeps only the answer. Ask instead "what would a non-forgetful fold retain?" and the zoo answers, member by member:
- para: X =
F[(S, A)]— the retained subterms are exactly a lens complement, andfrom= re-embed-them makes get-put hold definitionally. para is a lawful Lens wearing its decoration as X. - histo: X =
Attr[F, A]. And here's the kicker:Attr = νX. A × F[X]— Attr is literally the fixpoint of the Gather optic's per-layer leftover(Unit, F[W]). DuallyCoattr = μX. A + F[X]is the fixpoint of the Scatter side. Attr/Coattr aren't auxiliary data — they're the universal existentials of decorated schemes. The engine's out-array of W's is the X being threaded; the comonadic reading matches (lawful lens = store coalgebra; histo = gcata over cofree = the iterated store). - apo/futu: the same statement on the Prism side — the sum residual.
Sharpest corollary: deforestation is choosing the forgetful existential. Fused ana.cross(cata) vs the materializing composition are the same optic at two X-resolutions (Nothing vs the structure/memo) — the fusion law we pinned is an instance of a general principle.
What it buys if we pursue it (the brainstorm ranks these): para-as-lawful-Lens; memoized incremental refolds (cata.withHistory: X = Attr — only the spine above a change recomputes, lens laws become memo-coherence laws); the hylo fused/materializing choice exposed as an X parameter; and the BiAffine matrix row, where composing decorated schemes = composing their Xs.
Recommendation: not in this PR — it ships the forgetful citizens + the per-layer Gather/Scatter optics, which are exactly the substrate this builds on. I'd slot it as the act after the elgot follow-up: elgot completes the decoration vocabulary, this completes the existential story. (It's also the paper-shaped claim of the whole design: recursion schemes are optics indexed by their existential, and the (co)free (co)monads are the universal indices.)
Contributor
Author
There was a problem hiding this comment.
Yes, this is my intuition that's driving my insistence... I will try to thread this needle further myself. In fact it is the driving intuition for this entire branch and codebase
kryptt
commented
Jun 12, 2026
| * `pending` is consumed only on the [[Ascend]] arm, which is reached only after a real result | ||
| * was threaded in. | ||
| */ | ||
| private val NoResult: AnyRef = new AnyRef |
Contributor
Author
There was a problem hiding this comment.
private object NoResult also works
Contributor
Author
There was a problem hiding this comment.
Done in aa1dd43 — private object NoResult (and it now types the loop state: Pending[R] = R | NoResult.type).
| final private class Frame[F[_], N]( | ||
| val node: N, | ||
| val layer: F[N], | ||
| val arr: Array[AnyRef], |
Contributor
Author
There was a problem hiding this comment.
arr is a bad name, pick a more descriptive name
Contributor
Author
There was a problem hiding this comment.
Done in aa1dd43 — arr → slots (matching the new Slot alias), i → next (slots below next hold results, at/above still hold children — the invariant is in the field's scaladoc).
| // in which case `pending` carries the result to store against the top frame. | ||
| @tailrec def loop(op: AnyRef, pending: AnyRef, stack: List[Frame[F, N]]): R = | ||
| if op ne Ascend then | ||
| val n = op.asInstanceOf[N] |
Contributor
Author
There was a problem hiding this comment.
Extract transparent inline methods here (loop) to make it more readable
Contributor
Author
There was a problem hiding this comment.
Done in aa1dd43 — the loop body delegates to transparent inline def descend(n) / bubble phase helpers; their loop calls are in tail position after inlining so @tailrec still verifies. (One caveat discovered: the WALK itself can't also be inline — nested inline methods are an implementation restriction — so heapWalk is a plain method, which is fine: it only runs past OnStackLimit, the cold path.)
| // Same single-loop sentinel encoding as [[foldLayered]]'s heap walk; the graft arm | ||
| // (`Left`) feeds `pending` directly — finished, by reference. | ||
| @tailrec def loop(op: AnyRef, pending: AnyRef, stack: List[Frame[F, N]]): R = | ||
| if op ne Ascend then |
Contributor
Author
There was a problem hiding this comment.
We need more deduplication with the other loop, this is essentially duplicate
Contributor
Author
There was a problem hiding this comment.
Done in aa1dd43 — the two pure heap loops are now ONE shared heapWalk(root, expandOr, combine): foldLayered instantiates the Or-channel with a constant Right, foldLayeredOr adapts its combine. The hot on-stack recursions stay specialized per engine (that's where the 361k pin lives — verified byte-exact after the dedup), and the M machine is documented as the same walk threaded through tailRecM.
| */ | ||
| type Coalg[N, R] = N => (PSVec[N], PSVec[R] => R) | ||
| def fLayer[F[_], S](using P: Project[F, S], E: Embed[F, S]): Optic[S, S, S, S, Forget[F]] = | ||
| new Optic[S, S, S, S, Forget[F]]: |
Contributor
Author
There was a problem hiding this comment.
Avoid the anonymous class here
Contributor
Author
There was a problem hiding this comment.
Done in aa1dd43 — fLayer returns a named private FLayer[F, S] class; the anonymous Optic is gone.
kryptt
added
help wanted
kryptt
added a commit
that referenced
this pull request
Sep 4, 2026
…PR review) Addresses every inline thread on PR #24: - One entity per file in the new schemes.zoo package: Attr, Coattr, Gather, Scatter, Cata, Ana, FoldM, CataM, AnaM (Citizens/CitizensM/ Decorations.scala dissolved). FoldM is now PUBLIC with an open constructor — users can wrap their own S => M[A] as a citizen; CataM/AnaM still extend it (no longer sealed, noted in scaladoc). - Schemes.scala keeps only the factory methods; ALL machinery moved to private[schemes] object Machines (OnStackLimit, EmptyAnyRefs, childrenArr, AscendToken, rebuildLayer(+Paired), foldLayered/Or/M, fusedPairedFold(+M)). Machines' header documents the thread-safety model: every machine allocates its mutable state per invocation (per FORCE on the M path), the only shared values are immutable sentinels. EmptyAnyRefs' scaladoc answers what/who/why-threadsafe directly (the PR question): a zero-length leaf sentinel used by childrenArr in every engine; read-only by construction (store loops bound at arr.length). - The thread-safety claim is now TESTED, both places the review asked: SchemesConcurrencySpec (16 concurrent tasks mixing cata/hylo/ana/ para/histo/futu/cataM[Eval] over shared optics and fixtures) and PlatedConcurrencySpec in tests/ (concurrent Plated transform/universe/ rewrite over a shared structure). Fixed in review of the restructure itself: the concurrency spec's coalgebra guarded at n <= 0, making seed 1 its own child — a non-terminating unfold the engines correctly turn into heap exhaustion. Guard is n <= 1; the suite runs in the default 1GB test heap. All 511 tests green; mdoc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kryptt
force-pushed
the
feat/typed-recursion-schemes
branch
from
September 4, 2026 13:32
d87956c to
7073562
Compare
Contributor
Benchmark A/BAllocation (B/op) — authoritative
442 more benchmarks
Timing (ns/op) — directional only, same-VM but shared runner
New (head only, not diffed): base_sha: |
kryptt
changed the title
kryptt
added a commit
that referenced
this pull request
Sep 4, 2026
- docs/research/2026-06-15-typed-schemes-bibliography.md: anchor paper (O'Connor's Multiplate, arXiv:1103.2841) + its relevant references and citations, the recursion-schemes canon, and the Scala-ecosystem implementations, each mapped to what PR #24's surface claims. - docs/plans/2026-06-15-001-cleanup-typed-schemes-merge-plan.md: the merge-blocking cleanup (schemes.md teaches the retired 2-arg algebra API and a removed zoo.Gather type — the docs build is the red CI gate) and the non-blocking polish items, sequenced.
…loF) Opt-in, type-safe complement to #23's PSVec-driven Schemes.cata/ana/hylo. The user supplies a pattern functor F[_] + Traverse[F] and hand-writes a Basis (Project[F,S] / Embed[F,S]); algebras then pattern-match F's NAMED constructors instead of indexing an erased PSVec[AnyRef]. - Basis.scala: Project/Embed (+ combined Basis) type classes with coherence laws. - Schemes.{cataF,anaF,hyloF,fLayer}: a cats.Eval trampoline over Traverse[F] (the Plated.rewrite precedent / droste's stack-safe hyloM shape). cataF/hyloF return DirectGetter, anaF returns Review — Direct-carried, so they compose via andThen/cross with NO new core carrier instances. fLayer realizes the project/ embed layer as Optic[S,S,S,S,Forget[F]] (no Optic trait change). - Empirically stack-safe to 10^6 for all three (anaF the OOM frontier), plus a wide-and-deep RoseF; pure hylo law + Project/Embed coherence (BinF and RoseF) under ScalaCheck; cataF cross-checked against #23's Plated cata. #23 untouched. - mdoc docs section + top-of-page caveat pointing at the typed path. The Eval trampoline is O(depth) but allocation-heavy, so the module forks its tests with -Xmx2g (the 10^6 typed + #23 cases together OOM sbt's in-process heap). Deferred follow-ups: JMH B/op vs droste basic (Eval-vs-heap driver decision); deriving Project/Embed from the S<->F correspondence; a pure F[A]=>A overload. Plan: docs/plans/2026-06-09-002-feat-typed-recursion-schemes-plan.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the eoF (typed Eval) rows to SchemesBench alongside eo (PSVec) / droste / hand, and upgrades the bench fixture's Functor[BinF] to a Traverse[BinF] (+ a Basis[BinF, Bin]) so the typed schemes run on the same 2^12 workload. Result (-prof gc, B/op, 8191 nodes): the Eval trampoline is ~8-16x droste basic (cata 15.7x, hylo 7.9x, ana 8.2x) — ~316 B/node of Eval machinery. It MISSES the allocation-parity bar, so it ships as the correct/type-safe/stack-safe v1 with allocation as a documented tradeoff; the explicit typed-heap-machine driver becomes a tracked follow-up (no longer a v1-conditional). droste basic is neither stack-safe nor optic-composable, so the comparison is not apples-to-apples. Documents the table + decision in site/docs/benchmarks.md (new recursion-schemes section) and resolves the Eval-vs-heap Open Question in the plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…chine U6 profiling showed the Eval-trampoline cataF/anaF/hyloF at ~8-16x droste-basic B/op (~316 B/node of Eval machinery). Replaced it with the pre-planned explicit machine: the same <512-on-stack / heap-ArrayDeque hybrid as the PSVec engines, keeping F typed at the algebra seam. - foldLayered: foldLeft reads a layer's children into a per-node array (reused as the result accumulator, folded in place); the deep recursion runs on the machine; map rebuilds the typed F[result] the algebra destructures. - rebuildLayer: leaf layers (0 N-slots) skip the rebuild via a phantom F[N]->F[R] recast — half the nodes in a binary tree. Shared EmptyAnyRefs for leaf arrays. - Any lawful Traverse[F] now works: stack-safety is the machine's, not the user's foldRight, so the Eval-lazy-foldRight caveat is gone and the -Xmx2g test fork is removed (10^6 runs in the default heap, like #23's PSVec cases). B/op vs droste basic (SchemesBench, 8191 nodes): cata 2.2x, hylo 1.1x (parity), ana 1.6x (now beats eo's own PSVec ana) — a ~7x allocation cut from the Eval driver. Residual cata gap is inherent native-Bin-vs-Fix (eo projects a layer per node; droste's unfix is free). Still typed + stack-safe, which droste basic isn't. Updates site/docs/benchmarks.md + schemes.md and resolves the Eval-vs-heap Open Question in the plan (heap machine ships; no follow-up needed for parity). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a worked example + test: compose a lens chain (innerL.andThen(treeL): Lens[Doc, Bin]) to focus a recursive field inside a record, fold it with cataF (wrapping the lens read in a Getter, per eo's read-only-composition idiom), and write the field back through the same composed lens. Demonstrates the "schemes are optics" value prop concretely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fLayer adapts to main's opaque ForgetK carrier (wrap via ForgetK.apply, unwrap via .value; def-encoded to/from per the val→def encoding). DirectGetter references follow the Getter rename. docs/plans/2026-06-11-001: the twice-reviewed plan growing this PR into the recursion-scheme zoo (BiAffine carrier, Decor family, para/apo/ histo/futu, fused cross, M-generic driver). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dation Affine's data shape worn on the build seam: Done = "this slot is already finished — do not call the coalgebra" (payload meaning pinned per optic value via Fst[X]: apo grafts a finished subtree, futu unrolls a prebuilt layer); Step = keep going (focus + one-F-layer leftover via Snd[X]). Carrier-owned instances: ForgetfulFunctor/Fold/Traverse, PartialAccessor, and the new Graft capability (done/step build-channel injection — the vocabulary generic scheme drivers use). No AssociativeFunctor and no Composer bridges: the composition-matrix row is an explicit follow-up. Laws (cats-eo-laws + OpticsLawsSpec): instance laws mirroring AffineLaws plus graft-channel coherence — Done has no focus, folds empty, is map-inert; Step carries exactly its focus. BiAffineSpec pins the citizen-level graft-finality equations (from(Done(w)) == w, to/from round-trips) against a toy full citizen with X = (W, F[W]). Plan: docs/plans/2026-06-11-001 stage 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hand-rolled, droste-style (no cats-free dep, no Eval suspension fields the foldLayered machine never suspends on). Attr = cofree-without- laziness (a result decorating each node's full history); Coattr = free-without-suspension (a seed to expand, or a prebuilt layer unrolled without consulting the coalgebra). histo's O(n) decoration space is documented as inherent, not hidden. Plan: docs/plans/2026-06-11-001 stage 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…naF re-derived
The gather/scatter pair of a generalized scheme, reified as optic values
over the BiAffine carrier, sides pinned in the type:
- DecorGather[F, W, A] = Optic[Unit, W, Unit, A, BiAffine] {X = (Unit, F[W])}
fold side, build-only: from = gather (histo's gather IS the Attr
constructor); vestigial read throws (Unfold.algebra precedent).
- DecorScatter[F, W, A] = Optic[W, W, A, A, BiAffine] {X = (F[W], Unit)}
unfold side, full citizen: to = scatter (Step = call the coalgebra,
Done = prebuilt layer, no call), from on Step = the POINTED UNIT (gana's
pure: ana = id, apo = Right, futu = Coattr.Pure) — unit law
to(from(Step((), a))) == Step((), a).
Named values: Decor.cata/para/histo (gather) + Decor.ana/apo/futu
(scatter). cata/ana/histo/futu are identity-stable singletons; the
generic drivers recognise cata/ana by identity and take the direct
decoration-free engine path, so the re-derived cataF(alg)/anaF(coalg)
are byte-identical in behaviour AND cost. Generic-route honesty
documented: Decor.apo unrolls grafts via Project (distApo, O(graft)) and
Decor.para re-embeds — the native apoF/paraF engines (next stage) avoid
both.
New generic overloads: cataF(decor)(galg) (interior gather∘galg, root
galg alone — droste's gcata shape) and anaF(decor)(gcoalg) (scatter per
slot, root through the unit). Fully typed against the X refinements —
no casts at the decoration seam.
DecorLawsSpec: per-value gather/scatter equations, unit laws, vestigial
throws, identity-stability, fresh-user-id-value vs fast-path agreement
(both sides), histo heads-only == cata, futu two-layers-per-step build.
Plan: docs/plans/2026-06-11-001 stage 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, histoF, futuF paraF: native route — the machine already walks real S nodes and keeps each frame's projected layer, so child slots pair (original subterm, result) positionally via rebuildLayerPaired. No per-node re-embed (droste's Gather.para must reconstruct the subterm it threw away). apoF: native O(1) graft via foldLayeredOr, foldLayered's graft-aware sibling — Left(s) slots are already-finished results, placed BY REFERENCE: never recursed, never projected. (droste's scatter-apo re-walks grafts through project, O(graft) per graft.) histoF/futuF: definitional one-liners on the generic decorated drivers at Decor.histo / Decor.futu — the proof the Decor family is correctly shaped. SchemesZooSpec: degeneration laws (para-ignoring-subterms == cata, never-grafting apo == ana, heads-only histo == cata, single-layer futu == ana), the graft law (grafted subtree present by `eq` reference — the bench-noise-immune form of the O(1) claim), a real course-of-value algebra (grandchildren through history), and stack-safety to 10^6 per member including deep Coattr chains. Plan: docs/plans/2026-06-11-001 stage 4 (zoo half). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Propagates the renames that earlier commits left dangling in docs/comments:
- F-suffix: `eoHyloF` → `eoHylo`, "cataF/anaF benches" → "cata/ana"
- Decor → Gather/Scatter (BiAffine scaladoc, BiAffineLaws, BiAffineSpec,
benchmarks.md decoration-route note)
- benchmarks.md cross rows: `eoCrossFused`/`eoCrossMaterialized` →
`eoRefoldCross`/`eoRefoldManual`, and corrected the now-FALSE "fused cross
beats materialising" claim — post duality-fix `ana.cross(cata)` materialises
(885 577 B/op, == manual), `hylo` is the fusion (361 385); the X-indexed
`proto` spike is what recovers hylo cost through `cross`.
Comment/doc only; tests + mdoc green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o/chrono as fusion
Reset the schemes module around the thesis: a recursion scheme is an Optic
whose existential X is the index of the recursion, and the (co)free (co)monads
are the universal indices.
- New `Scheme[X, A]` carrier (opaque, = A): keeps the (co)algebra alive so the
fused seams are reachable (Direct's collapsed closures cannot fuse).
- Citizens in `schemes.zoo`: Cata (X = Nothing, node-blind F[A] => A),
Ana (X = S), Hylo, Histo (X = Attr = cofree), Futu (X = Coattr = free).
- hylo is no longer a primitive: `ana.cross(cata)` fuses into the zero-S refold
over the Scheme carrier. chrono = `futu.cross(histo)` is hylo at the universal
indices (free -> cofree), also fusing; `Schemes.{hylo,chrono}` need only
Traverse[F] (no Project/Embed) — the compile-time deforestation proof.
- cata is now genuinely node-blind (was paramorphism-flavored `(S, F[A]) => A`).
- Machine fix (type safety, not less): rebuildLayer now runs inside the engine,
so the raw union `Array[Slot[N,R]]` never escapes into concretely-instantiated
code. Algebras receive a typed F[R]. Fixes a latent ClassCastException when
both slot halves are Serializable (Coattr/Attr) — the union erased to
Serializable[] and failed the Object[] checkcast.
- cross is the sole fusion seam (no andThen alias — keeps andThen's
compose-at-focus meaning unambiguous).
FusionSpec/ChronoSpec witness deforestation via an instrumented Basis
(project/embed never called on the fused path). 29/29 green.
Stripped the prior decorated/effectful surface (para/apo/gather/scatter/
cataM/anaM/hyloM) and its specs; they return as follow-ups off this spine.
Out of scope (still on the old API): benchmarks/*Schemes*, site/docs/schemes.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…→unfold dual Completes the 2×2: refold vs metamorphism × trivial vs universal index. - meta = cata.meta(ana) / Schemes.meta: fold an F-recursive S to a neck value A, then unfold a DIFFERENT G-recursive T. The direction-dual of hylo. - metaChrono = histo.meta(futu) / Schemes.metaChrono: the same at the universal indices (cofree fold → free unfold). Degenerates to meta. - New Meta[S, A, T] citizen, X = A (the neck). The non-trivial existential is the point: meta CANNOT fuse — fold over F and unfold over G ≠ F leave no project∘embed to cancel, so the neck is genuinely materialised. Honest mirror of Hylo's X = Nothing. The compile-time tell: meta keeps BOTH Bases where hylo drops them. - meta hosts the seam on the fold (Cata/Histo, fold-first), dual to cross hosting on the unfold (Ana/Futu); distinct name, no andThen collision. MetaSpec (6) uses a genuinely heterogeneous Bin --leafSum--> Int --spine--> Rose (F = BinF, G = RoseF) and witnesses no-fusion via split counting bases: the F-fold's project AND the G-build's embed both fire (contrast hylo: neither). 35/35 green, formatted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e barrier Empirical refutation of the F=:=G fusion hypothesis: a same-functor metamorphism (F = G = BinF, one Basis serving both the fold's Project and the unfold's Embed) STILL calls both project and embed. The functor mismatch was only a sufficient witness for no-fusion; the cause is the scalar neck — the fold's projects (input) are never adjacent to the unfold's embeds (output), so no project∘embed cancels. Contrast hylo, where ana's embed-step and cata's project-step share one layer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three groups off the cata/ana/hylo/histo/futu/chrono/meta spine:
1. Subterm-retaining schemes (X = the optic complement):
- para (Getter, X = F[(S,A)] — store-comonad complement): alg F[(S,A)] => A
sees each child's original subterm, not just its summary. Degenerates to
cata. (X is the writable-Lens complement; get-put is definitional, put-get
needs algebra-coherence, so the lawful put stays a scoped follow-up — shipped
as the sound read.)
- apo (Review, X = Either[S,A] — Prism residual): coalg A => F[Either[S,A]]
grafts a finished subtree by reference (O(1), test pins `eq`). Build-side
dual of para. Degenerates to ana.
2. Short-circuit / seed-reading refolds (both fuse → Hylo):
- elgot (coalg A => Either[B, F[A]] short-circuits) — wires the previously
dead Machines.foldLayeredOr.
- coelgot (alg (A, F[B]) => B reads the seed).
Both degenerate to hylo.
3. Refold-quadrant diagonals (fuse, Traverse-only):
- dyna = ana.cross(histo) (plain unfold → cofree fold)
- codyna = futu.cross(cata) (free unfold → plain fold; descriptive name)
Added as cross overloads on Ana(Histo)/Futu(Cata) + Schemes ctors. Degenerate
to hylo.
para re-projects each node to recover subterms (no engine change, no raw-Slot
escape). ZooExtendedSpec (11): degeneration laws, the distinguishing capability
of each, apo graft-by-reference, dyna/codyna == their seams, 10^6 stack-safety.
47/47 green, formatted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code-review follow-up addressing two challenges: 1. Scheme carrier was redundant. Direct[X,A]=A is value-identical and strictly more capable (it already has Accessor+ReverseAccessor PLUS ForgetfulFold/ Applicative/Traverse + AssociativeFunctor). The original "Direct can't fuse" rationale conflated the Getter/Review *classes* (which store opaque closures) with the Direct *carrier* — fusion comes from the citizens carrying their (co)algebra + cross being a member, which is carrier-independent. Deleted Scheme.scala; all citizens now extend Optic[...,Direct]. Bonus: scheme reads compose with core Getters directly (getter.andThen(cata)) — the .readOnly bridge is gone. 2. One final class + file per scheme (uniform with the primitives). New: Chrono, Dyna, Codyna, Elgot, Coelgot, MetaChrono — each a final class storing its run/refold fn with the machine-wiring in its companion. Schemes is now a thin factory listing of one-line delegations. cross seams return the named type (Ana.cross(Histo)→Dyna, Futu.cross(Histo)→Chrono, Futu.cross(Cata)→ Codyna, Histo.meta(Futu)→MetaChrono). Honesty: the fused-refold classes (Chrono/Dyna/Codyna/Elgot/Coelgot) share X=Nothing — fusion discards the index, so they are nominally-distinct named types, not distinct existentials. Documented as such rather than faking a per-refold X. The genuine indices stay with cata/para/histo/ana/apo/futu/meta. final-class-stores-fn (not an abstract base) is deliberate — matches core Getter/Review and avoids the documented ~1.8x megamorphic-dispatch penalty. Closed two review gaps: a deep (>512) apo graft test exercising the heapWalk Left arm, and a wide/variadic-functor (RoseF) para zip-alignment test. Bounded the 10^6 para+apo test's peak heap (scope each tree) to kill a GC-pressure flake. 49/49 green, formatted. foldLayeredM kept as reserved (per decision). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ecorate Functionally-equivalent cleanup pass (49/49 unchanged): - ReadScheme / BuildScheme base shapes (SchemeShapes.scala) centralize the Direct carrier wrapping (to/from) into ONE place instead of respelling it in all 14 citizens — so the next carrier change is a 2-file edit, not the 14-file churn the Scheme→Direct migration just paid by hand. Each citizen now supplies only its `read`/`write` fn + its `type X`. (The virtual read/write is one dispatch per O(n) fold — immaterial here, unlike core Getter's hot composed reads; documented.) - Coattr.expand extracts the futu unrolling repeated in Futu/Chrono/Codyna/ MetaChrono (4 copies → 1). - Attr.decorate extracts the cofree-decorating combine repeated in Histo/Dyna/ Chrono/MetaChrono (4 copies → 1). - Deleted dead Machines.rebuildLayerPaired (orphaned when the engine moved rebuildLayer inside foldLayered; para re-projects instead). - Dropped now-unused per-citizen imports (Optic/Direct live in the base); tightened run-field visibility to private where same-class-only. Net -87 lines. Also added two review-gap tests last commit (deep apo graft, variadic-functor para alignment) and bounded the 10^6 para+apo test's peak heap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete the recursion-scheme zoo's two index towers and add the orthogonal natural-transformation axis: - zygo / mutu: the comonad-tower auxiliary rung (X = F[(B,A)] / F[(A,B)]); para is zygo at B=S, aux=embed. mutu generalises zygo (mutual recursion). - cozygo / comutu: the build-side g-apo duals (X = Either[B,A] / Either[A,B]). - prepro / postpro: keep the trivial index, decorate the layer optic with an accumulating natural transformation η : F ~> F (O(n·depth); η = id recovers cata/ana). The axis orthogonal to the (co)monad towers. ghylo deliberately omitted — the v2 biaffine-zoo plan rejected free-range generic g-schemes. All wired through Schemes.* factories reusing the shared foldLayered engine; ZooTowersSpec pins each scheme's degeneracy-to-base law + a real-behaviour case + stack-safety. schemes/test now forks with -Xss8m: the 10^6 stack-safety specs take a bounded 512-frame on-stack prefix before the heap walk, which overflows specs2's small-stack parallel pool threads under the heavier suite. Fork keeps the realistic 10^6 tests AND parallel execution rather than shrinking either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t engine The M-class needed no engine work: Machines.foldLayeredM (the Monad[M]-lifted walk, tailRecM-driven, stack-safe, Or-shaped for graft/short-circuit) already existed but had zero callers. This adds only the public surface, with no code explosion: - Two function-wrapping citizens carry the WHOLE M-zoo: FoldM (read side: cataM/paraM/histoM/hyloM/chronoM) and BuildM (build side: anaM/apoM/futuM). The recursion index isn't erased by the consolidation — it rides each citizen's phantom type param XI (cataM = FoldM[..., Nothing], paraM = FoldM[..., F[(S,A)]], ...), so the X-as-index thesis holds per factory. - The pure layer decorations (Attr.decorate, Coattr.expand, the para zip, the apo Either) compose with M for free, so each *M factory is a one-liner over foldLayeredM. - Contract documented on the family: M must be single-pass / linear / sequential (Id, Eval, State, IO) — a branching/replaying M corrupts the engine's mutable walk state. ghylo-style free-range generics stay rejected (v2 biaffine plan); the *M schemes are named, consistent with the rest of the zoo. SchemesMSpec pins each scheme's M=Id agreement with its pure twin, Option short-circuiting (a real effect the pure scheme can't express), and empirical Eval stack-safety folding/building 10^6-deep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…↔Plated bridge
Project/Embed/Basis (the S↔F pattern-functor correspondence) move from the
schemes module into core (optics package), so core's Plated can derive from
them. A top-level `export optics.{Basis, Embed, Project}` in the schemes package
keeps every scheme citizen referring to them unqualified — no per-file churn.
New: Plated.fromBasis[F, S] (using Traverse[F], Project[F,S], Embed[F,S]) — the
same Basis that drives cata/ana now also drives core's Plated recursion
combinators (children/universe/transform/rewrite) and the MultiFocus[PSVec]
carrier. A scheme's single layer IS this self-traversal; Plated is its
non-recursive face. Children are built copy-free via ObjArrBuilder +
PSVec.unsafeWrap (fresh per call, honouring fromChildrenVec's contract).
PlatedBridgeSpec pins it end-to-end: one Basis[BinF, Bin] feeds both
Plated.children/universe/transform AND Schemes.cata, and they agree.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-traversal fLayer was a read-only Forget[F] optic. Re-carrier it on core's MultiFocus[F][X, A] = (X, F[A]) — the Traversal/AlgLens/Grate carrier — with to(s) = ((), project(s)), from((_, fs)) = embed(fs), X = Unit (the F-shape rides inside the foci, so embed needs no extra leftover). Now that the layer optic shares the carrier of Plated.plate and Traversal.each, it composes with the rest of core and gains the write half for free: .foldMap (read foci), .modify / .replace (rewrite immediate children), .modifyA / .all (effectful) — where the Forget spelling was read-only. fLayer is one layer; Plated.fromBasis (Step 1) is its recursive face. SchemesSpec's fLayer cases move to the MultiFocus carrier and add a .modify case proving the read+write upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
para is unconditionally a Getter; this promotes it to a core GetReplaceLens (Optic[S,S,A,A,Tuple2]) so a recursion scheme reads AND writes, and composes with hand-written/derived Lenses on the fused Tuple2 path. The lawfulness boundary, handled honestly: a fold result is not in general a recoverable component of S, so the store-comonad "auto-put" (re-embed the retained subterms, X = F[(S,A)]) makes get-put definitional but leaves put-get conditional on the algebra having a coherent inverse. Rather than ship a Lens that is only conditionally lawful, paraLens takes the coherent put-direction as a parameter — get/enplace then obey the ordinary Lens laws. (The automatic decorated optic stays read-only, a follow-up enrichment of fLayer.) ParaLensSpec pins all three Lens laws on a coherent instance (leftmost-leaf: get = a left-spine paramorphism, enplace = rewrite that leaf) and shows it composing under a core Lens (Box → leftmost leaf). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BiAffine (the build-seam carrier — Done = finished/graft, Step = keep-going) shipped its forgetful instances + Graft but, by design, NOT its composition row: its own scaladoc said "AssociativeFunctor[BiAffine] ... deliberately NOT shipped here". That made it the one carrier that couldn't compose — latent, like foldLayeredM was. This adds AssociativeFunctor[BiAffine], a mechanical build-side mirror of Affine.assoc (Done ↔ Miss, Step ↔ Hit, identical Z), so the generic Optic.andThen now resolves for BiAffine-carried optics: biaffine.andThen( biaffine) type-checks and runs, with Done short-circuiting (a finished outer slot ends the composition) and Step threading the focus through inner and recombining the one-layer contexts. This is the keystone the brainstorm's candidate #4 needs — "composing decorated schemes = composing their Xs" — and the prerequisite for re-carriering the build-side schemes (apo/futu) onto BiAffine so they actually consume it. BiAffineSpec gains a composition section reaching all three arms (outer Done, Step∘Step, Step∘Done) with round-trip + short-circuit checks; CompositionMatrixSpec (121) confirms no resolution regression. Cross-carrier Composer bridges into BiAffine remain follow-up, alongside those scheme citizens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Composer[Tuple2, BiAffine] (tuple2biaffine) and Composer[Either, BiAffine] (either2biaffine), the build-side mirrors of Affine's tuple2affine/either2affine: a Lens lifts to an always-Step; a Prism maps Right→Step, Left→Done. With BiAffine.assoc (same-carrier) these complete BiAffine's composition-matrix row, so lens/prism compose into BiAffine-carried decorations (e.g. the apo scatter). BiAffineSpec gains round-trip checks for both bridges via .morph[BiAffine]; CompositionMatrixSpec (121) and the full aggregate confirm no resolution regression from the new givens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
apo's per-slot residual is now a genuine BiAffine-carried optic. Schemes.apoScatter
(Apo.scatter) = Optic[Either[S,A], Unit, A, Unit, BiAffine]{ X = (S, Unit) }:
Left(s) → Done(s) (the O(1) graft, Fst[X] = S), Right(a) → Step((), a) (keep
unfolding). The X is refined/exposed so Fst[X] reduces at use sites.
Apo's engine now constructs AND consumes its decision through this optic — every
slot goes residual → scatter.to → Done/Step → engine — so apo speaks the carrier
BiAffine was written for. The shared pure foldLayeredOr engine still recurses over
an Either at its boundary (it's shared with elgot/cozygo); the Done/Step decision
collapses onto that boundary at the last step.
As a real Optic[…, BiAffine] value, apoScatter composes via BiAffine.assoc (proven)
and the either2biaffine bridge. ApoScatterSpec pins Done/Step semantics, three
composed arms (Step∘Step, outer-Done short-circuit, inner-Done), and that the
re-carriered scheme still builds with grafts intact; full apo behaviour suite
(ZooExtendedSpec) unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…'s factoring Two surface-preserving dedups the f1a3266 audit didn't reach (it predates the M-family and BiAffine work): 1. M-lifting adapters. The 8 *M factories re-inlined the pure side's layer wiring lifted into M. Mirror it: Coattr.expandM / Attr.decorateM (M-twins of expand/decorate) + three private lifters in Schemes (liftProject / liftCoalg / embedM) for the project/coalg/embed wiring. Each *M factory now reads as terse as its pure twin, and the Right/pure/Either wrapping lives in one place. 2. Machines.buildLayered — foldLayered with the combine fixed to embed, the shape every non-grafting unfold shares. ana/futu/cozygo/comutu and the unfold halves of meta/metaChrono drop their repeated `(_, fr) => E.embed(fr)` (6 sites). No public surface change; SchemesMSpec / ZooSpec / ZooExtendedSpec (96) and the full aggregate (325) pass unchanged. Left the 10 function-wrapping citizens and apoScatter's engine round-trip alone (merging would cost named-type surface / needs the shared-engine re-carrier respectively). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hylo) is gone schemes-laws existed solely to law-check the untyped PSVec hylo fusion (Schemes.hylo/ana/cata), which the typed-recursion-schemes work deletes. The branch predates the module, so the rebase could not remove it; this reconciliation commit does: module directory, build.sbt definition + aggregate + mutationAll entry, CLAUDE.md rows, and the regenerated ci.yml cache paths. Typed-scheme laws live in schemes' own test suite (SchemesFLawsSpec / SchemesFMSpec / zoo laws). A typed-API HyloLaws rewrite is the natural follow-up if the law artifact should return.
- docs/research/2026-06-15-typed-schemes-bibliography.md: anchor paper (O'Connor's Multiplate, arXiv:1103.2841) + its relevant references and citations, the recursion-schemes canon, and the Scala-ecosystem implementations, each mapped to what PR #24's surface claims. - docs/plans/2026-06-15-001-cleanup-typed-schemes-merge-plan.md: the merge-blocking cleanup (schemes.md teaches the retired 2-arg algebra API and a removed zoo.Gather type — the docs build is the red CI gate) and the non-blocking polish items, sequenced.
…aws tree - Schemes.scala / ChronoSpec / ParaLensSpec / SchemesMSpec: fix or inline the Scaladoc links that could not resolve ([[Optic]], [[FusionSpec]], [[Traversal.each]], [[Coattr]]/[[Attr]], [[Lens]], [[Machines.foldLayeredM]]). - schemes-laws/: remove the empty untracked directory tree (can be recreated when the D5 discipline law specs get their own module).
The page taught the retired surface — node-supplied 2-arg algebras ((node, folded) => ...), a removed zoo.Gather type, 4-type-arg cata, Forget[M]-carried FoldM consumed via .run — which failed the docs build (10 mdoc errors, the red CI gate). Rewritten against the shipped API: - algebras are node-blind 1-arg (F[A] => A / F[(S, A)] => A); - zygo shown as the named citizen constructor Schemes.zygo[...](aux)(alg); - the M family reads via .get / .reverseGet yielding M[...], hyloM the fused effectful spelling (no Kleisli andThen exists on the citizens); - fLayer rides MultiFocus[F] (read+write), not Forget[F]; - the zoo table drops the Gather/Scatter vocabulary (named citizens on the BiAffine carrier; assoc + cross-carrier bridges shipped); - the BiAffine section no longer claims the matrix row and elgot/coelgot are follow-ups — both shipped; - lens-composition example uses the unambiguous spelling: Getter.andThen is 3-way ambiguous for Direct-carried citizens (recorded as cleanup C8). sbt scalafmtCheckAll scalafmtSbtCheck docs/mdoc docs/laikaSite all green.
…ography pointers (C2) - CHANGELOG [Unreleased]/Added: cats-eo-schemes (the typed zoo, fusion, M-family, paraLens, the removal of the untyped path), core BiAffine (+ assoc matrix row, cross-carrier bridges, Graft, apoScatter), and Basis-in-core with the Plated.fromBasis derivation. - plans 2026-06-09-002 / 2026-06-11-001: point their reference sections at the new bibliography so the branch's citations live in one place.
…d citizens; drop the BiAffine clone C8 — Getter.andThen overload tie: A Direct-carried read-only citizen (the schemes' cata/ana/hylo — Optic[A, Unit, C, Unit, Direct]) matched three Getter andThen overloads at once (the any-carrier member, the re-homed read-only override, the trait's same-carrier inline andThen); dotty called the ranking a three-way draw and getter.andThen(cata) did not compile. Fixed with one new most-specific member on Getter: andThen[C, D](Optic[A, Unit, C, Unit, Direct])(using DummyImplicit): Getter[S, C] — the DummyImplicit makes its parameter-list shape comparable with the other overloads so the strictly-more-specific parameter type wins outright, returning the concrete Getter so ascribed compositions type-check. Pinned by GetterAndThenResolutionSpec (fused-Getter route, other-carrier PickFold route, citizen route); the full CompositionMatrixSpec still passes. BiAffine dropped (review comment from kryptt): data/BiAffine.scala was a field-identical clone of Affine (Done/Step ↔ Miss/Hit) whose map/fold/traverse/assoc/composer instances mirrored Affine's line for line. The build-seam reading of the arms — Miss = finished slot (no coalgebra call), Hit = keep going — is the decoration vocabulary, so it lives on Affine directly: - new Graft[Affine] instance (done = Miss, step = Hit), the injection vocabulary the schemes' build-side citizens construct and consume; - graft-finality laws folded into AffineLaws/AffineTests (finished arm is focus-free, map-inert, folds empty; step carries its focus); - Apo.scatter / Schemes.apoScatter re-carriered onto Affine; - tests: BiAffineSpec -> AffineBuildSeamSpec, ApoScatterSpec updated; - BiAffineLaws/BiAffineTests deleted (folded into the Affine rule set); - docs + CHANGELOG updated; the BiAffine name is left free for a genuinely two-sided-failure carrier if one is ever needed. All gates: root/test 536 green, scalafmt/scalafix clean, mdoc 0 errors.
…catch); re-pin benchmarks at the merge candidate C7's JMH re-pin on temurin@21 caught a real regression the dedup audit (f1a3266) introduced: Para's combine re-projected each node and materialized its child layer into a List per node (F.toList) — 1 409 788 B/op vs the pinned 557 945, pushing para past droste (1.26x). Fixed by restoring the positional pairing as Machines.foldLayeredSlot (slot-threading sibling of foldLayered whose combine receives the machine's expanded layer + filled slot buffer; heapWalkSlot is the cold-path twin) and rebuildLayerPaired over them. Para now pairs off the layer the machine already holds — no re-project, no per-node List: 557 947 B/op, back to the pinned 0.50x of droste. Also caught by the re-pin: - eoRefoldCross now FUSES: 361 387 B/op, byte-identical to hylo (the C8 overload fix routes ana.cross(cata) to the fused composite — decision 9's design intent; FusionSpec's no-intermediate-S witness covers it). The materializing contrast is the hand-written eoRefoldManual (885 589). - benchmarks module had drifted off the shipped API (2-arg algebras, removed Gather/Scatter vocabulary, removed proto package): fixed SchemesBench/fixtures to 1-arg node-blind algebras, dropped the generic-route row with the vocabulary, deleted ProtoFusionBench (proto spike carrier is gone). site/docs/benchmarks.md tables re-pinned to the merge-candidate sweep (temurin@21, -f 1 -i 5 -wi 3); the two changed rows and their bullets re-scoped (para regression + fix noted; cross-fusion bullet rewritten; droste histo/futu re-measured).
…ed upcast, formatting The concurrent force-push rebased the branch onto a newer main (#26, Unfold): Miss dropped its second type parameter (Miss[A] extends Affine[A, Nothing], re-typing subsumed by subtyping, widenB gone). Adapts the C8/BiAffine-drop commit's touched files to that arity — Graft[Affine] constructors, Apo.scatter, the spec toys (widenB check rewritten as the allocation-free upcast it now is) — plus scalafmt on the drifted files.
…t the phantom ns regression The C7 fix restored para's positional subterm pairing through a slot-level engine entry that handed the raw Array[Slot[N, R]] to the combine - contradicting Machines' documented invariant (the raw Slot buffer never leaves the engine) - and Para's own scaladoc still described the audited re-project route. - Machines: new `foldLayeredPaired` - the typed sibling of `foldLayered`, whose combine receives the node's layer with each child paired with its folded result (`F[(N, R)]`): para's shape, with no re-project and no per-node List. `foldLayeredSlot` and `rebuildLayerPaired` drop to `private`, so the slot buffer stays inside the file and the two `foldLayered*` drivers are the typed surface. Same walk, same per-node work - the pinned para B/op is unchanged by construction. - Para: wired to `foldLayeredPaired`; scaladoc states the actual route. - docs(benchmarks): the zoo table drops the ns/op column - B/op is the gate metric, and the mixed single-fork ns column showed a para "time regression" that focused runs do not reproduce. Stale results rewritten to the re-pinned numbers: para 0.50x, apo 0.68x, histo 1.54x, futu 1.25x, graft 280 vs 240, and `ana.cross(cata)` fusing (byte-identical to `hylo`) with the materialising spelling named as the manual pair. The Gather/Scatter generic-route bullet and the removed `proto` spike reference are gone.
…itness) ParaRouteSpec counts `project` calls over a 7-node tree: a para fold must peel each node exactly once (7), because the retained subterms come off the layer the machine already peeled. The audited route - re-`project`ing each node, or materializing its children into a List - peels every node twice and allocates an extra layer per node, which is what let para's B/op climb past droste unnoticed. Second case pins that the algebra really reads the subterms (a left-leaf-weighted sum a plain cata cannot express).
kryptt
force-pushed
the
feat/typed-recursion-schemes
branch
from
October 1, 2026 14:35
b2122a3 to
d703be5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Typed recursion schemes as composable optics — the zoo, the fusion, the M path
Thesis
A recursion scheme is an
Opticover theDirectcarrier whose existentialXis the index of the recursion — what the scheme retains — and the (co)free (co)monads are the universal indices:cata'sX = Nothing(the forgetful fold),para'sF[(S, A)](the store-comonad complement — which makesparaa lawful Lens:paraLens),histo'sAttr = νX. A × F[X](the cofree index),apo'sEither[S, A](the Prism residual worn build-side),futu'sCoattr = μX. A + F[X](the free index). Deforestation is choosing the forgetful index: the fusedhyloand the materialisingana.cross(cata)are the same optic at two X-resolutions.The categorical ground: lens = store-comonad coalgebra and biplate = Cartesian-store coalgebra (O'Connor, Functor is to Lens as Applicative is to Biplate, arXiv:1103.2841); the zoo's unification via comonadic/adjoint folds (Uustalu–Vene–Pardo 2001; Hinze–Wu–Gibbons, ICFP 2013). Full bibliography:
docs/research/2026-06-15-typed-schemes-bibliography.md.What ships
cats-eo-schemes(new artifact):cata/ana/hyloover a user-supplied pattern functorF[_](Traverse[F]) + hand-writtenBasis(Project/Embed), and the zoo —para/apo/histo/futu/zygo/mutu/cozygo/comutu, fuseddyna/codyna/chrono/elgot/coelgot,meta/metaChrono,prepro/postpro— plus the effectful*Mfamily onMonad[M].tailRecM(single-pass linear-M contract) andparaLens(the paramorphism as a lawful Lens with a caller-supplied coherent put).ana.cross(cata)(pure) fuses intoHylo— no intermediateS— because the citizens keep their(co)algebra;hyloMis the fused M spelling.FusionSpecpins the hylo law and the no-intermediate-Switness.Affineworn on the build seam (Miss= finished slot,Hit= keep going) with the newGraft[Affine]instance (done/step) — the injection vocabulary the build-side citizens construct and consume. An earlier draft shipped a separateBiAffinecarrier with an identical shape; it was dropped pre-merge (review: the arms are isomorphic and the duplication bought nothing).Machines.foldLayered):< 512-deep on-stack fast path, heapArrayDequepast it, stack-safe to 10⁶ (tested per driver).Plated/PSVecpath is removed — the typed path subsumes it (erased positional indexing made algebra arity slips a runtime error).Getter.andThenC8 fix: a Direct-carried read-only citizen previously tied three overloads (E051) andgetter.andThen(cata)did not compile; a strictly-most-specificGettermember (parameter typeOptic[A, Unit, C, Unit, Direct],DummyImplicit-shaped) now resolves it to the concreteGetter. Pinned byGetterAndThenResolutionSpec; the fullCompositionMatrixSpecstill passes.Benchmark deltas vs droste (B/op, 8 191-node tree, CI sweep)
cata2.2×,hylo1.1×,ana1.6× droste — the residual is the stack-safety machinery (droste's basic schemes are naive call-stack recursion, stack-unsafe).paraandapohalve droste's allocation (~0.5×) — eo decorates on the same array machine; droste's zoo re-embeds subterms / re-allocates theEitherspine.eqguarantee (droste's nativezoo.apois also O(1); parity with a guarantee).histo/fututrail ~1.2–1.4× — the price of stack-safety droste doesn't pay.hylobuilds no intermediateS: ~2.4× less allocation than the materialisingcrossspelling (byte-identical to the hand-written manual pair).eoHyloM(thetailRecMper-event floor atId): −49% over two optimisation rounds.Full tables:
site/docs/benchmarks.md(CI-swept numbers).Test/law coverage
Law and behaviour suites across
laws+tests+schemes(~536 examples): hylo fusion law + no-intermediate-Spin, degeneration laws (para↦cata, apo↦ana, heads-only histo↦cata, single-layer futu↦ana), grafteqguarantee, decoration round-trips,M = Idcross-architecture agreement, stack-safety to 10⁶ per driver, linear-M contract, and the carrier laws onAffine's build seam.Follow-ups (out of scope here)
Calculator.selectionport (the seam sketch indocs/brainstorms/2026-06-12-elgot-seam-sketch.mdPASSed — additive follow-up).docs/brainstorms/2026-06-12-existential-x-is-the-decoration.md): memoized refolds, the honest-hylo X-parameter, the Affine composition-row extensions.Review notes: doc claims are scoped to the shipped seams (see
site/docs/schemes.md, rewritten against the final API); the papers are consolidated indocs/research/2026-06-15-typed-schemes-bibliography.md.