Add UpdateOrganizationalPrethinkContext for multi-repository context - #30
Open
mtthwcmpbll wants to merge 5 commits into
Open
Add UpdateOrganizationalPrethinkContext for multi-repository context#30mtthwcmpbll wants to merge 5 commits into
mtthwcmpbll wants to merge 5 commits into
Conversation
Prethink today leaves every repository describing only itself. This adds an organizational counterpart that combines many repositories into one directory on the filesystem, which may live outside all of them, so a single coding agent started there can reason across an organization. Each repository contributes rows to combined CSVs keyed by a leading `Repository` column, plus its FINOS CALM architecture at `architecture/<repository>.json`, and the agent configuration at the root of the collection is updated to describe them. Re-analyzing a repository replaces exactly that repository's rows, so a collection is refreshed one repository at a time. Updates are streamed rather than read into memory, and are guarded by a JVM monitor plus a file lock so repositories analyzed in parallel -- by one `mod run` or several -- can share one collection. No list of data tables is configured: a store only registers a table once a row is inserted, so iterating it exports whatever the composite discovered. Two things follow from that. A table a repository stops producing is absent from the run entirely, so its stale rows are swept out of the collection instead. And the combined tables are catalogued in `tables.csv`, so a repository with a narrower composite does not erase the description of a table other repositories are still filling. The repositories analyzed are left unchanged: the collection is written directly to the filesystem rather than expressed as source file changes, so there is no diff to review or commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nvqoq3ZbyimLdm7jWrJJ6T
…e target directory Two things kept a collection from holding what a run actually discovers. The combined tables were described by looking each table's row class up by name, which resolves only against the classloader of the artifact this module ships in. The CLI loads every recipe artifact separately, so a run of the 166 Moderne discovery recipes contributed 13 tables -- exactly the ones declared here -- and silently dropped every table another artifact declared, test coverage and code quality among them. The schema is now read from the table instance the store already holds, which is the same object whichever artifact declared it, and those columns also document the tables this run exported. The regression test compiles a data table into its own directory and loads it through its own classloader, since a test where everything shares one classloader cannot observe this at all. `targetDirectory` is now optional, defaulting to the working directory of the process running the recipe, so a working set collects into itself with `mod run .`. That is the directory the run was started from rather than the path it was given: every repository in a run is analyzed by one process, so there is no per-repository directory to resolve against, and naming a directory stays the only way to collect somewhere else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A context groups data tables under one name and description, but that grouping lives only in the arguments of the ExportContext that declares it, where nothing else in the run can see it. The organizational export discovers tables from the store rather than being told about them, so it has no way to learn that TestMapping belongs to "Test Coverage" -- which is why a collection gets one undifferentiated context where a repository gets seventeen. ExportContext now records the composition in a ContextTables data table, one row per context and table. A sibling table rather than a column on ContextRegistry: the registry's grain is one row per context and it is derived by UpdateAgentConfig from the markdown it finds, whereas this is one row per context and table and is known first-hand by the recipe that exports it. Widening the registry would multiply its rows for every consumer that reads it today. Every configured table is recorded, not just those with rows here: the composition is a property of the composite, so a table this repository has nothing to say about is still part of the context in a collection other repositories contribute to. The rows are written from generate() because a DataTable only accepts rows in cycle 1, while the export itself runs in cycle 2. Also simplifies the organizational export to take its row schema from DataTable#getType(), the accessor ExportContext already uses, rather than walking the type hierarchy by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A collection is only as navigable as the per-repository context it combines, and a repository gets seventeen named contexts where a collection got one. The grouping is now available -- ExportContext records it -- so the organizational export reads it from the store and documents each context in its own markdown, exactly as a repository does, with the recipe's own context left as the umbrella for tables no exporter claimed. The grouping is discovered one repository at a time, so it has to outlive the run that discovered it the way the table catalog does. It is kept in contexts.csv alongside tables.csv, which gains a Context column; both are read by header name, so a collection written before either existed reads back with the new columns empty rather than failing. That catalog is also what makes removal exact: a context is this recipe's to delete because it is listed there, which keeps the sweep away from the markdown the CALM export writes into the same directory. A table this run has no grouping for keeps whichever context an earlier run filed it under, so one repository analyzed by a composite that declares no contexts does not flatten the collection for everyone else. The mapping table is read by class and column name rather than by type, for the same reason its rows have to be: the recipe that declares a context is loaded by its own artifact's classloader, so the ContextTables it writes is a different class from this one even when both come from this module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Filing a table only when the repository that produced it is analyzed again leaves a collection in whatever layout it was first built with: one repository run against a composite that declares contexts moved only its own tables, and the rest stayed in the umbrella until every other repository had been re-analyzed too. The grouping describes the composite rather than the repository it arrived with, so it is applied to every catalogued table it covers. Two things the first run against a real collection turned up: The mapping table was being combined into the collection like any other discovery. How the contexts are laid out is not something to read about inside one of them, so it is left out of the export. A catalogued table whose combined CSV is no longer there was never dropped, because the sweep walks the files it finds and that file is not among them. Reconciling the catalog against the collection covers the table a composite stops exporting entirely, which is exactly what the mapping table just became. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mtthwcmpbll
marked this pull request as ready for review
August 10, 2026 19:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an organizational counterpart to
UpdatePrethinkContextthat combines many repositories into one directory on the filesystem — which may live outside all of them — so a single coding agent started there can reason across an organization. Each repository contributes rows to combined CSVs keyed by a leadingRepositorycolumn, plus its FINOS CALM architecture atarchitecture/<repository>.json, and the agent config at the collection root is updated to describe them; re-analyzing a repository replaces exactly that repository's rows.No list of data tables is configured — a store only registers a table once a row is inserted, so iterating it exports whatever the composite discovered, and it works with any Prethink composite. Two things follow: a table a repository stops producing is absent from the run entirely, so its stale rows are swept out of the collection instead, and the combined tables are catalogued in
tables.csvso a repository with a narrower composite doesn't erase the description of a table others are still filling.The repositories analyzed are left unchanged — the collection is written directly to the filesystem rather than expressed as source file changes, so there's no diff to review or commit. Merges stream rather than buffer, and are guarded by a JVM monitor plus a file lock so repositories analyzed in parallel can share one collection.
Also extracts
UpdateAgentConfig's marker/template/parsing logic into a sharedAgentConfigSection(behavior unchanged, its 15 tests still pass) and makesGenerateCalmArchitecture.generateCalmJsonFromDataTablespublic so the organizational exporter renders the same document rather than reimplementing it.How to test
You can run this as a locally-installed YAML recipe with:
This runs the normal prethink recipe AND then generates this into a combined location, which removes the need for the user to manage a full-on fork of our batteries-included Moderne Prethink recipe just to swap in all of the UpdateContext with UpdateOrganizationalPrethinkContext calls.
Areas of feedback
This introduces a side effect of the recipe where it creates new files outside of the repository patch semantics of a normal recipe run. I'm not sure if this make sense or is the right shape for this kind of collector, and works fine locally with the Moderne CLI (or presumably an automated mass-run scenario) but might not work in more constrained environments like the Moderne SaaS.
Example of this running on the Default organization:
