feat(080): row-type discovery probe + template-seeded generation (cxs2 spec 080)#65
Merged
Conversation
…2 spec 080) - NEW POST /v1/discover-row-types: exact two-pass row-type discovery (novelty = full snapshot every run; stats via per-table cursors: semantic_events (timestamp,event_gid), data_points (timestamp,signature), entities _version + uniqExact(gid) snapshot counts); monotone map-key/ JSON-path inventories; series head-label enrichment (headSchema). - NEW GET /v1/published-template: the published global template by name (the FR-020 maturation package's seed). - smart-generate: optional template_name (seed via buildCubesFromTemplate, probe-pruned by the request filters) + cube_meta (family-#2 provenance, replaces the 013 default_model/template_checksum stamp); template path forces skip_llm, disables deep-profile sampling (registry pruning must not flap), forces the target table into internalTables (partition scoping), and merges regenerations via mergeTemplateModel so ai_generated/team content survives. - buildCubesFromTemplate: additive filters/cubeName/cubeMeta options. - FraiOS JIT provisioning: the row-type-pipeline provider provisions as team ADMIN (dataschemas select/update perms are owner/admin-only) with an upgrade path for already-provisioned identities. Verified live from cxs2 e2e (somi.is, 13.9M-row semantic_events + 2.4M-row data_points): cxs2 suites US2 6/6, US3 10/10, US5 2/2. cubejs npm test: 620/622 (2 pre-existing partition.test.js failures on clean main). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ies head labels - discover-row-types (semantic_events): identity = event value, but rows with an empty event register under synthetic 'type:<type>' keys (owner amendment — page/screen/identify/log/custom carry no event name). - discover-row-types (data_points): optional headSchema — series display labels resolved from <headSchema>.timeseries (non-fatal when absent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner refinement: an event type spans multiple sources across rows (source.label/type are scalar Nullable(String) per row). Add a sourceEvidence config on semantic_events and groupUniqArray(64) over `source.label` in both the novelty + stats passes, surfaced as observedSources alongside observedTypes. groupUniqArray drops NULLs, so sourceless platform events fall out cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner: count() + max(timestamp) are the PRIMARY active-indicator and were missing/zero for most types because they came from the cursor-bounded delta. - Novelty (full-partition) pass now computes total_count + min/max per key — always correct, independent of the delta cursor. count/min/max are cheap columnar aggregates. - The delta pass carries ONLY the expensive union-mergeable evidence (distinct Segment types, source labels, map keys, JSON paths). - Cursor bound is derived from the delta slice (its max IS the global max), removing a second full-partition scan. - Truncation: raise caps (types 100, sources 256) + uniqExact totals + evidenceTruncated flag so capped arrays are no longer silently lossy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…spec 080 §4/A4) Trim smart-generated model metadata to shape + semantics only; stop baking data snapshots and redundant/volatile provenance. Applies to ALL smart-gen (row-type pipeline AND external SL), per the owner decision. cubeBuilder.js: - per-member meta keeps only `auto_generated` (the merge key) plus the structural markers map_key / native_map / nested_lookup_key / source_group / array_join_alias. Drops source_column, field_type, unique_values, lc_values, known_values. - cube-level meta drops volatile `generated_at` (was breaking byte-identical reruns, SC-002) and `grain_description` (a sentence duplicating `grain`). - descriptions are stamped only on the column's own member (name === column), so derived timestamp_min/_max measures no longer duplicate the sentence. - remove the now-dead value-baking helpers (lcBakeList, bucketCardinality, looksLikeIdentifierValues, toJsonFieldType, isInt8Boolean, identifier regexes, LC_BAKE_CAP). source_column had two live consumers — the ARRAY JOIN exclusion in buildArrayJoinCube and the AJ-SQL pruner in smartGenerate.js. Both switched to a transient `_sourceColumn` on the in-memory field objects; yamlGenerator whitelists serialized keys so `_`-props never persist in the model. merger.js / templateMerger.js: shed grain_description/generated_at/ generation_filters from legacy files on the next reconcile. Merge-invariant safe (§4b): every generated member keeps `auto_generated: true`; the trim only removes keys no merger reads. Verify: full cube suite 620/622 (2 pre-existing partition.test.js failures, unrelated); intentional-change tests realigned to the lean contract; a buildCubes->generateYaml pipeline harness confirms the serialized model carries none of the 9 trimmed markers, keeps auto_generated/map_key/native_map, leaks no PII/value snapshots, drops duplicate derived-measure descriptions, and is byte-identical on rerun (SC-002). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… map keys (spec 080)
Row-type models must contain ONLY fields the slice actually populates (the probe
is the mechanism). Two leaks let unused fields through:
1. processColumns force-kept `hasValues===false` GROUPED/nested columns via a
stale exception ("profiler never profiles arrays"). It does profile them —
accurately (commerce.revenue: hasValues=false for a work-item slice,
hasValues=true for a bookings slice). The exception leaked dozens of all-empty
nested members (commerce.*/screen.*/traits.*/network.*/device.*/analysis.*/
entity_linking.*/access.*/…) into slices that never populate them. Fix: skip
every hasValues===false column uniformly (AJ-group columns are re-emitted by
buildArrayJoinCube, so dropping the empty carry-over is safe).
2. Map keys whose only values are ''/'0' were kept. Fix: string-key uniq now
excludes ''/'0'/whitespace (profiler); measure map-keys drop when
min===max===0 (cubeBuilder), mirroring the basic-column all-zero skip.
Verified live vs real ClickHouse across 3 slices (harness runs the real
profileTable→buildCubesFromTemplate→generateYaml, then validates EVERY member
against the data): Work Item Outcomes 132→45 members (1238→294 lines), Bookings
95→61, User Profile Updated →20 — 0 unused members in each, and used nested
fields (commerce.revenue/units/unit_price on bookings) correctly survive. Full
cube suite still 476/478 (2 pre-existing partition failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to smartdataHQ/cxs2#807 (internal SL row-type models). All changes additive.
New endpoints
POST /v1/discover-row-types— exact two-pass row-type discovery: novelty is a FULL snapshot every run (presence never rides on cursor semantics); stats ride per-table cursors (semantic_events(timestamp,event_gid)tuple-compare,data_points(timestamp,signature),entities_version+uniqExact(gid)current-state counts). Returns monotone map-key/JSON-path inventories (capped) and the next cursor from scan bounds. Series head-label enrichment viaheadSchema(non-fatal).GET /v1/published-template?name=— the published global template (seed for the FR-020 maturation context package).smart-generate (backward compatible)
template_name: seed viabuildCubesFromTemplate+ probe-prune against the requestfilters(published-template fetch replicated in cubejs —templateResolver.js, same GraphQL surface the reconciler's feed reads).cube_meta: caller provenance (marker family Fix: resolve datasource access issues by simplifying user query logic #2,managed_by: row-type-pipeline) stamped INSTEAD of the 013default_model/template_checksumpair.skip_llm; disables deep-profile sampling (registry pruning must not flap on sampled key inventories — the pre-existingpartition.test.jsfailures document the adjacent gap); forces the target table intointernalTablesso the generated cube bakes partition scoping even when team settings don't enumerate the table; regenerations merge viamergeTemplateModel(ai_generated/team content survives; auto_generated refreshes; template converges).Auth/JIT
FraiOS JIT provisioning: tokens with
provider: row-type-pipelineprovision as team admin (dataschemas select/update permissions are owner/admin-only — a plain member reads an EMPTY dataschemas list, silently), with an upgrade path for already-provisioned identities.Verification
npm test: 620/622 — the 2 failures are pre-existing on clean main (verified via stash).🤖 Generated with Claude Code