PoC: deduplicate structurally identical structs in the provider generator

[jsteinich]

Proposal: #389 — this PR is the proof-of-concept code referenced there. Read #389 first for the full investigation; this PR covers the implementation only.

Draft / proof of concept — not for merge. Opened to show the code behind the proposal. Behind CDKTN_STRUCT_DEDUP=1; unset, generated output is byte-identical to today.

What this is

The generator names a struct after the path that reaches it, so one block shape reachable by many paths is emitted once per path — along with its mapper functions and OutputReference/List classes, which are 84% of generated bytes.

On datadog 4.18, 95.5% of generated interfaces are structurally identical duplicates. The largest equivalence class is 364 byte-identical interfaces.

This is what OOM-kills package-go: not the Node heap ceiling (tuned three times), but the go build verification inside jsii-pacmak, which needs ~18 GB for a single generated package on a runner with ~24 GB usable. That memory is not governed by NODE_OPTIONS at all.

The duplication is an artifact of schema serialization

terraform providers schema -json inlines every call site and discards sharing the provider author wrote by hand:

getComputeSchema()                              <- 1 Go function
  ^ called 2x inside
getApmLogNetworkRumSecurityAuditQuerySchema()   <- 48 call sites
  + getMetricQuerySchema (11), getFormulaQuerySchema (10),
    getProcessQuerySchema (10), getApmStatsQuerySchema (2)
  v
364 generated TypeScript interfaces

Cross-resource duplication has the same origin: resource_datadog_powerpack.go calls getNonGroupWidgetSchema() defined in resource_datadog_dashboard.go. Dedup recovers structure the provider actually expressed — it is not merging things that merely look alike.

detectAttributeLoops is not broken

It threads knownStructs down each DFS branch, so it merges only when a shape reappears among its own ancestors (true recursion). Sibling branches are never compared and each top-level attribute restarts from {}. The 364-member class is ...ApmQueryComputeQuery vs ...LogQueryComputeQuery — siblings, invisible to an ancestor-only check by construction. It works as designed; its scope cannot see this.

Equality must be a full recursive signature

This pass deliberately does not reuse getAttributeIdentifier. That comparison uses attribute names plus one level of nesting (with an in-code caveat that it is an approximation). Safe for the ancestor-only case; applied resource-wide it yields 199 classes where an exact recursive hash yields 200 — wrongly merging DashboardV2WidgetCohortDefinition with DashboardV2WidgetGroupDefinitionWidgetSloListDefinition, which share a shallow shape and diverge deeper. That would emit incorrect bindings.

Measured — datadog 4.18, end to end

Stage	Baseline	With dedup
Generated TS	99.6 MB	12.3 MB (-87.7%)
Interfaces	13,220	1,491
tsc	0 errors, peak 4,674 MB	0 errors, peak 744 MB
jsii	—	0 errors, assembly 22 MB
jsii-pacmak --target go	—	rc=0, 8,110 files, 56.5 MB
go build ./... (GOMAXPROCS=8, cold)	~18 GB, OOM-killed	925 MB, 11.1 s, exit 0

925 MB fits a standard 7 GB runner at full parallelism, with the compilation check kept.

Why this is not ready to merge

1. Naming is a placeholder. The canonical-name rule (shortest name, lexicographic tie-break) is order-independent but still not stable across provider releases — 0.2-0.8% of shapes renamed on datadog 4.17 -> 4.18, because the elected member can leave the equivalence class. The intended fix is a checked-in signature -> name registry with incumbents winning collisions; not implemented here.
2. Breaking change. 13,220 -> 1,491 exported names on datadog. jsii has no type aliases, and interface A extends B {} would not help since it retains the per-name classes and functions that are 84% of the bytes. Major bump per provider.
3. No tests yet.

Scope: per-resource, not provider-wide (tested)

	datadog	aws	awscc
Shapes per-resource -> provider-wide	1,257 -> 879	6,544 -> 4,758	12,698 -> 8,763
Name length	30 -> 25	37 -> 30	41 -> 57
Resolve at 1 segment	75% -> 36%	94% -> 43%	95% -> 2%
Rename rate 4.17->4.18	0.56% -> 1.80%	—	—

Provider-wide is never clearly better and is actively bad for awscc (machine-generated CloudFormation schemas give every resource the same generic block names).

Fleet impact — datadog is an outlier

Provider	Interfaces	After dedup	Delivered	Provider-wide ceiling
datadog	13,220	1,491	88.7%	90.3%
aws	12,188	8,923	26.8%	54.3%
awscc	21,026	16,749	20.3%	73.2%

Most providers' duplication is across resources, which per-resource scope does not capture. Six providers (acme, cfncompat, external, http, null, time) have 0% redundancy — a no-op with no API impact.

Caveats

• Prototyped against generator 0.21 (baseline reproduced published 0.24 output exactly: 99.6 MB / 346 files), then ported here; typechecks clean.
• dashboardv2 compile figures measured on a Windows host with 32 GB, not Linux CI.
• The ~18 GB for 4.18 is the measured 4.17 figure scaled by output growth — not directly measured.
• Go compile impact for aws/awscc after dedup not measured.

🤖 Generated with Claude Code