[Testing] Add incremental-versus-clean build equivalence testing

[cssbruno]

[Testing] Add incremental-versus-clean build equivalence testing

Priority

High — generated-output correctness and dev-loop trust

Context

GOWDK has meaningful incremental-build coverage already. Existing tests verify behaviors such as:

• unchanged artifacts retain modification times;
• changed page sources selectively regenerate page output;
• route manifests retain the complete route set;
• backend bindings remain available during incremental generation;
• component assets can be emitted incrementally;
• stale page output is removed after a route change;
• generated output is deterministic for clean builds.

These tests validate important individual cases, but they do not establish the strongest invariant expected from an incremental compiler:

For the same final project state and configuration, an incremental build must produce the same committed artifact tree as a clean build.

Problem

Incremental systems can pass focused unit tests while still accumulating state-dependent errors, including:

• stale files after rename, removal, render-lane changes, or feature removal;
• manifests that include old records or omit unchanged records;
• asset hashes and references that depend on edit order;
• component/layout reverse-dependency gaps;
• CSS or generated client runtime that is not invalidated by the right source;
• incremental output that differs depending on the sequence of edits;
• different file modes or line endings;
• generated app output that uses stale endpoint, guard, contract, or import metadata;
• output directories that look valid during dev but differ from release builds.

Because gowdk dev serves the incremental result, equivalence is a user-visible correctness guarantee, not only a test optimization.

Goal

Add a reusable equivalence harness and CI matrix proving that supported incremental edit sequences converge to the exact artifact set produced by a clean build of the final source snapshot.

Core invariant

For a project state S0, an edit sequence E, and final state Sn:

CleanBuild(S0) → IncrementalBuild(E1) → ... → IncrementalBuild(En)

must be equivalent to:

CleanBuild(Sn)

Comparison should include all compiler-owned output relevant to the selected target:

• relative paths;
• file bytes;
• executable/file modes where meaningful;
• symlink policy, if generated symlinks ever become supported;
• absence of stale files;
• route, asset, build, OpenAPI, AsyncAPI, audit, and other manifests/reports;
• generated Go application files;
• frontend/backend split metadata;
• WASM and generated browser assets when selected.

Nondeterministic fields should be eliminated from generated output rather than broadly ignored. Any intentional exception must be narrow, documented, and normalized by the comparison helper.

Proposed test harness

Create a package-level test utility that can:

1. Materialize an initial project fixture.
2. Run a clean build into initial/.
3. Apply one or more explicit filesystem/config mutations.
4. Run incremental rebuilds into incremental/ using the real invalidation path.
5. Materialize the final project state separately.
6. Run a clean build into clean-final/.
7. Compare complete output trees and produce a useful diff.

Example API:

scenario := equivalence.Scenario{
    Name: "component rename and transitive dependency update",
    Initial: fixture,
    Steps: []equivalence.Step{
        Rename("components/card.cmp.gwdk", "components/tile.cmp.gwdk"),
        Replace("pages/home.page.gwdk", "<Card", "<Tile"),
    },
    Target: "site",
}
scenario.Assert(t)

The comparison report should identify:

• files only in incremental output;
• files only in clean output;
• byte differences with the owning artifact kind;
• mode differences;
• manifest record differences;
• the edit step after which divergence first appeared.

Required scenario matrix

Pages and routes

• add, remove, rename, and modify a page;
• route rename and static/dynamic route changes;
• paths {} record addition/removal/reordering;
• locale addition/removal and default-locale changes;
• static ↔ SSR ↔ hybrid lane changes;
• guard and cache policy changes;
• custom error-page addition/removal.

Components and layouts

• direct component edit;
• transitive component dependency edit;
• component add/remove/rename;
• component asset add/remove/rename/content change;
• layout edit and nested-layout edit;
• page moves between layouts;
• scoped CSS/keyframe changes;
• island/client/WASM declaration changes.

Backend and contracts

• action/API/fragment add/remove/route/signature changes;
• guard, CSRF, body-limit, rate-limit, and error-page changes;
• Go binding ABI changes;
• command/query/event/job registration changes;
• realtime subscription/invalidation changes;
• OpenAPI/AsyncAPI output changes.

Configuration and targets

• addon/feature enable and disable;
• output-affecting config changes;
• selected module changes;
• target addition/removal/rename;
• static, app, binary, backend, Docker, recipe, and WASM target changes;
• environment declaration changes that affect generated code shape but not runtime values.

Assets and reports

• stylesheet add/remove/rename/content change;
• content-hashed filename turnover and stale-hash removal;
• generated client runtime change;
• SEO/sitemap/robots/JSON-LD changes;
• audit policy addition/removal;
• report and manifest ordering stability.

Filesystem and platform behavior

• non-ASCII paths;
• spaces in project paths;
• case-sensitive platform scenarios, plus portable-path diagnostics once supported;
• Windows path separators and executable naming in the Windows lane.

Randomized/state-machine testing

After the deterministic matrix is stable, add a bounded property/state-machine test:

• generate a small valid project;
• choose a seeded sequence of supported edits;
• run incremental builds after each step;
• compare the final result with a clean build;
• print and retain the seed on failure;
• minimize failing sequences where practical.

This need not run with a large search space on every pull request. A short deterministic seed set can run in merge CI, with broader scheduled coverage.

Relationship to transactional publication

#669 should guarantee that only successful generations become visible. This issue verifies that the successfully committed incremental generation is semantically identical to a clean generation. The two guarantees are complementary:

• Publish build and generated-app output transactionally #669: no partial publication;
• this issue: no state-dependent final publication.

Non-goals

• Requiring every edit to use the incremental path; conservative full rebuild fallback remains valid.
• Comparing compiler debug logs or wall-clock timings byte-for-byte.
• Treating expected filesystem timestamps as generated contract data.
• Optimizing invalidation performance before correctness is established.

Acceptance criteria

• A reusable harness compares incremental output with a clean build of the same final state.
• The comparison covers complete artifact trees, bytes, meaningful modes, and stale-file absence.
• Deterministic scenarios cover pages, routes, layouts, components, assets, backend endpoints, contracts, config, and selected targets.
• Manifest and report equivalence is included rather than checking only HTML.
• Generated app and split frontend/backend output are covered.
• At least one scenario uses multiple edits whose order could expose stale state.
• Failure output identifies exact divergent files and the responsible edit step.
• A bounded seeded state-machine test is included or tracked as an immediately dependent follow-up.
• Pull-request CI runs a focused matrix; broader scenarios may run on a scheduled lane.
• Any intentional normalization/ignore rule is narrowly documented and tested.
• A divergence is treated as a correctness failure even when both outputs compile.

Related

• [Dev Server] Expand state-preserving HMR beyond mapped JS component roots #639 — broader HMR and stale generated asset handling
• [CI] Add a Windows pull-request lane for the supported CLI #645 — Windows CI coverage
• Introduce phase-typed validated program and application plan boundaries #667/Split gwdkir.Program into semantic IR and target-specific plans #668 — validated program and target-plan boundaries
• Publish build and generated-app output transactionally #669 — transactional artifact publication
• Extract shared project compilation orchestration from the CLI #671 — canonical compilation orchestration