Skip to content

Make RuntimeUsageFilter package heuristic configurable (opt-in) + parameterized advice filter - #2

Open
marceltft wants to merge 17 commits into
mainfrom
runtime-usage-configurable-heuristic
Open

marceltft wants to merge 17 commits into
mainfrom
runtime-usage-configurable-heuristic

Conversation

@marceltft

Copy link
Copy Markdown

Summary

Makes the Appian fork's false-positive filtering configurable and generic, so it can be maintained on appian/main and contributed upstream without hardcoded Appian assumptions.

Sourced from the appian fork so the contribution is attributed to Appian.

Changes

  1. Configurable runtime-usage package heuristic (runtimeUsage { packageHeuristic { ... } }, opt-in, disabled by default). Removes hardcoded appian-/eng- prefixes, appian-libraries path assumption, and appian/libraries stopwords — all now configuration. Also fixes that the heuristic was previously constructed but never invoked.
  2. Parameterized advice coordinate filter (advice { includeCoordinates(...); excludeCoordinates(...) }) — pure regex over dependency coordinates, applied to all advice. Enables internal-only reports without any hardcoded group names.
  3. Configurable transitive-exposure depth (advice { transitiveDepth(N) }, default 1) — bounded BFS over the reverse dependency graph to catch multi-hop transitive usage.
  4. generateBuildHealth directory fix — feed the filtered-advice directory's *.json contents (not the directory) to GenerateBuildHealthTask, fixing a FileNotFoundException on the 3.19.x base.

De-Appianization

No Appian-specific strings remain in the plugin source. Reproduce the old hardcoded behavior via config:

dependencyAnalysis {
  runtimeUsage { packageHeuristic { enabled(true); stripPrefixes('appian-','eng-'); skipLeadingSegments(1); stopwords('appian','libraries') } }
  advice { includeCoordinates('^:.*', '^com\\.appian.*'); transitiveDepth(2) }
}

Testing

  • Unit: FilterTransitiveExposureTest (8), RuntimeUsageFilterTest (10) — 0 failures.
  • Validated end-to-end on ae master (759 projects): filters run correctly; internal-only filter reduces external advice to 0; transitive-depth sweep behaves as expected (depth 1→2 catches ~17% more real false positives, curve flattens by depth 3).

Note: our earlier ConcurrentModificationException patch was dropped during the rebase — main (3.19.x) already fixes it (iterator-based, in AndroidTransform).

Marcel Santos added 17 commits September 4, 2026 09:15
Projects are partitioned into configurable groups (default 100).
Configure via -Ddependency.analysis.batch.size=N.
Batch size <= 0 disables batching (all projects in one group).
Collects per-project health and metadata reports for a subset of
projects into a batch output directory on disk.
When batch size > 0 and project count exceeds batch size, intermediate
BatchAggregateTask instances add execution ordering constraints.
This prevents all project analyses from being in-flight simultaneously,
reducing peak memory usage on very large builds.
Single-batch builds retain original behavior.
Adds FilterTransitiveExposureTask that runs between per-project analysis
and generateBuildHealth. It cross-references type usage data to suppress
'remove' advice for project deps that downstream consumers access
transitively.

This eliminates ~91% of false positives (80/88 confirmed cases in ae)
where DAGP flagged a dep as unused but removing it broke downstream
compilation because consumers relied on transitive class access.

Algorithm:
- Build reverse dep graph from AggregateTypeUsageReport data
- For each 'remove dep D from project A' advice:
  - Get D's public classes (from PublicTypes)
  - Check if any consumer of A uses those classes
  - If yes: suppress the removal advice

Also updates CONTEXT.md with correct run command (don't override
gradle.properties heap with -Dorg.gradle.jvmargs) and updates
fixing-false-positives.md with July 23 run results.
Wire FindRuntimeDepsTask (Spring DI, Liquibase, @componentscan detection)
into the cross-project filtering pipeline. This suppresses false-positive
'remove' advice for dependencies that are used at runtime via:
- @bean return types
- @componentscan packages
- @import class references
- Liquibase migration class references
- Spring XML bean definitions

The runtime filter runs after the transitive exposure check, providing
two layers of false-positive suppression in a single aggregation task.

Part of LCP-60585.
The package-naming heuristic (Strategy 2) that suppresses removal advice for
runtime-used deps was previously hardcoded to Appian's module/artifact naming
and, moreover, was never actually invoked by FilterTransitiveExposureTask.

This change:
- Adds a RuntimeUsageHandler DSL: runtimeUsage { packageHeuristic { ... } }
  with enabled/stripPrefixes/stripSuffixes/skipLeadingSegments/stopwords/
  minSegmentLength. Disabled by default (opt-in), so upstream behavior is
  unchanged.
- Refactors extractPackageSegments to be driven by configuration, removing the
  hardcoded appian-/eng- prefixes, appian-libraries assumptions, and
  appian/libraries stopwords.
- Wires the config through FilterTransitiveExposureTask and actually invokes
  the heuristic when enabled.
- Tests: an Appian-shaped config reproduces the previously hardcoded segments;
  the default config is generic; the disabled path is a no-op.
…eBuildHealth

GenerateBuildHealthTask reads each projectHealthReports entry as a JSON file
(projectHealthReports.files.map { it.fromJson<ProjectAdvice>() }). Passing the
filtered-advice output directory itself caused FileNotFoundException '(Is a
directory)' on the 3.19.x base. Pass the directory's *.json contents as a
file tree built by filterTransitiveExposure instead.
…e depth

- New 'advice { }' DSL block (AdviceFilterHandler):
  - includeCoordinates(...) / excludeCoordinates(...) regex over dependency
    coordinate identifiers, applied to ALL advice (add/remove/change) in
    FilterTransitiveExposureTask. Enables e.g. internal-only reports via
    includeCoordinates('^:.*', '^com\\.appian.*').
  - transitiveDepth(Int) controls how many downstream hops the transitive-
    exposure check follows (default 1 = direct consumers, preserving prior
    behavior). Implemented as a bounded BFS over the reverse dep graph.
- Tests: multi-hop depth (kept at depth 1, suppressed at depth 2), include
  filter, exclude filter, empty-filter no-op. 8/8 FilterTransitiveExposureTest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant