245: Add support for com.android.test modules

[DanielJette]

What does this change accomplish?

Fixes #245

Applying the plugin to a com.android.test module failed at configuration time:

A problem occurred configuring project ':screenshot-testing'.
> Gradle project must contain an `android` closure

Project.android looked only for ApplicationExtension and LibraryExtension. A test-only module
has a TestExtension, so the lookup fell through to the GradleException.

How have you achieved it?

Three plugin changes, plus a sample module that proves the round trip.

• Project.android also resolves TestExtension, and a new Project.isTestModule reports it.
• baselineSourceDir defaults to the main source set for a test module. A com.android.test
  module has no androidTest source set — its tests are its main sources — so the old default of
  src/androidTest/assets pointed at a directory that does not exist.
• The Testify dependency is added as implementation rather than androidTestImplementation, for the
  same reason.

Samples/Flix/FlixTest is a worked example: a com.android.test module with
targetProjectPath ':FlixSample' and one Compose screenshot test, with its recorded baseline.

The two defects review found, both fixed here:

1. Nothing installed the application under test. A test module's own installDebug installs its
   test APK; the application named by targetProjectPath was never a dependency, so on a clean
   device am instrument could not find its target package. screenshotTest and screenshotRecord
   now depend on the target project's install task.
2. An instrumentation error reported success. finalizeTaskAction looks for FAILURES!!!,
   INSTRUMENTATION_CODE: 0 and Process crashed; Error=Unable to find instrumentation target package matched none of them, so the task passed having run nothing. It now also fails on
   INSTRUMENTATION_STATUS: Error= — which required giving runProcess an opt-in
   redirectErrorStream, because it read only standard output and the error never reached the log.
   The wider problem, that adb's exit code is discarded everywhere, is filed as The Gradle plugin discards adb's exit code and standard error #340.

Both package ids are inferred, so the sample configures neither. applicationPackageId comes
from the target project; testPackageId from the test module's namespace, since it has no
applicationId of its own. Getting this wrong is a trap worth removing rather than documenting: with
both set to the test module's id, screenshotRecord reported ✓ Recording baseline and
screenshotPull then looked in the wrong package and silently pulled nothing.

Scope note. The branch this came from also carried IntelliJ plugin changes for gutter icons in
test modules. That work is unfinished — commented-out alternatives, duplicate imports and unbalanced
braces — and it does not compile, so it is not included here. #245 is specifically about the Gradle
plugin refusing to apply, which this closes. The IDE-side support is #339; it needs runIde to
verify rather than CI.

The FlixTest build file also needed modernising for the current toolchain: the standalone
org.jetbrains.kotlin.android plugin is rejected under AGP 9, lintOptions is gone, and the Java
target was still 21.

Scope of Impact and Testing instructions

Additive for application and library modules. baselineSourceDir and the package-id inference change
only when the module is a com.android.test; verified unchanged for FlixSample, FlixLibrary,
GmdSample and LegacySample. The instrumentation-error check applies to every module, and is the
one behaviour change that could turn a previously green build red — correctly, since such a build ran
no tests.

CHANGELOG entries added under Unreleased. recipes/21-library-projects.md has a new Test-only
modules section.

Verified locally:

./gradlew Plugin:ktlintCheck Plugin:test Plugin:assemble   # BUILD SUCCESSFUL
cd docs && npm run build                                   # clean

End to end on an API 37 emulator, with both packages uninstalled first:

$ ./gradlew FlixTest:screenshotTest
> Task :FlixSample:installDebug
> Task :FlixTest:installDebug
OK (1 test)

$ ./gradlew FlixTest:testifySettings
  targetPackageId        = dev.testify.samples.flix        # inferred from targetProjectPath
  testPackageId          = dev.testify.samples.flix.test   # inferred from the namespace
  baselineSourceDir      = src/main/assets

And the silent-pass check, with the install dependency disabled and the target removed:

SCREENSHOT TESTS HAVE FAILED!!!
BUILD FAILED

FlixTest is in the Bitrise test_flix workflow, with APK paths checked against the build output.
flix_sample.yml only assembles :FlixSample, so it does not cover this module.

[DanielJette]

Filed #339 for the IDE-side support I left out of this PR — gutter icons and Go to image for com.android.test modules. It notes that the unfinished attempt is on the 245-test-module-support branch and that it needs runIde rather than CI to verify, since Samples/Flix/FlixTest from this PR is the project to test it against.

[AndroidTestifyBot]

Changes requested

The plugin now applies to a com.android.test module, and the three plugin changes are individually correct. But the round trip the description presents as proof only works if the app under test is already on the device, and when it isn't, the task reports success.

1. screenshotTest does not install the target app, and passes anyway

On an API 37 emulator, with dev.testify.samples.flix and dev.testify.samples.flix.test uninstalled first:

$ ./gradlew FlixTest:screenshotTest
> Task :FlixTest:installDebug
Installing APK 'FlixTest-debug.apk' ...
INSTRUMENTATION_STATUS: Error=Unable to find instrumentation target package: dev.testify.samples.flix
INSTRUMENTATION_STATUS_CODE: -1

BUILD SUCCESSFUL in 11s

Two defects:

• Nothing installs :FlixSample. testifySettings shows installTask = installDebug, installAndroidTestTask = null — that installDebug is the test module's own APK. The app named by targetProjectPath is never a dependency.
• Zero tests ran and the build is green. finalizeTaskAction looks for FAILURES!!!, INSTRUMENTATION_CODE: 0 and Process crashed; this output matches none. FlixTest:screenshotRecord on the same device also reports BUILD SUCCESSFUL, pulls nothing and records nothing.

The verification in the description passed because FlixSample was left installed from an earlier run. A user following the sample on a clean device or CI agent gets a silent false pass. The plugin can read TestExtension.targetProjectPath, so it can depend on that project's install task; and an instrumentation error status should fail the task.

2. Nothing in CI runs it

FlixTest isn't in bitrise.yml, and flix_sample.yml only assembles and tests :FlixSample. The committed baseline is never checked, so "a sample module that proves the round trip" is unguarded from the first commit. Running it in CI on a fresh emulator would also have caught (1).

3. The trap is documented only in a build-file comment

The description calls the applicationPackageId mix-up "a trap anyone writing a test module will hit". The mitigation is a comment in one sample's build.gradle. There is no docs page, no mention in settings.md or the library-projects recipe, and the plugin still infers nothing for a test module — both ids must be set by hand or validate() throws. The target id is derivable from targetProjectPath. At minimum this needs a recipe; better, inference.

4. Sample build file

• Unused dependencies copied from FlixSample: hilt-navigation-compose, hilt-android, ktor-client-android, kotlinx-serialization-json, coil-compose, idling-concurrent. The single test renders a Text.
• buildscript { ext { versions … } } and composeOptions { kotlinCompilerExtensionVersion } are dead with the Compose compiler plugin applied.
• .gitignore gains Plugins/IntelliJ/.kotlin/, which is unrelated and already covered by the existing .kotlin/ line.

Verified

• Plugin:ktlintCheck Plugin:test Plugin:assemble pass locally.
• baselineSourceDir resolves to src/main/assets for FlixTest, and the pull destination follows it.
• With FlixSample installed, the committed baseline matches on a 37-1080x2220@440dp-en_US emulator.
• Merged with #332, FlixTest still configures. settings.gradle merges cleanly with #334; CHANGELOG conflicts with the other five open code PRs.

[DanielJette]

Agreed on all four. Point 1 was the important one, and my verification had the hole you identified — FlixSample was installed from an earlier run, so I never saw the failure.

1. Both defects fixed

The target application is now installed. The plugin reads TestExtension.targetProjectPath and depends on that project's install task. It handles either configuration order, since the target may be evaluated before or after the test module:

$ ./gradlew FlixTest:screenshotTest --dry-run
:FlixSample:installDebug SKIPPED
:FlixTest:installDebug SKIPPED

With both packages uninstalled first, which is your test:

$ adb uninstall dev.testify.samples.flix
$ adb uninstall dev.testify.samples.flix.test
$ ./gradlew FlixTest:screenshotTest
> Task :FlixSample:installDebug
> Task :FlixTest:installDebug
OK (1 test)

An instrumentation error now fails the task. This took a second attempt. My first fix added INSTRUMENTATION_STATUS: Error= to finalizeTaskAction and did nothing, because runProcess reads only process.inputStream — the error text was never in log. runProcess now takes an opt-in redirectErrorStream, which the am instrument call passes. Proven by disabling the install dependency and uninstalling the target:

SCREENSHOT TESTS HAVE FAILED!!!
> Screenshot tests have failed
BUILD FAILED

Worth flagging that runProcess also discards the process exit code, so other adb failures remain invisible. I have deliberately not widened this PR to that — some tasks may tolerate a non-zero exit and it wants checking per call site. I will file it.

2. CI

FlixTest is now in Bitrise test_flix, with the APK paths checked against what the build produces (Samples/Flix/build/outputs/apk/debug/FlixSample-debug.apk as the app, FlixTest-debug.apk as the test APK). Your point that flix_sample.yml only assembles :FlixSample is correct and I had claimed otherwise.

3. Inference rather than a comment — agreed, and done

You were right that the target id is derivable, so documenting a trap was the wrong answer. Both ids are now inferred and the sample configures neither:

$ ./gradlew FlixTest:testifySettings
  targetPackageId        = dev.testify.samples.flix        # from targetProjectPath
  testPackageId          = dev.testify.samples.flix.test   # this module's namespace
  baselineSourceDir      = src/main/assets

applicationPackageId resolves through the target project's applicationTargetPackageId; testPackageId falls back to the TestExtension namespace, since a test module has no applicationId of its own. Checked that FlixSample, FlixLibrary, GmdSample and LegacySample all resolve unchanged.

There is also a Test-only modules section in recipes/21-library-projects.md now, covering what differs — the package ids and src/main/assets — and that both APKs are installed.

4. Sample build file

• Dropped the copied dependencies. What is left is Library, ComposeExtensions, the Compose BOM plus material3, ext:junit, test:runner, and test:rules — that last one is genuinely needed, since ComposableScreenshotRule extends ActivityTestRule. Lifecycle alignment kept with the comment explaining why.
• buildscript { ext { versions … } } and composeOptions removed; the Compose compiler plugin supersedes them.
• .gitignore reverted. It was unrelated and already covered by the existing .kotlin/ line.

[AndroidTestifyBot]

Changes requested

Big improvement: target install, id inference, the docs section and CI coverage are all in, and I verified each. The failure detection still misses a case it is meant to catch, and the sample's test package collides with FlixSample's.

Verified

• Clean device: with dev.testify.samples.flix and dev.testify.samples.flix.test uninstalled, FlixTest:screenshotTest runs :FlixSample:installDebug then :FlixTest:installDebug, and gives OK (1 test) against the committed baseline.
• Inference: testifySettings gives targetPackageId = dev.testify.samples.flix, testPackageId = dev.testify.samples.flix.test, baselineSourceDir = src/main/assets, with neither id configured.
• runProcess swap (Runtime.exec → ProcessBuilder, which every adb call goes through): LegacySample:screenshotTest gives OK (88 tests) and FlixSample:screenshotTest gives OK (22 tests) on this branch. So the red Legacy Sample check doesn't reproduce here either; please re-run it.
• Plugin:ktlintCheck Plugin:test Plugin:assemble pass; docs build is clean; the sample build file is trimmed as asked.

1. "Target not installed" can still pass

I repeated your negative test: target uninstalled, install dependency bypassed.

$ adb uninstall dev.testify.samples.flix
$ ./gradlew FlixTest:screenshotTest -x FlixSample:installDebug
> Task :FlixTest:screenshotTest
android.util.AndroidException: INSTRUMENTATION_FAILED: dev.testify.samples.flix.test/androidx.test.runner.AndroidJUnitRunner
	at com.android.commands.am.Instrument.run(Instrument.java:549)
BUILD SUCCESSFUL in 2s

Same emulator (API 37), same situation, but a different shape from the INSTRUMENTATION_STATUS: Error=Unable to find instrumentation target package I got in the first round. finalizeTaskAction only matches the second form, so zero tests ran and the build is green again.

Matching on error strings will keep missing variants. The robust check is the other way round: am instrument -w always ends a run that actually executed with OK (n tests) or FAILURES!!!, so treat a log with neither as a failure. Keep the explicit matches as well if you want the clearer message.

2. FlixTest's package is FlixSample's test package

testPackageId falls back to the namespace, dev.testify.samples.flix.test. That is also FlixSample's own androidTest APK id (applicationId "dev.testify.samples.flix" plus the default .test suffix). The two modules install over each other: after this run the device has a single dev.testify.samples.flix.test, and which APK it is depends on which task ran last. Both Bitrise steps in test_flix and anyone running FlixSample's tests from Android Studio after FlixTest are affected. Please give the sample a distinct namespace, e.g. dev.testify.samples.flix.screenshots.

3. The namespace isn't always the test APK's id

testModulePackageId reads TestExtension.namespace, but a com.android.test module can set defaultConfig.applicationId (and a build-type suffix), and that is what gets installed. Prefer defaultConfig.applicationId ?: namespace. The docs sentence "testPackageId is this module's own namespace" needs the same qualification.

[DanielJette]

Points 1 and 2 are fixed. Point 3 I dug into and I think it does not hold on AGP 9 — evidence below, happy to be shown otherwise.

1. Detection inverted, as you suggested

You were right that matching error strings would keep missing variants, and your INSTRUMENTATION_FAILED-only output proves it. finalizeTaskAction now fails when the log carries neither OK ( nor FAILURES!!!, since a run that actually executed always produces one of them. The explicit matches are kept for the clearer message when tests genuinely failed.

Your negative test:

$ adb uninstall dev.testify.samples.flix
$ ./gradlew FlixTest:screenshotTest -x FlixSample:installDebug
> `am instrument` did not run any tests. The output has no test summary, which means the
  instrumentation could not start. Check that the application under test is installed and that
  testPackageId and testRunner are correct; run with `-Pverbose=true` to see the command and its
  full output.
BUILD FAILED

I also applied it to InternalScreenshotTestRecordTask, which overrode finalizeTaskAction to do nothing. A recording run still never fails because a screenshot differs — that is the point of it — but it now fails if nothing ran, which was the other half of your first-round finding.

One thing worth noting in favour of "OK (" over a stricter pattern: GmdSample:screenshotTest legitimately reports OK (0 tests) on this emulator, and it passes.

2. Namespace collision — confirmed and fixed

FlixTest is now dev.testify.samples.flix.screenshots, with a comment saying why, and the Bitrise test_package follows. All five packages coexist and both suites pass back to back, which they did not before:

$ ./gradlew FlixTest:screenshotTest      OK (1 test)
$ ./gradlew FlixSample:screenshotTest    OK (22 tests)
$ adb shell pm list packages | grep flix
dev.testify.samples.flix
dev.testify.samples.flix.library.test
dev.testify.samples.flix.screenshots      # was overwriting the line below
dev.testify.samples.flix.test

I left the Kotlin package as dev.testify.samples.flix.test — it is independent of the namespace and renaming it is directory churn with no effect on the APK id or the baseline. Say the word if you would rather they matched.

3. AGP 9 ignores applicationId on a test module

I implemented defaultConfig.applicationId ?: namespace and it does not compile: TestDefaultConfig is TestBaseFlavor, DefaultConfig and exposes no applicationId in the typed DSL.

Setting it from the Groovy DSL is accepted silently and then ignored. With applicationId "dev.testify.samples.flix.customid" in FlixTest's defaultConfig:

$ ./gradlew FlixTest:installDebug
BUILD SUCCESSFUL
$ adb shell pm list packages | grep -E "flix.(customid|screenshots)"
package:dev.testify.samples.flix.screenshots

output-metadata.json agrees — applicationId: dev.testify.samples.flix.screenshots. So reading defaultConfig.applicationId would report a package that is not installed, which is worse than reading the namespace.

I have kept the namespace and recorded the finding in the KDoc rather than leaving it as an unexplained choice. The docs sentence is qualified the other way round: it now says AGP does not let a test module choose a separate applicationId, and warns against reusing the application's id plus .test for exactly the collision in point 2.

If you have a case where a test module's APK does install under a configured applicationId, I will take the change — I could not produce one on AGP 9.3.1.

Verified locally

./gradlew Plugin:ktlintCheck Plugin:test Plugin:assemble   # BUILD SUCCESSFUL
cd docs && npm run build                                   # clean

On the emulator, confirming the inverted check did not make anything else brittle:

LegacySample   OK (88 tests)
FlixLibrary    OK (3 tests)
GmdSample      OK (0 tests)
FlixSample     OK (22 tests)
FlixTest       OK (1 test)

That also covers the red Legacy Sample check, which I cannot reproduce either; the new commit should re-run it.

[AndroidTestifyBot]

Approved

Both fixes hold and the applicationId finding is convincing.

Verified on an API 37 emulator, all Flix packages uninstalled first

$ ./gradlew FlixTest:screenshotTest      > :FlixSample:installDebug, :FlixTest:installDebug ... OK (1 test)
$ ./gradlew FlixSample:screenshotTest    OK (22 tests)
$ ./gradlew FlixTest:screenshotTest      OK (1 test)      # back to back, nothing clobbered
$ adb shell pm list packages | grep flix
dev.testify.samples.flix
dev.testify.samples.flix.screenshots
dev.testify.samples.flix.test
...

Negative cases, target uninstalled and its install bypassed:

$ ./gradlew FlixTest:screenshotTest   -x FlixSample:installDebug   THE SCREENSHOT TESTS DID NOT RUN!!!  BUILD FAILED
$ ./gradlew FlixTest:screenshotRecord -x FlixSample:installDebug   THE SCREENSHOT TESTS DID NOT RUN!!!  BUILD FAILED

Extending the check to InternalScreenshotTestRecordTask closes the record half of the original finding, which I hadn't asked for explicitly.

No regressions from the detection change

• LegacySample:screenshotTest: OK (88 tests).
• A filtered LegacySample:screenshotRecord -PtestClass=…: OK (2 tests), passes.
• Plugin:ktlintCheck Plugin:test Plugin:assemble pass. The docs build is clean. CI is fully green, including Legacy.

On point 3

Accepted. "The typed DSL doesn't expose it, the Groovy DSL silently ignores it, and the installed package and output-metadata.json both show the namespace" is a better standard of evidence than my claim. Recording that in the KDoc, and steering the docs away from <app id>.test, is the right outcome.

Keeping the Kotlin package as dev.testify.samples.flix.test is fine.

Merge note: #332 and this PR both edit ScreenshotTestTask.setDependencies. Whichever lands second needs a careful rebase, and the conflict won't only be in the CHANGELOG.