Android Studio: Fix “@Ignore Tests Become <failure></failure>” in AGP 9.4 JUnit XML

After upgrading to Android Gradle Plugin (AGP) 9.4, instrumented tests annotated with JUnit4 @Ignore can be written to JUnit XML as empty <failure></failure> elements instead of <skipped />. The Gradle task succeeds, yet CI dashboards and report parsers show failures. Google tracks this regression as Issue #563732801 and lists it as fixed in AGP 9.5.0-alpha08.

Representative incorrect JUnit XML
Cause
Affected conditions and versions
Recommended resolution
Alternatives when the fixed version cannot be adopted
How to verify the fix
Cautions
Summary
Official references

Representative incorrect JUnit XML

<testsuite tests="3" failures="3" skipped="0">
  <testcase name="ignoredTest" classname="ExampleInstrumentedTest">
    <failure></failure>
  </testcase>
</testsuite>

The expected representation is:

<testsuite tests="3" failures="0" skipped="3">
  <testcase name="ignoredTest" classname="ExampleInstrumentedTest">
    <skipped />
  </testcase>
</testsuite>

Key signature: The test task exits successfully, but each @Ignore test receives an empty <failure></failure> element and increases the suite’s failure count. This differs from a real assertion failure or exception.

Cause

AGP 9.4 introduced a regression in the conversion of Android test results to JUnit XML. An ignored test status was not mapped to the XML skipped representation and was serialized as a failure instead. Device-side execution can therefore succeed while downstream CI systems interpret the generated report as failed.

The annotation itself and the CI XML parser are not necessarily defective. If the same tests were reported as skipped before the AGP 9.4 upgrade, this known result-conversion regression is a strong match.

Affected conditions and versions

Android Gradle Plugin 9.4 is used.
JUnit4 instrumented tests include @Ignore.
A CI service or report tool consumes AGP-generated JUnit XML.
The Gradle task succeeds while only the XML failure count is wrong.

The official issue title explicitly identifies AGP 9.4 as the regression. Google’s Android Studio Rabbit 2 Canary 3 closed-issues page lists #563732801 as fixed in the release shipped with AGP 9.5.0-alpha08.

Recommended resolution

1. Verify the effective AGP version

Check the root build.gradle(.kts), settings.gradle(.kts), or gradle/libs.versions.toml.

[versions]
agp = "9.4.0"

Convention plugins and composite builds can declare another version, so verify that the expected plugin is actually resolved.

2. Upgrade to AGP 9.5.0-alpha08 or later

Where preview tooling can be evaluated, use AGP 9.5.0-alpha08 or a later version containing the fix.

[versions]
agp = "9.5.0-alpha08"

[plugins]
android-application = { id = "com.android.application", version.ref = "agp" }
android-library = { id = "com.android.library", version.ref = "agp" }

Confirm the required Gradle Wrapper, JDK, and Android Studio versions in the official compatibility information. Test a Canary or alpha upgrade in a branch before changing production CI.

3. Rerun the same tests and inspect the XML

./gradlew --stop
./gradlew clean
./gradlew connectedDebugAndroidTest --stacktrace

Replace the task name when the module or variant differs. Inspect build/outputs/androidTest-results/ or the result directory configured by the project.

Alternatives when the fixed version cannot be adopted

Temporarily return to the last verified AGP version

For stable-only CI pipelines, return to the project version that produced correct XML before the AGP 9.4 upgrade. Keep the Gradle Wrapper and related plugins within that AGP version’s compatibility range.

Record task status separately from report status

Until the upgrade is possible, preserve the Gradle exit code, device-side result, and generated XML as separate artifacts. This helps distinguish a real test failure from the report-generation regression. Do not globally ignore XML failures, because that can hide legitimate defects.

Use XML post-processing only as a narrow temporary workaround

A script could replace an empty failure with a skipped element only for known @Ignore test cases. A broad replacement is unsafe because a genuine failure can also have an empty message. If post-processing is unavoidable, match an explicit allowlist of test names and retain the original XML and execution log.

How to verify the fix

Create or identify one small test annotated with @Ignore.
Run the affected instrumented-test task.
Confirm that Gradle exits with status 0.
Verify that the testcase contains <skipped />.
Confirm that failures does not increase and skipped does.
Check that the CI report also labels the test as skipped.

Cautions

A <failure> containing a stack trace for a non-ignored test should be investigated as a real failure.
Deleting caches does not repair AGP 9.4’s XML conversion logic.
Similar behavior involving JUnit5 @Disabled or local unit tests is not automatically the same issue.
AGP 9.5.0-alpha08 contains the confirmed fix but is a preview release; apply normal release validation.

Summary

If AGP 9.4 writes an @Ignore test as <failure></failure>, a known JUnit XML conversion regression is the likely cause. Upgrade to AGP 9.5.0-alpha08 or later. If that is not yet practical, use the last verified AGP version and keep execution status separate from the generated report rather than suppressing all CI failures.