Running our Android Espresso tests in Parallel

5 min readOct 27, 2023

We had the ability to run our iOS test cases in parallel but kept eagerly waiting for similar functionality to become available for Android. It is still fairly new (thus some headaches to overcome 😆) but it is finally on its way to becoming a reality! 🎉. I’d like to discuss some things we have figured out and some hurdles we encountered.

Parallel testing allows you to run your test cases across multiple device simulators/emulators at once instead of running them sequentially on one particular device simulator/emulator. This can greatly reduce the overall run time of your automation test suites! Example: Instead of running 10 test cases one after another on an a single Pixel 4 emulator, parallel testing enables us to split (a.k.a shard) those 10 test cases across several Pixel 4 emulators. If you spawn up 3 emulators then it may run 4 test cases on 1 of the emulators and 3 test cases on each of the others.

Getting Started

We got the ability to use gradle managed devices around the time of Android Studio versions Dolphin/Eel. I recommend using at least Android Studio version Giraffe or newer though and make sure you get the latest emulator versions. This can be done by going to Android Studio > Check For Updates…

⚠️ Before starting make sure you have a beefy enough machine to run on because Android likes to use a lot of memory. I recommend having a machine with at least 32 GB if you want to see successful results. We are in the process of getting ourselves a few 64GB machines to use. We run our tests on API 30+. API 27+ is supposed to be supported though, although for 29, I kept having out of memory issues. If you are looking for more stable runs i’d recommend 30+.

If you encounter memory issues you can also try updating the projects gradle.properties file to have org.gradle.jvmargs=-Xmx4g -XX:MaxMetaspaceSize=512m -XX:MaxPermSize=512m

Suites & Categories

Determine what tests to include in your parallel runs. If you only want a subset of tests to run or all tests to run or all but a few…. one way you can control this is by making use of Suites and the Category annotation.

@RunWith(Categories::class)
@SuiteClasses(
 TestClassOne::class,
 TestClassTwo::class
)
class CITestSuite

You can make a class file that contains the names of all the class files you want run when you run this Suite.

To exclude some tests, you can use ExcludeCategory as shown here:

@RunWith(Categories::class)
@Categories.ExcludeCategory(ExcludedTests::class)
@SuiteClasses(
     TestClassOne::class,
     TestClassTwo::class
)
class CITestSuite

Then that test would contain that annotation:

@Test
@Category(ExcludedTests::class)
fun myTestCase() {
...
}

So when the Suite CITestSuite is ran, it would run all tests listed except those with this excluded annotation. We will come back to using this in a little.

Configuring the Managed Devices

In your app build.gradle file, add the managedDevices section within testOptions

android.testOptions {
    animationsDisabled = true

    managedDevices {
        devices {
            pixel4api33(ManagedVirtualDevice) {
                device = "Pixel 4"
                apiLevel = 33
                systemImageSource = "google"
                require64Bit = true
            }

            pixel4api32(ManagedVirtualDevice) {
                device = "Pixel 4"
                apiLevel = 32
                systemImageSource = "google"
                require64Bit = true
            }

            pixel4api31(ManagedVirtualDevice) {
                device = "Pixel 4"
                apiLevel = 31
                systemImageSource = "google"
                require64Bit = true
            }
        }
    }
}

Specify the device, api, systemImageSource, and require64Bit flag. ATD images are supposed to be faster however don’t miss this note 👀 // ATDs currently support only API level 30 found within the section Run tests using Automated Test Devices.

Getting Ready to Run 🏃 ….

One thing I noticed when running the gradle managed devices is that sometimes Java processes are spun up and not closed upon test completion. You can view this by opening up the Activity Monitor application on your Mac. So before starting a run, there are a few ‘cleanup’ things I do to make sure the tests are in a ready state to be ran.

Open Activity Monitor and kill any Java processes running
Quit all applications possible to free up as much memory as you can, including Android Studio and any emulators you had running within.
In a terminal window for your project, run the following

./gradlew clean
./gradlew cleanManagedDevices
./gradlew --stop

Takeoff! 🚀

Here is the command you can use to actually kick off the tests to run:

./gradlew pixel4api30DebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.xyz.Suites.CITestSuite -Pandroid.experimental.androidTest.numManagedDeviceShards=2 --enable-display --info -Pandroid.testoptions.manageddevices.emulator.gpu="swiftshader_indirect"

-Pandroid.testInstrumentationRunnerArguments.class: this allows you to specify that Suite that we created earlier and will thus run all those classes/tests.
-Pandroid.experimental.androidTest.numManagedDeviceShards: this lets you specify the number of emulators you want to shard the tests across.
— — enable-display: by default the emulators are not viewable so if you want to watch the emulators as the tests run, pass this flag.
— — info: this is helpful if you are running into issues and want more info logged to the terminal
-Pandroid.testoptions.manageddevices.emulator.gpu=”swiftshader_indirect”: see note

I have seen these 2 options also mentioned in various issues or articles but I haven’t had to personally use them yet or know when they are exactly they should be used. But if you have feedback on them, please feel free to add info in the comments! Sometimes half the battle with Android is just figuring out what exists though so just wanted to mention 😅

-Pandroid.experimental.testOptions.managedDevices.setupTimeoutMinutes=180
-Dorg.gradle.workers.max=1

Results 🏆

Within your project repo, it will generate an output file at build > outputs > androidTest-results > managedDevice. You should find a test-result.pb file. Within Android Studio, you can import these tests results by going to Run > Import Tests from File. Navigate to and select that pb file that was generated.

You can open the index.html that gets generated at build > reports > androidTests > managedDevice > allDevices > index.html.

Open Issues / Troubleshooting

Even with the above working, sometimes, we still have some open issues that we are working thru and hoping that these issues will get fixed with newer emulators versions or android gradle plugin updates. I’ll try to update this with new open issues as I can but here are some of the current items:

Other things that may, or may not, help if you run into issues:

Running ./gradlew cleanManagedDevices command should clear the devices at .android/avd/gradle-managed
When you create/delete emulators from Android Studio Device Manager — you should see the respective action be applied to .android/avd folder
Sometimes the daemon for gradle aren’t getting removed or get hung up. You can clear these by deleting them at .gradle/daemon/<version>
Within Android Studio, File > Invalidate Caches.