Continuous integration (CI) is the practice of merging all developers’ working copies to shared mainline several times a day.
One of the most important aspects of CI is keeping the mainline healthy. That’s why it’s crucial to have proper tests, both unit and UI, that can run against any PR and/or commit to the mainline. At the same time, these tests should also be fast. It’s usually not a problem with unit tests, but UI tests, especially if you want to run them on multiple devices, can take a rather long time.
As a balance between speed and reliability, for PRs into the mainline, we use smoke UI tests that only prove the main functionality is working. Proper UI testing with manual QA verification is done only when we’re preparing a new release candidate.
In this article, I will show you how we’ve set up Azure Pipelines for running automated UI smoke tests for a Flutter app.
Unfortunately, Azure Pipelines don’t provide agents with Flutter pre-installed, and, as we use Microsoft-hosted agents, we need to perform a bunch of actions on each job run (you can avoid these steps by setting up and using self-hosted agents). These actions are simpler to invoke as bash scripts, so almost every step in pipeline configuration just delegates the work to some script.
We keep these scripts in a separate file:
#!/bin/sh
set -e
export PATH=$BUILD_SOURCESDIRECTORY/flutter/bin:$BUILD_SOURCESDIRECTORY/flutter/bin/cache/dart-sdk/bin:$PATH
# All scripts will be placed here
"$@"
We use the set -e option, so that each failed command will fail the job, then add the Flutter installation directory to $PATH.
"$@" allows us to use functions defined in this file in other scripts — if we have a function install_flutter() defined in this file, we can call it later with sh scripts.sh install_flutter.
Install Flutter
First, we need to install Flutter:
install_flutter() {
git clone -b stable https://github.com/flutter/flutter.git
flutter precache
yes | $ANDROID_HOME/tools/bin/sdkmanager --licenses
flutter doctor
}
Here we download a stable branch of Flutter, install it, accept all licenses, and print doctor output (you can skip this, but it can be useful to check when dealing with certain problems).
Install AVD
Next, install and launch AVD:
launch_avd() {
echo "Installing SDK"
$ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-29;default;x86'
echo "Creating emulator"
$ANDROID_HOME/tools/bin/avdmanager create avd -n "pixel" --device "pixel" -k "system-images;android-29;default;x86"
echo "Starting emulator"
$ANDROID_HOME/emulator/emulator -avd "pixel" -no-snapshot &
$ANDROID_HOME/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed | tr -d '\r') ]]; do sleep 1; done; input keyevent 82'
echo "Emulator started"
}
We use Android SDK 29 without Google APIs. You can use a different SDK, just replace 'system-images;android-29;default;x86' with the other option. You can get a full list of options by running sdkmanager --list | grep system-images | sort | uniq.
Screen recording
In Google Play automated tests there’s a cool feature: it records the screen while running tests, so you can download and check it later. It’s a good idea to have it in our tests as well. For that, we can use the built-in screenrecord utility. Unfortunately, it has a hard-coded max length of three minutes. So we need a hack to fix this:
start_recording() {
# Each video is 3 minutes max, so 5 videos will give us up to 15 minutes,
# should be enough for test
$ANDROID_HOME/platform-tools/adb shell mkdir /sdcard/video
$ANDROID_HOME/platform-tools/adb shell screenrecord /sdcard/video/1.mp4
$ANDROID_HOME/platform-tools/adb shell screenrecord /sdcard/video/2.mp4
$ANDROID_HOME/platform-tools/adb shell screenrecord /sdcard/video/3.mp4
$ANDROID_HOME/platform-tools/adb shell screenrecord /sdcard/video/4.mp4
$ANDROID_HOME/platform-tools/adb shell screenrecord /sdcard/video/5.mp4
}
No worries, this doesn’t mean that all five records will be created with each job — we’ll stop recording once tests are done.
Run tests
Now we’re ready to create a script that will build and run tests:
flutter_test() {
flutter packages get
flutter packages pub run build_runner build
flutter test
launch_avd
start_recording &
flutter drive --target=test_driver/app.dart
pkill -f screenrecord || true
}
If you don’t have any code generation in your project, remove the build_runner action.
This script runs unit tests with flutter test and UI tests with flutter drive --target=test_driver/app.dart. As you can see here, before running UI tests we launch AVD and start video recording in the background. And after the test is done, we kill the running screenrecord command. Thanks to the set -e option defined in the script, it will also end the start_recording command.
Download record
We also need a helper script for getting the record out of the device:
pull_video() {
$ANDROID_HOME/platform-tools/adb pull /sdcard/video $BUILD_SOURCESDIRECTORY/screenshots
}
If you provide a directory as the pull argument it will download all files in that directory.
Pipeline configuration
Now let’s create a pipeline configuration file putting everything together:
trigger: none
pr:
- master
jobs:
- job: Test
timeoutInMinutes: 20
pool:
vmImage: "macOS-latest"
steps:
- script: pipelines/scripts.sh install_flutter
displayName: Install Flutter
- script: pipelines/scripts.sh flutter_test
displayName: Test app
- script: ../pipelines/scripts.sh pull_video
displayName: Pull screen video record
condition: always()
- task: PublishBuildArtifacts@1
displayName: Publish screenshots
condition: always()
inputs:
pathtoPublish: "$(System.DefaultWorkingDirectory)/screenshots/"
artifactName: screenshots
If you’re not sure where to put this file, check this tutorial.
What about the UI test itself? Let’s create a basic smoke test that allows us to check the app state (you can find a detailed tutorial on the official site).
First, create a test_driver folder and put an app.dart file into it with the following content:
import 'package:commander/main.dart' as app;
import 'package:flutter_driver/driver_extension.dart';
void main() {
enableFlutterDriverExtension();
app.main();
}
This file contains an "instrumented" version of the app. While it can have any name that makes sense, for the sake of simplicity we just call it app.dart.
Now let’s create a file that contains the test suite which drives the app and verifies that it works as expected. The name of the test file must correspond to the name of the file that contains the instrumented app, with _test added at the end. In our case, it means that name should be app_test.dart:
import 'dart:io';
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';
import 'app_test_resources.dart';
void main() {
group('Smoke tests', () {
FlutterDriver driver;
// Connect to the Flutter driver before running any tests.
setUpAll(() async {
driver = await FlutterDriver.connect();
await Directory('screenshots').create();
});
// Close the connection to the driver after the tests have completed.
tearDownAll(() async {
if (driver != null) {
await driver.close();
}
});
test('Commander Walkthrough', () async {
await loginToCommander(driver);
await goToMessages(driver);
await goToTasks(driver);
await goToProfile(driver);
await goToRooms(driver);
}, timeout: const Timeout(Duration(minutes: 1)));
});
}
As you can see, in the setUpAll function we create screenshots directory — we will put screenshots there if needed (and screen recording as well).
All the helper functions, such as loginToCommander or goToRooms are defined in the app_test_resources.dart file, e.g.:
Future<void> goToRooms(FlutterDriver driver) async {
await driver.tap(find.byValueKey('rooms'));
await driver.assertElementPresent('sliverList');
await driver.assertElementPresent('floorHeader');
}
We also have a couple of extensions for the driver:
import 'dart:io';
import 'package:flutter_driver/flutter_driver.dart';
extension DriverExt on FlutterDriver {
Future<void> saveScreenshot() async {
final path = 'screenshots/'
'${DateTime.now().toIso8601String().replaceAll(':', '-')}'
'.png';
final List<int> pixels = await screenshot();
final file = File(path);
await file.writeAsBytes(pixels);
}
Future<void> assertElementPresent(
String valueKey, {
Duration timeout = const Duration(seconds: 5),
}) async {
try {
return await waitFor(find.byValueKey(valueKey), timeout: timeout);
// ignore: avoid_catching_errors
} on DriverError catch (_) {
await saveScreenshot();
rethrow;
}
}
}
We replace “:” with “-” in the file name because Azure Pipelines fails to publish a file to artifacts if it contains “:” in the name.
saveScreenshot is called automatically when the driver fails to find an element. In that case, we want to be able to look at a screenshot and figure out what the problem is. It can also be called manually if we want to screenshot the app at some checkpoint.
Here’s the downloaded video with screen recording:
That’s it for running our UI tests on an Android emulator. Although, despite the cross-platform nature of Flutter, it could be useful to run these smoke tests on an iOS simulator as well, right? But that’s a topic for another article. Stay tuned ;)