Hi all! My name is Taras Egorov; I'm an engineer at inDrive. I'll show you how we set up an infrastructure capable of running more than 5,000 tests per day on iOS and Android devices combined. The secret is simple: we used Selenoid. Preface Last year, our colleagues conducted a study of auto testing, and we took part in a survey as part of the study. We were pleased with the survey's findings, so we decided to write an article to share our experience with you and get some advice in return. We thought it a good idea to divide the material into two parts: the first one focusing on Android and the second on iOS. Let's begin with Android. Running the tests on Android Selenoid is a tool for running and managing browsers and Android emulators in a Docker container. You can read more about this . popular in the documentation To write Appium tests, we use: Kotlin; JUnit 5; Maven. The first run. Setting up Selenoid Create a config file: browsers.json {

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        "image": "browsers/android:10.0",

        "port": "4444",

        "path": "/wd/hub"

      }

    }

  }

} The emulator image is specified in . The guys from have prepared ready-made images of Android emulators. You can check them out or . The images don't differ from each other in any way. image aerokube here here Let’s take the image as an example. The image must be downloaded beforehand: , otherwise the tests will not run: browsers/android:10.0 docker pull browsers/android:10.0 Original error: create container: Error response from daemon: No such image: browsers/android:10.0 The next step is to run Selenoid. We do that directly via , or there’s the option of using . docker Configuration Manager docker run -d \

           -v /var/run/docker.sock:/var/run/docker.sock \

           -v "$(pwd)/selenoid/config/":/etc/selenoid/:ro \

           -p 4444:4444 \

           --name selenoid \

           aerokube/selenoid:1.10.7 You can check to see if Selenoid is working properly by following the link in your browser: http://localhost:4444 You are using Selenoid 1.10.7! Specify the Selenoid address in the Appium tests in the driver: ...
val driver = AndroidDriver(URL("http://localhost:4444/wd/hub"), capabilities)
... Next, specify the link to the build in capabilities: ... 
capabilities.setCapability("appium:app", "https://storage.example.com/builds/app.apk")
... If unable to provide a link, you can specify the path to the build:** ...
capabilities.setCapability("appium:app", "/builds/app.apk")
... Where is the path inside the container where the emulator is being run. For this option to work properly, be sure to specify the in : /builds/app.apk volumes browsers.json {

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        ...

        "volumes": [

          "/home/username/app.apk:/builds/app.apk:ro"

        ]

        ...

      }

    }

  }

} Where is the path to the build on the host platform. /home/username/app.apk That's it, we've almost set up Selenoid, and now we can try running the tests: ./mvnw test But, unfortunately, the tests will not be able to run. Let's look into this and see what's wrong. The second run. Taking a look at the logs and video The first thing to do after a failed startup is to check out the Selenoid logs: docker logs selenoid [INIT] [Loading configuration files...]
[INIT] [Loaded configuration from /etc/selenoid/browsers.json]
[INIT] [Video Dir: /opt/selenoid/video]
[INIT] [Your Docker API version is 1.41]
[INIT] [Timezone: UTC]
[INIT] [Listening on :4444]
[NEW_REQUEST] [unknown] [172.17.0.1]
[NEW_REQUEST_ACCEPTED] [unknown] [172.17.0.1]
[LOCATING_SERVICE] [android] [10.0]
[USING_DOCKER] [android] [10.0]
[CREATING_CONTAINER] [selenoid/android:10.0]
[STARTING_CONTAINER] [selenoid/android:10.0] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[CONTAINER_STARTED] [selenoid/android:10.0] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa] [0.40s]
[0] [REMOVING_CONTAINER] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[0] [CONTAINER_REMOVED] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[0] [SERVICE_STARTUP_FAILED] [http://172.17.0.3:4444/wd/hub does not respond in 30s] We see that the status is SERVICE_STARTUP_FAILED. Go to the and look at the status value: documentation SERVICE_STARTUP_FAILED - Failed to start Docker container or driver binary The error doesn’t tell you much, and more information is required. It would be good to take a look at the container logs. Let's do that by enabling logging: docker run -d \

           -p 4444:4444 \

           -v /var/run/docker.sock:/var/run/docker.sock \

           -v "$(pwd)/selenoid/config/":/etc/selenoid/:ro \

           -v  "$(pwd)/selenoid/logs/":/opt/selenoid/logs/ \

           aerokube/selenoid:1.10.7 \

           -log-output-dir /opt/selenoid/logs We also enable logs in the Capabilities section: ...
capabilities.setCapability("enableLog", true)
... Run the tests and review the logs using the browser : http://localhost:4444/logs/ 2023-04-16T13:44:43.909768530Z Waiting X server...
2023-04-16T13:44:44.009494775Z Logging to: /tmp/fluxbox.log
2023-04-16T13:44:44.047587277Z Waiting X server...
2023-04-16T13:44:44.151933325Z Waiting X server...
2023-04-16T13:44:44.262850410Z * daemon not running; starting now at tcp:5037
2023-04-16T13:44:44.457972956Z * daemon started successfully
2023-04-16T13:44:44.458249266Z adb: no devices/emulators found
2023-04-16T13:44:45.463480812Z adb: no devices/emulators found
2023-04-16T13:44:46.471547723Z adb: no devices/emulators found
2023-04-16T13:44:47.476093515Z adb: no devices/emulators found
2023-04-16T13:44:48.481987351Z adb: no devices/emulators found
2023-04-16T13:44:49.486503149Z adb: no devices/emulators found
2023-04-16T13:44:50.492757801Z adb: no devices/emulators found
2023-04-16T13:44:51.499094108Z adb: no devices/emulators found
2023-04-16T13:44:52.505862428Z adb: no devices/emulators found
2023-04-16T13:44:53.513276412Z adb: no devices/emulators found
2023-04-16T13:44:54.520642210Z adb: no devices/emulators found
2023-04-16T13:44:55.527420189Z adb: no devices/emulators found
2023-04-16T13:44:56.534631013Z adb: no devices/emulators found
2023-04-16T13:44:57.316094939Z WARNING. Using fallback path for the emulator registration directory.
2023-04-16T13:44:57.335415397Z checkValid: hw configs not eq
2023-04-16T13:44:57.541959741Z adb: device offline
2023-04-16T13:44:58.547907700Z adb: device offline
2023-04-16T13:44:58.565504866Z emulator: WARNING: System image is writable
2023-04-16T13:44:58.565528396Z emulator: Cold boot: different AVD configuration
2023-04-16T13:44:58.565532576Z Your emulator is out of date, please update by launching Android Studio:
2023-04-16T13:44:58.565536346Z  - Start Android Studio
2023-04-16T13:44:58.565539506Z  - Select menu "Tools > Android > SDK Manager"
2023-04-16T13:44:58.565543076Z  - Click "SDK Tools" tab
2023-04-16T13:44:58.565546216Z  - Check "Android Emulator" checkbox
2023-04-16T13:44:58.565549216Z  - Click "OK"
2023-04-16T13:44:58.565552146Z 
2023-04-16T13:44:59.554451514Z adb: device offline
2023-04-16T13:45:00.560926060Z adb: device offline
2023-04-16T13:45:01.568777440Z adb: device offline
2023-04-16T13:45:12.124226047Z emulator: INFO: boot completed
2023-04-16T13:45:12.124251007Z emulator: INFO: boot time 27848 ms
2023-04-16T13:45:12.124255077Z emulator: Increasing screen off timeout, logcat buffer size to 2M.
2023-04-16T13:45:12.152557294Z emulator: Revoking microphone permissions for Google App. The container logs are not much help here either, because you can't see the Appium logs. Now let's try to enable them. To do this, let's look at the script : entry point.sh ...
if [ -z "$VERBOSE" ]; then
    APPIUM_ARGS="$APPIUM_ARGS --log-level error"
else
    EMULATOR_ARGS="$EMULATOR_ARGS -verbose"
fi
... To enable Appium logs, the parameters and : must be passed to the container: VERBOSE=true APPIUM_ARGS=--log-level debug {

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        ...

        "env": [

          "VERBOSE=true",

          "APPIUM_ARGS=--log-level debug"

        ]

        ...

      }

    }

  }

} To enable Appium to debug logs, you need to pass VERBOSE; in this case, the emulator logs turn on and start filling the "ether"” But we fixed that for future images =) Now it is enough to pass to . APPIUM_ARGS=-log-level debug ...
[HTTP] --> POST /wd/hub/session/c89fa9c2-ca2b-49cd-ab31-590eeccf77d1/element
[HTTP] {"using":"id","value":"authorization_edittext_phone"}
[debug] [W3C (c89fa9c2)] Calling AppiumDriver.findElement() with args: ["id","authorization_edittext_phone","
c89fa9c2-ca2b-49cd-ab31-590eeccf77d1"]
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -and
roid uiautomator
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [WD Proxy] Matched '/element' to command name 'findElement'
[debug] [WD Proxy] Proxying [POST /element] to [POST http://127.0.0.1:8200/wd/hub/session/65943f03-3b35-4d3eb221-d6dc7988f935/element] with body: {"strategy":"id","selector":
"authorization_edittext_phone","context":"","multiple":false}
[WD Proxy] Got response with status 404: {"sessionId":"65943f03-3b35-4d3e-b221-d6dc7988f935","value":{"error"
:"no such element","message":"An element could not be located on the page using the given search parameters","stacktrace":"io.appium.uiautomator2.common.exceptions.El
ementNotFoundException: An element could not be located on the page using the given search parameters\n\tat io.appium.uiautomator2.handler.FindElement.safeHandle(Find
Element.java:73)\n\tat io.appium.uiautomator2.handler.request.SafeRequestHandler.handle(SafeRequestHandler.java:41)\n\tat io.appium.uiautomator2.server.AppiumServlet.
handleRequest(AppiumServlet.java:253)\n\tat io.appium.uiautomator2.server.AppiumServlet.handleHttpRequest(AppiumServlet.java:247)\n\tat io.appium.uiautomator2.http.Se
rverHandler.channelRead(ServerHandler.java:68)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)\n\tat io
.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.chann...
[debug] [W3C] Matched W3C error code 'no such element' to NoSuchElementError
[debug] [W3C (c89fa9c2)] Encountered internal error running command: NoSuchElementError: An element could not
be located on the page using the given search parameters.
[debug] [W3C (c89fa9c2)]     at AndroidUiautomator2Driver.findElOrEls (/opt/node_modules/appium/node_modules/appium-android-driver/lib/commands/find.js:75:11)
[debug] [W3C (c89fa9c2)]     at process._tickCallback (internal/process/next_tick.js:68:7)
[HTTP] <-- POST /wd/hub/session/c89fa9c2-ca2b-49cd-ab31-590eeccf77d1/element 404 23 ms - 444
... As seen from the logs, Appium cannot find our element. Let's see what is happening on the emulator screen. To do this, we have to run the Selenoid UI: docker run -d \

           --name selenoid-ui \

           -p 8080:8080 \

           --link selenoid:selenoid \

           aerokube/selenoid-ui:1.10.4 \

           --selenoid-uri "http://selenoid:4444" Go to and open the Selenoid UI: http://0.0.0.0:8080 Be sure to enable VNC and video recording in the tests: ...

capabilities.setCapability("enableVNC", true)

capabilities.setCapability("enableVideo", true)

... The Selenoid startup command ends up looking like this: docker run -d \

-v /var/run/docker.sock:/var/run/docker.sock \

-v "(pwd)/selenoid/logs/":/opt/selenoid/logs/ \

-v /opt/selenoid/video/:/opt/selenoid/video/ \

-e OVERRIDE_VIDEO_OUTPUT_DIR="/opt/selenoid/video/" \

-p 4444:4444 \

-name selenoid \

aerokube/selenoid:1.10.7 \

-log-output-dir /opt/selenoid/logs Open the Selenoid UI once the tests are up and running: And here's . a video of it https://www.youtube.com/watch?v=Hm7s5ZjnVYQ&embedable=true We found the cause of the startup error. Great! Let’s move on. The third run. Building an emulator image As it turns out, the images of Selenoid emulators won’t work without Google Play services. To remedy this situation, you must build an emulator image yourself. The guys from aerokube have assembled everything you need for this: and . a repository with photos documentation the repository. Downloading Go to the folder. selenium Run the script and answer the questions. This is what it looks like in our case: ./automate_android.sh Specify Appium version: [1.18.1]

>> 1.18.1

Specify Android image type (possible values: "default", "google_apis", "google_apis_playstore", "android-tv", "android-wear"): [default]

>> google_apis

Specify Application Binary Interface (possible values: "armeabi-v7a", "arm64-v8a", "x86", "x86_64"): [x86]

>> x86

Specify Android version: [8.1]

>> 10.0

Specify device preset name if needed (e.g. "Nexus 4"):

>>

Specify SD card size, Mb: [500]

>>

Specify userdata.img size, Mb: [500]

>>

Are you building a Chrome Mobile image (for mobile web testing): [n]

>> y

Specify Chromedriver version if needed (required for Chrome Mobile):

>> 74.0.3729.6

Specify image tag: [selenoid/chrome-mobile:74.0]

>> android-emulator:10.0

Add Android quick boot snapshot? [y]

>> n When I saw the question Add Android quick boot snapshot?, I thought it was an emulator . But if you look at the , it calls a , which is needed to install APK applications. Basically, it doesn't give us any gain. snapshot code script We already use snapshot emulators , but we'll talk about that in other articles. to speed up container launches Once the image is built, we will be prompted to push it into the registry. Since we don't need that yet, we will kindly decline the offer: Push?  >> n We have put together a build with Google Play services. Remember to change the image parameter in browsers.json, and restart Selenoid. Now let's try rerunning the tests: [INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------ And of the test run. here’s a video https://www.youtube.com/watch?v=w-uzB7hg7g8&t=1s&embedable=true To sum up What we have done: Configured Selenoid to run Android tests. Learned how to check out the logs and videos to troubleshoot. Built our own emulator image with Google Play services. Something else I would like to tell you about: Selenoid timeouts. If your app is large, you might run into problems with timeouts. How we tried making the container run faster. The video shows that the run takes about a minute or so. How we tried running native Android tests written in Espresso on Selenoid. Spoiler alert: this works! Meanwhile, in the next part, we'll tell you how we scaled up the infrastructure and ran tests on iOS.

How to Run 5000+ Tests on Mobile Devices Everyday; From inDrive's Playbook (Part 1)

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI in Product Design: Three Practical Cases From inDrive

105 Stories To Learn About Ios App Development

10 Things to check on Android code reviews

10 good rules for bad App

The Noonification: Accelerating Android Application Development With the ADB Idea Plugin (10/12/2022)

AI in Product Design: Three Practical Cases From inDrive

105 Stories To Learn About Ios App Development

10 Things to check on Android code reviews

10 good rules for bad App

The Noonification: Accelerating Android Application Development With the ADB Idea Plugin (10/12/2022)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps