Develop Reference

Android emulator fails on Gitlab CI

I'm trying to run android instrumented tests on Gitlab CI on a remote linux machine, everything works fine until it runs the gradle task for the tests, it fail without any further information.
I should also mention that one out of ten times, it works.
Can you spot something wrong?
Here is my gitlab CI file:
variables:
ANDROID_COMPILE_SDK: "30"
ANDROID_BUILD_TOOLS: "30.0.3"
ANDROID_SDK_TOOLS: "6858069"
ANDROID_APK_FOLDER: "app/build/outputs/apk"
EMULATOR_VERSION: "24"
ENVIRONMENT: Staging
before_script:
- export ANDROID_HOME=$PWD/android-sdk
- export ANDROID_SDK_ROOT=${ANDROID_HOME}
- export ANDROID_TOOLS_PATH=${ANDROID_HOME}/platform-tools
- export ANDROID_SDK_MANAGER=${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager
- export ANDROID_AVD_MANAGER=${ANDROID_HOME}/cmdline-tools/latest/bin/avdmanager
- export ANDROID_EMULATOR_HOME=${ANDROID_HOME}/emulator
- export ANDROID_EMULATOR_PATH=${ANDROID_HOME}/avd
- export PATH="$PATH:${ANDROID_TOOLS_PATH}"
- export ADB_INSTALL_TIMEOUT=10
# Update system
- apt-get --quiet update --yes
- apt-get --quiet install --yes wget tar unzip lib32stdc++6 lib32z1 tree libx11-dev libpulse0 libgl1 libnss3 libxcomposite-dev libxcursor1 libasound2
# Download Android SDK
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_TOOLS}_latest.zip
- mkdir ${ANDROID_HOME}
- unzip -d ${ANDROID_HOME}/cmdline-tools android-sdk.zip
- echo y | rm android-sdk.zip
- mv ${ANDROID_HOME}/cmdline-tools/cmdline-tools ${ANDROID_HOME}/cmdline-tools/latest
# Install Android Build Tools
- echo y | ${ANDROID_SDK_MANAGER} --sdk_root=${ANDROID_HOME} --licenses
- echo y | ${ANDROID_SDK_MANAGER} --sdk_root=${ANDROID_HOME} --update > update.log
- echo y | ${ANDROID_SDK_MANAGER} --sdk_root=${ANDROID_HOME} "platforms;android-${ANDROID_COMPILE_SDK}" "build-tools;${ANDROID_BUILD_TOOLS}" "extras;google;m2repository" "extras;android;m2repository" > installPlatform.log
# Create untracked files
- echo -e "android.useAndroidX=true\nandroid.enableJetifier=true" > gradle.properties
- echo ${RELEASE_KEYSTORE} | base64 -d > app/upload.jks
stages:
- setup
- e2e_test
setup:
stage: setup
script:
- chmod +x ./gradlew
- ./gradlew clean
############################################## TESTS ###############################################
e2e_test:
stage: e2e_test
only:
- develop
- tags
script:
- wget --quiet --output-document=android-wait-for-emulator.sh https://raw.githubusercontent.com/travis-ci/travis-cookbooks/0f497eb71291b52a703143c5cd63a217c8766dc9/community-cookbooks/android-sdk/files/default/android-wait-for-emulator
- chmod +x android-wait-for-emulator.sh
- echo y | ${ANDROID_SDK_MANAGER} --update > update.log
- echo y | ${ANDROID_SDK_MANAGER} "platform-tools" "emulator" "system-images;android-${EMULATOR_VERSION};default;armeabi-v7a" > installEmulator.log
- echo no | ${ANDROID_AVD_MANAGER} --verbose create avd -n test --force -k "system-images;android-${EMULATOR_VERSION};default;armeabi-v7a"
- ${ANDROID_EMULATOR_HOME}/emulator -avd test -no-snapshot-save -no-audio -no-window -debug -verbose &
- ./android-wait-for-emulator.sh
- adb shell input keyevent 82
- ./gradlew connected${ENVIRONMENT^}DebugAndroidTest --stacktrace --continue || true
- adb devices | grep emulator | cut -f1 | while read line; do adb -s $line emu kill; done
And here is the error I get :
> Task :app:connectedStagingDebugAndroidTest
Unable to install /builds/onl-dev-team/retail_checkout_app/app/build/outputs/apk/staging/debug/app-staging-2.2.1.apk
com.android.ddmlib.InstallException
at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1316)
at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1136)
com.android.build.gradle.internal.testing.ConnectedDevice > runTests[test(AVD) - 7.0] FAILED
com.android.builder.testing.api.DeviceException: com.android.ddmlib.InstallException
at com.android.build.gradle.internal.testing.ConnectedDevice.installPackage(ConnectedDevice.java:133)
at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1112)
at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1101)
at com.android.build.gradle.internal.testing.ConnectedDevice.installPackage(ConnectedDevice.java:127)
at com.android.build.gradle.internal.testing.SimpleTestRunnable.run(SimpleTestRunnable.java:135)
at com.android.build.gradle.internal.tasks.Workers$ProfileAwareExecutorServiceAdapter$submit$submission$1.run(Workers.kt:165)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: com.android.ddmlib.ShellCommandUnresponsiveException
at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:691)
at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:470)
at com.android.ddmlib.internal.DeviceImpl.executeShellCommand(DeviceImpl.java:727)
at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1307)
... 12 more
[no message defined]

As explained in this post https://githubmemory.com/repo/mingchen/docker-android-build-box/issues/58,
at the startup, the android system undergoes a lot of processes and may be busy when trying to install the apk. This would cause for example a ShellCommandUnresponsiveException.
To solve this I just added as recommended a sleep 180 after the line with adb shell input keyevent 82

Gitlab CI - Android Sdk tools?

I use this Link for add gitlab CI in my android project. The CI work but I don't understand this part :
ANDROID_SDK_TOOLS: "4333796"
In the tutorial, android SDK tool is the last version available.
ANDROID_SDK_TOOLS is a little funny. It's what version of the command line tools we're going to download from the official site. So, that number really just comes from the latest version available there.
Installing packages.
But when click on the link, they send me in android studio download page.
How can i find the version code to put here ? Is it important to change it ?
I also tried with sdkmanager.bar --list in SDK > tool > bin directory
ERROR: JAVA_HOME is set to an invalid directory: D:\Software\JDK\openjdk-15_windows-x64_bin\jdk-15\bin
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

At the download page you'll find the "Command line tools only" section. The file name ends with the version number. As of now this is: 8092744 (it'll change in the future).
The name of the download URL and the zipped directory have been changed though, so you'll need to update your ci file accordingly. Using the template in the linked tutorial, the file could look like this (I only build a debug apk, you can add release versions too ofc, just like in the tutorial):
image: openjdk:8-jdk
variables:
ANDROID_COMPILE_SDK: "31"
ANDROID_BUILD_TOOLS: "29.0.2"
ANDROID_SDK_TOOLS: "8092744"
before_script:
- apt-get --quiet update --yes
- apt-get --quiet install --yes wget tar unzip lib32stdc++6 lib32z1
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_TOOLS}_latest.zip
- unzip -d android-sdk-linux android-sdk.zip
- echo y | android-sdk-linux/cmdline-tools/bin/sdkmanager --sdk_root=android-sdk-linux/cmdline-tools/latest/ "platforms;android-${ANDROID_COMPILE_SDK}" >/dev/null
- echo y | android-sdk-linux/cmdline-tools/bin/sdkmanager --sdk_root=android-sdk-linux/cmdline-tools/latest/ "platform-tools" >/dev/null
- echo y | android-sdk-linux/cmdline-tools/bin/sdkmanager --sdk_root=android-sdk-linux/cmdline-tools/latest/ "build-tools;${ANDROID_BUILD_TOOLS}" >/dev/null
- export ANDROID_HOME=$PWD/android-sdk-linux
- export PATH=$PATH:$PWD/android-sdk-linux/platform-tools/
- chmod +x ./gradlew
# temporarily disable checking for EPIPE error and use yes to accept all licenses
- set +o pipefail
- yes | android-sdk-linux/cmdline-tools/bin/sdkmanager --sdk_root=android-sdk-linux/cmdline-tools/latest/ --licenses
- yes | android-sdk-linux/cmdline-tools/bin/sdkmanager --sdk_root=/builds/pauni/lara/android-sdk-linux --licenses
- set -o pipefail
stages:
- build
assembleDebug:
stage: build
script:
- ./gradlew assembleDebug
artifacts:
paths:
- app/build/outputs/

How to cache libs at Pipelines?

here is my bitbucket-pipelines.yml file for android project:
# This is a sample build configuration for Java (Gradle).
# Check our guides at https://confluence.atlassian.com/x/zd-5Mw for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: java:8
pipelines:
branches:
'{master}':
- step:
caches:
- gradle
- android-sdk
script:
# Download and unzip android sdk
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip
- unzip -o -qq android-sdk.zip -d android-sdk
# Define Android Home and add PATHs
- export ANDROID_HOME="/opt/atlassian/pipelines/agent/build/android-sdk"
- export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/platform-tools:$PATH"
# Download packages.
- yes | sdkmanager "platform-tools"
- yes | sdkmanager "platforms;android-19"
- yes | sdkmanager "platforms;android-23"
- yes | sdkmanager "platforms;android-28"
- yes | sdkmanager "build-tools;27.0.3"
- yes | sdkmanager "build-tools;28.0.3"
- yes | sdkmanager "extras;android;m2repository"
- yes | sdkmanager "extras;google;m2repository"
- yes | sdkmanager "extras;google;instantapps"
- yes | sdkmanager --licenses
# Build apk
- echo “Start default step”
- # Add Android SDK license in the default file
- mkdir "${ANDROID_HOME}/licenses" || true
- echo "8933bad161af4178b1185d1a37fbf41ea5269c55" > $ANDROID_HOME/licenses/android-sdk-license
- echo "d56f5187479451eabf01fb78af6dfcb131a6481e" > $ANDROID_HOME/licenses/android-sdk-license
# Build the app
- chmod a+x gradlew
- ./gradlew assembleDebug
- echo "Amazing"
definitions:
caches:
android-sdk: android-sdk
gradle: gradle
But when I rerun the script, then it always downloads libs for build:
Downloading https://services.gradle.org/distributions/gradle-4.4-all.zip
..............................................................................................
Unzipping /root/.gradle/wrapper/dists/gradle-4.4-all/9br9xq1tocpiv8o6njlyu5op1/gradle-4.4-all.zip to /root/.gradle/wrapper/dists/gradle-4.4-all/9br9xq1tocpiv8o6njlyu5op1
Set executable permissions for: /root/.gradle/wrapper/dists/gradle-4.4-all/9br9xq1tocpiv8o6njlyu5op1/gradle-4.4/bin/gradle
Starting a Gradle Daemon (subsequent builds will be faster)
Download https://dl.google.com/dl/android/maven2/com/android/tools/build/gradle/3.1.4/gradle-3.1.4.pom
Download https://jcenter.bintray.com/org/jetbrains/kotlin/kotlin-gradle-plugin/1.2.61/kotlin-gradle-plugin-1.2.61.pom
Download https://dl.google.com/dl/android/maven2/com/android/tools/build/bundletool/0.1.0-alpha01/bundletool-0.1.0-alpha01.pom
Download https://dl.google.com/dl/android/maven2/com/android/tools/build/gradle-core/3.1.4/gradle-core-3.1.4.pom
Is any way to avoid redownload of libs for project, may be exist a case to cache and unzip them, like "android-sdk" and "gradle")?

Docker GitLab Error running Android Emulator: Cannot decide host bitness. 32 bits assumed

I am trying to automate Android Builds and Tests. The instrumentation test is not working because I get the following error while starting the android emulator in the Docker Runner:
Running with gitlab-ci-multi-runner 9.4.2 (6d06f2e) ...
Using Docker executor with image kmindi/android-ci:latest ...
...
$ chmod +x ./gradlew
$ echo no | avdmanager -v create avd --force --name test --abi google_apis/x86_64 --package 'system-images;android-25;google_apis;x86_64'
Do you wish to create a custom hardware profile? [no] $ export SHELL=/bin/bash && echo "no" | emulator -avd test -noaudio -no-window -gpu off -verbose -qemu
sh: 1: file: not found
sh: 1: file: not found
WARNING: Cannot decide host bitness because $SHELL is not properly defined; 32 bits assumed.
ERROR: 32-bit Linux Android emulator binaries are DEPRECATED, to use them
you will have to do at least one of the following:
- Use the '-force-32bit' option when invoking 'emulator'.
- Set ANDROID_EMULATOR_FORCE_32BIT to 'true' in your environment.
Either one will allow you to use the 32-bit binaries, but please be
aware that these will disappear in a future Android SDK release.
Consider moving to a 64-bit Linux system before that happens.
ERROR: Job failed: exit code 1
I created the following Dockerfile (https://hub.docker.com/r/kmindi/android-ci/)
FROM openjdk:8-jdk
ENV ANDROID_BUILD_TOOLS "26.0.1"
ENV ANDROID_SDK_TOOLS "25.2.5"
ENV ANDROID_HOME "/android-sdk"
ENV PATH=$PATH:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools
# Prepare dependencies
RUN mkdir $ANDROID_HOME \
&& apt-get update --yes \
&& apt-get install --yes wget tar unzip lib32stdc++6 lib32z1 libqt5widgets5 expect \
&& apt-get clean \
&& rm -fr /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Install sdk tools
RUN wget -O android-sdk.zip https://dl.google.com/android/repository/tools_r${ANDROID_SDK_TOOLS}-linux.zip \
&& unzip -q android-sdk.zip -d $ANDROID_HOME \
&& rm android-sdk.zip
# Workaround for
# Warning: File /root/.android/repositories.cfg could not be loaded.
RUN mkdir /root/.android \
&& touch /root/.android/repositories.cfg
# Workaround for host bitness error with android emulator
# https://stackoverflow.com/a/37604675/455578
RUN mv /bin/sh /bin/sh.backup \
&& cp /bin/bash /bin/sh
# Add tools from travis
ADD https://raw.githubusercontent.com/travis-ci/travis-cookbooks/ca800a93071a603745a724531c425a41493e70ff/community-cookbooks/android-sdk/files/default/android-wait-for-emulator /usr/local/bin/android-wait-for-emulator
RUN chmod +x /usr/local/bin/android-wait-for-emulator
# Add own tools
COPY assure_emulator_awake.sh /usr/local/bin/assure_emulator_awake.sh
RUN chmod +x /usr/local/bin/assure_emulator_awake.sh
# Update platform and build tools
RUN echo "y" | sdkmanager "tools" "platform-tools" "build-tools;${ANDROID_BUILD_TOOLS}"
# Update SDKs
RUN echo "y" | sdkmanager "platforms;android-26" "platforms;android-25"
# Update emulators
RUN echo "y" | sdkmanager "system-images;android-25;google_apis;x86_64"
# Update extra
RUN echo "y" | sdkmanager "extras;android;m2repository" "extras;google;m2repository" "extras;google;google_play_services"
# Constraint Layout
RUN echo "y" | sdkmanager "extras;m2repository;com;android;support;constraint;constraint-layout;1.0.2"
RUN echo "y" | sdkmanager "extras;m2repository;com;android;support;constraint;constraint-layout-solver;1.0.2"
In GitLab I use the following .gitlab-ci.yml:
image: kmindi/android-ci:latest
variables:
ANDROID_COMPILE_SDK: "25"
before_script:
- chmod +x ./gradlew
stages:
- build
- test
build:
stage: build
script:
- ./gradlew assembleDebug
artifacts:
name: "App_{$CI_BUILD_ID}"
expire_in: 1 week
paths:
- "app/build/outputs/apk/**/*.apk"
except:
- tags
test:unit:
stage: test
script:
- ./gradlew test jacoco
artifacts:
name: "tests-unit-${CI_BUILD_NAME}_${CI_COMMIT_REF_NAME}_${CI_BUILD_REF}"
expire_in: 1 week
paths:
- "**/build/reports/**"
test:instrumentation:25:
stage: test
script:
- echo no | avdmanager -v create avd --force --name test --abi google_apis/x86_64 --package 'system-images;android-25;google_apis;x86_64'
- export SHELL=/bin/bash && echo "no" | emulator -avd test -noaudio -no-window -gpu off -verbose -qemu &
- adb wait-for-device
- android-wait-for-emulator
- export TERM=${TERM:-dumb}
- assure_emulator_awake.sh "./gradlew cAT"
- ./gardlew createDebugCoverageReport
artifacts:
name: "tests-instrumentation-${ANDROID_COMPILE_SDK}-${CI_BUILD_NAME}"
expire_in: 1 week
paths:
- "**/build/reports/**"
The workaround proposed in https://stackoverflow.com/a/37604675/455578 (chaning of bash/sh) is not working (applied in the Dockerfile). Also setting the SHELL variable does not help. If I use emulator64-x86 instead of emulator I get the error command not found.
What am I missing?
Do I have to install other packages via apt?
Maybe it originates in the used openjdk-8 / debian stretch docker image?
Thanks :)

With SDK Revision 25.3.0 (March 2017) emulator was released separately from the SDK Tools (https://developer.android.com/studio/releases/emulator.html#25-3).
Practically it means that now there is a new ${ANDROID_HOME}/emulator folder which you have to add to PATH. (NOTE: it has to be added before tools folder, because for whatever reason old tools/emulator binary is still there). Example:
ENV PATH ${ANDROID_HOME}/emulator:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools:${PATH}

Git lab pipeline Android project

I create an android project in git lab. I'm trying to use pipelines .
The build failed with message
Before building your project, you need to accept the license agreements and complete the installation of the missing components using the Android Studio SDK Manager.
How can I accept this licences automatically?
Adding --no-ui --all is not enough (I have already tried to add them!).
I tried all the soluctions presented [here](you need to accept the license agreements and complete) with no success.
Does someone actually managed to make git lab pipelines works for android?
Below my .gitlab-ci.yml file.
image: openjdk:8-jdk
variables:
ANDROID_COMPILE_SDK: "24"
ANDROID_BUILD_TOOLS: "25.0.3"
ANDROID_SDK_TOOLS: "25.0.3"
before_script:
- export ANDROID_HOME=$PWD/
- export PATH=$PATH:$PWD/
- apt-get --quiet update --yes
- apt-get --quiet install --yes wget tar unzip lib32stdc++6 lib32z1
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/tools_r${ANDROID_SDK_TOOLS}-linux.zip
- unzip android-sdk.zip
- echo y | tools/android --silent update sdk --no-ui --all --filter android-${ANDROID_COMPILE_SDK}
- echo y | tools/android --silent update sdk --no-ui --all --filter platform-tools
- echo y | tools/android --silent update sdk --no-ui --all --filter build-tools-${ANDROID_BUILD_TOOLS}
- echo y | tools/android --silent update sdk --no-ui --all --filter extra-android-m2repository
- echo y | tools/android --silent update sdk --no-ui --all --filter extra-google-google_play_services
- echo y | tools/android --silent update sdk --no-ui --all --filter extra-google-m2repository
- chmod +x ./gradlew
stages:
- build
build:
stage: build
script:
- ./gradlew assembleDebug --stacktrace
artifacts:
paths:
- app/build/outputs/

The sdkmanager cli can be used to update sdks/ accept licenses.
To accept all licenses you can run
echo y | tools/bin/sdkmanager --licenses

I finnally resolve the problem using sdkmanager but I had to download the sdk zip instead of the tools zip like the code in my original question.
This is the final code I copied from Nick Petrovsky post on gitlab issue tracker
image: openjdk:8-jdk
variables:
ANDROID_COMPILE_SDK: "24"
ANDROID_BUILD_TOOLS: "25.0.3"
# ANDROID_SDK_TOOLS: "25.0.3"
ANDROID_SDK_TOOLS_REV: "3859397" # "26.0.1"
before_script:
- mkdir $HOME/.android # For sdkmanager configs
- echo 'count=0' > $HOME/.android/repositories.cfg # Avoid warning
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/sdk-tools-linux-${ANDROID_SDK_TOOLS_REV}.zip
- mkdir $PWD/android-sdk-linux
- unzip -qq android-sdk.zip -d $PWD/android-sdk-linux
- export ANDROID_HOME=$PWD/android-sdk-linux
- export ANDROID_NDK_HOME=$ANDROID_HOME/ndk-bundle # Remove if you don't need NDK
- export PATH=$PATH:$ANDROID_HOME/platform-tools/:$ANDROID_NDK_HOME
- echo y | $ANDROID_HOME/tools/bin/sdkmanager --update
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'tools'
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'platform-tools'
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'build-tools;'$ANDROID_BUILD_TOOLS
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'platforms;android-'$ANDROID_COMPILE_SDK
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'extras;android;m2repository'
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'extras;google;google_play_services'
- echo y | $ANDROID_HOME/tools/bin/sdkmanager 'extras;google;m2repository'
- chmod +x ./gradlew

At the end I used this gitlab-ci-android docker image
It is faster and cause much less problems:
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- mkdir -p $GRADLE_USER_HOME
- chmod +x ./gradlew
cache:
paths:
- .gradle/wrapper
- .gradle/caches
build:
stage: build
script:
- ./gradlew assemble
test:
stage: test
script:
- ./gradlew check

I have created a CI image for Android, which can be found at https://gitlab.com/mvglasow/android-gitlab-ci.
It is based on what is described at https://about.gitlab.com/2018/10/24/setting-up-gitlab-ci-for-android-projects/, but it has all the Android tools in the image already so you don’t have to install them again for every CI run. Compared to the instructions, the image saves you about 30 seconds per pipeline.