Gradle builds really slow with a multi-project structure - android

When building with gradle on a multi-project setup containing roughly 140 projects/libraries, the build time took 1 hour and 22 minutes. And i was using --parallel. And our ANT build takes less than 20 minutes without parallel building.
Here is exactly what i did.
./gradlew clean
./gradlew build --parallel
I did a little testing it seems like the dexing is taking the longest amount of time. Is there a way to get the gradle process to re-use the stuff it has already dexed? If the libraries have already been built, it should re-use the already dexed libraries.
I saw the option --no-rebuild, but when i run with that option it says the following
File '/path/to/project/build/libs/project.aar' specified for property 'bundle' does not exist.
I replaced the file path and project name with generic stuff.
Using Gradle 1.9-rc-3
Additional information(15 Jan 2014):
preDexDebug and preDexRelease took a VERY long time on each project. Much longer than any other task.
Progress(15 Jan 2014):
Ok, for now, i put preDexLibraries = false into all of the build.gradle files. However, i still would like to know a centralize place that i can put that entry and it affect all the other build.gradle files.
However, now dexRelease and dexDebug are taking a long time. Is there any way that i can tell the build to only do the dexDebug or dexRelease and skip the other one?
Progress(15 Jan 2014):
Using assembleDebug worked. However, it still seems like it is not re-using the already dexed libraries. Because dexing is still taking forever. It takes about a minute for each project. Is there a way to get gradle to re-use the already dexed libraries? Or is there a different reason why the build is still taking about an hour? Our ANT process takes less than 15 minutes.

Doing a clean you actually delete the already predexed libraries.
As suggested in this thread you could save some time on clean builds by disabling predexing (because at the next build they will be deleted):
android {
dexOptions {
preDexLibraries = false
}
}

According to this post.
Right now each project will pre-dex its dependencies on its own. This
means 2 components depending on the same library will both run pre-dex
on that library's classes.jar which is silly. We're looking at fixing
this.

Did you try setting the parameters of the Gradles compiler?
If you have remote dependencies, each time you run the project it will be operated with remote resources using the network. You should define the instruction to the Gradles compiler to that him to work offline using:
--offline
Here I leave you a screenshot of the compiler settings for better performance:
Or what is the same...
Source: http://gradle.org/docs/current/userguide/gradle_command_line.html

Related

Android studio is slow in gradle building

Android studio is getting slow in grade building process.I noticed this problem after updating to new version 3.5.Is there any ways to
speed up the building process?
1- Make sure you’re using the latest version of Gradle. Generally with every new update there is a significant improvement in performance.
Note: Java 1.8 is faster than 1.6. Make sure it’s updated too.
2- Try to minimize the use of modules. There are many cases where we need to fork the library to modify it to fit according to our needs. A module takes 4x greater time than a jar or aar dependency. This happens due to the fact that the module needs to be built from the scratch every time.
3- Enable gradle Offline Work from Preferences-> Build, Execution, Deployment-> Build Tools-> Gradle. This will not allow the gradle to access the network during build and force it to resolve the dependencies from the cache itself.
Note: This only works if all the dependencies are downloaded and
stored in the cache once. If you need to modify or add a new
dependency you’ll have to disable this option else the build would
fail.
4-Open up the gradle.properties file from the root of your project. Add the following lines of code in it.
org.gradle.daemon=true
Gradle daemon is a background process. Adding this would consume some extra memory while building.
org.gradle.parallel=true
The above line of code enables compilation of multiple modules at the same time. Besides that it also gives us other benefits such as;
Re-using the configuration for unchanged projects
Project-level is up-to-date checks
Using pre-built artifacts in the place of building dependent projects
Adding the following line of code also aids us in speeding up the build.
org.gradle.configureondemand=true
Another important property is;
org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
The above line is used to allow Java compilers to have available memory up to 2 GB (2048 MB). It should only be used if you have available memory more than 2 GB.
This is how the gradle.properties file should look like:
5- Avoid dynamic dependencies such as
compile 'com.google.maps.android:android-maps-utils:0.4+'.
Dynamic Dependencies slow down your build since they keep searching for the latest builds every time. To improve the performance we need to fix the version in place.
6- Use only those dependencies that you need. For example google maps dependency, instead of importing , like :
implementation 'com.google.android.gms:play-services:17.0.0'
implementation 'com.google.android.gms:play-services-maps:17.0.0'
Gradle build speed depends on a lot of factors including the specification of your machine as well as your build type and android studio settings. You can check out this article on how to reduce your build time or go through the steps in the Android developer website.
Personal experience:
When I faced this issue, enabling offline mode drastically reduced my build time. The only problem is that I have to toggle this setting (on and off) every time I want to add a new dependency and this almost made me go berserk on several occasions. However, if properly handled, this helps a great deal.
I hope this helps. Merry coding!
you can set file->setting->search 'Gradle'-> check 'use local gradle distribution' and check 'offline work'.. in android studio. it will improve gradel building time.
Note: In newer version of Android studio, View->Tool Windows->Gradle->Toggle button of online/offline

Android builds on Bamboo are hanging for 20+ minutes on lintVital task

For some time now, all of our Android builds (running on Bamboo) are hanging for 20+ minutes at the lintVital task, at which point the build is usually force stopped:
build 28-Jan-2019 09:25:25 > Task :app-bigscreen:lintVitalProdFiretvRelease
build 28-Jan-2019 09:26:36 Calling mockable JAR artifact transform to create file: /appl/bamboo/gradle-home/caches/transforms-2/files-2.1/552f5f21376c4f273769ade73c6fef92/android.jar with input /appl/bamboo/android-sdk-linux/platforms/android-28/android.jar
error 28-Jan-2019 09:47:00 Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.
However, this is only the case on one our two Bamboo build agents - on the other one, the same build is running without any problems, with the lintVital task finishing within a second. (Builds on the development machines are also running fine.) According to our Bamboo admins the setup on the two build agents is "identical" - I also made sure that the Android SDK folder are in sync and since the line before the error references some files in the Gradle cache I also deleted that folder but that didn't do any good, either.
So, I am basically looking for any hints on what could make the lintVital task hang?
Or, more generally: What does the lintVital task do and what tools or other input (besides the content of our Git repository) does it depend on?
Lint is memory intensive. Give your build more memory.
Put this in the <project root>/gradle.properties file:
# Gradle will get 1 GB of memory to play with.
org.gradle.jvmargs=-Xmx1g
(or add the -Xmx argument to a preexisting org.gradle.jvmargs attribute)
Increase it if it's still not enough. But don't give more than it needs, garbage collection gets slower as the available memory grows. The default value is 512m.

Android multi-module Gradle build slow even without any changes

I have an Android project with multiple modules where the build time is slow even if I try to rebuild it without changing anything.
Its structure is similar to:
There are more "parent" (and independent parent siblings) and "feature" modules but the setup is that of the picture.
My gradle.propertiesfile is as follows:
org.gradle.jvmargs=-Xmx4096m -Xms2048m
org.gradle.parallel=true
org.gradle.daemon=true
org.gradle.configureondemand=true
And I'm using the Gradle version 3.5 and the plugin version 2.4.0-alpha7.
The project also uses multidex (javaMaxHeapSize "4g" and jumboMode true), data binding, kotlin alongside java, and several annotation processors.
When a build, I can see multiple threads running different tasks in parallel for most of the project, except mostly for the final "app" module (which I assume is normal).
But a lot of time seems to be spent on different
::compileJavaWithJavac tasks. Even without changing anything, build takes about 30 secs on a MacBook Pro i7 with 16 Gb RAM.
If I change one line of code inside a method of a feature module (without even changing the method's signature), the build time goes over 1:30 minutes.
Are these normal times? Is there something I can optimise further to improve build times?
I also checked some of the other answers:
From this one, setting preDexLibraries = false doesn't help and I already use assembleDebug
incremental = true suggested here doesn't work and is deprecated
Disabling instant run seems to help a little but not much
The project's build flavour discussed above already sets minSdkVersion 21
you should make the feature module independent as much you can and any dependency with other module please consider the stub. A kind of the proxy which contain only interface feature1 -> featureonestub->feature2

Open the existing Android project and take less time to build with Gradle? [duplicate]

Since the last update (Build from june 25) any changes in the Android studio Gradle is painfully slow. And it also seems to autotrack changes when you edit the file and recompile on keyup.
Each change takes several minutes on my i5.
Any idea how I can speed up my Gradle changes?
Definitely makes a difference: How To… Speed up Gradle build time
Just create a file named gradle.properties in the following directory:
/home/<username>/.gradle/ (Linux)
/Users/<username>/.gradle/ (Mac)
C:\Users\<username>\.gradle (Windows)
Add this line to the file:
org.gradle.daemon=true
After change this settings my compile time 10 mins reduced to 10 secs.
Step 1:
Settings(ctrl+Alt+S) ->
Build,Execution,Deployment ->
Compiler ->
type "--offline" in command-line Options box.
Step 2:
check the “Compile independent modules in parallel” checkbox.
& click Apply -> OK
Step 3: In your gradle.properties file -> Add following lines
org.gradle.jvmargs=-Xmx2048M -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.parallel=true
org.gradle.configureondemand=true
org.gradle.daemon=true
Update:
If you are using Android studio 2.0 or above try the Instant Run
Settings → Build, Execution, Deployment → Instant Run → Enable Instant
Run.
More info about Instant Run - https://developer.android.com/studio/run/index.html#instant-run
I was able to reduce my gradle build from 43 seconds down to 25 seconds on my old core2duo laptop (running linux mint) by adding the following to the gradle.properties file in android studio
org.gradle.parallel=true
org.gradle.daemon=true
source on why the daemon setting makes builds faster:
https://www.timroes.de/2013/09/12/speed-up-gradle/
The dev are working on it. Like I posted in this answer the fastest solution right now is to use gradle from the command line and you should switch to binary libs for all modules you do not develop. On g+ there is a discussion with the developers about it.
Following the steps will make it 10 times faster and reduce build time 90%
First create a file named gradle.properties in the following directory:
/home/<username>/.gradle/ (Linux)
/Users/<username>/.gradle/ (Mac)
C:\Users\<username>\.gradle (Windows)
Add this line to the file:
org.gradle.daemon=true
org.gradle.parallel=true
And check this options in Android Studio
There is a newer version of gradle (ver 2.4).
You can set this for your project(s) by opening up 'Project Structure' dialog from File menu,
Project Structure -> Project -> Gradle version
and set it to '2.4'.
You can read more about boosting performance at this link.
This is what I did and my gradle build speed improved dramatically! from 1 min to 20sec for the first build and succeeding builds became from 40 sec to 5 sec.
In the gradle.properties file
Add this:
org.gradle.jvmargs=-Xmx8192M -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
In the Command Line Arguments via Go to File > Other Settings> default Settings >Build, Execution, Deploy> Complier and add the following arguments to Command Line Arguments
Add this:
--debug --stacktrace -a, --no-rebuild -q, --quiet --offline
With Android Studio 2.1 you can enable "Dex In Process" for faster app builds.
You can get more info about it here: https://medium.com/google-developers/faster-android-studio-builds-with-dex-in-process-5988ed8aa37e#.vijksflyn
Try to avoid using a Mac/PC that has only 8 GB of RAM when doing Android development. As soon as you launch even 1 emulator (Genymotion or otherwise), your build times become extremely slow in Android Studio with gradle builds. This happens even if you make a simple one-line change to 1 source file.
Closing the emulator and using a real device helps a lot, but of course this is very limiting and less flexible. Reducing the RAM usage setting of the emulator can help, but the best way is to ensure your laptop has at least 12-16 GB of RAM.
Update (June 2017): There are now several good medium.com articles that explain how to speed up Android Studio gradle builds in detail, and it even works on 8 GB machines:
How to decrease your Gradle build time by 65%:
https://medium.com/#kevalpatel2106/how-to-decrease-your-gradle-build-time-by-65-310b572b0c43
Make your Gradle builds fast again!: https://medium.com/#wasyl/make-your-gradle-builds-fast-again-ea323ce6a435
The summarised consensus is:
Create a gradle.properties file (either global at ~/.gradle/gradle.properties or local to project), and add the following lines:
org.gradle.daemon=true
org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
org.gradle.parallel=true
org.gradle.configureondemand=true
dexOptions {
incremental true
javaMaxHeapSize "4g"
}
Enable Offline Work
Improve Gradle Performance by adding following code in gradle.properties
org.gradle.daemon=true
org.gradle.parallel=true
Step by step guide:http://www.viralandroid.com/2015/08/how-to-make-android-studio-fast.html
Acording to this page of the Android Team of Wikimedia Apps, a good way of optimize Gradle builds is adding this lines to your ~/.gradle/gradle.properties
org.gradle.daemon=true
org.gradle.parallel=true
org.gradle.configureondemand=true
org.gradle.jvmargs=-Xmx2048M
For those who do not have the file there are two ways to do it:
Add the file locally in your project by creating a file called gradle.properties in the project root or,
You can set them globally for all your projects by creating the same file in your home directory (%UserProfile%.gradle on Windows, ~/.gradle on Linux and Mac OS X)
It is a good practice to set the properties in your home directory, rather than on a project level.
You can also use command line for better performance.You can use the command ./gradlew <task name> from inside the root folder of your project from linux or use gradlew.bat file like gradlew <task name>.when you first run one of the commands above for a given Gradle version, it will download the corresponding Gradle distribution and use it to execute the build.
When importing a Gradle project via its wrapper, your IDE may ask to use the Gradle 'all' distribution. This is perfectly fine and helps the IDE provide code completion for the build files.
Not only does this mean that you don’t have to manually install Gradle yourself, but you are also sure to use the version of Gradle that the build is designed for. This makes your historical builds more reliable.
for more info refer Executing a build with the Wrapper
For faster builds, increase the maximum heap size for the Gradle daemon to more than 2048 MB.
To do this set org.gradle.jvmargs=-Xmx2048M in the project gradle.properties.
Add this to your gradle.properties file
org.gradle.daemon=true
org.gradle.parallel=true
org.gradle.configureondemand=true
org.gradle.jvmargs=-Xmx2048M
Add a build.gradle file:
android {
...
dexOptions {
javaMaxHeapSize "4g" //specify the heap size for the dex process
}
...
}
I hope it helps.
I’m running a 5th gen i7 with Windows 10 and a 1TB Solid State.
I compressed the Android Studio Projects folder and got about an 80% boost.
Hope this helps.
I then combined it with the above solutions ie (org.gradle.parallel=true, org.gradle.daemon=true). The performance boost was quite impressive.
Additionally:
All of the above answers are totally correct but I must state as an experience Android developer (of 4 and a half years) that: No Android/Gradle developer should be working on a machine with a spinner drive, you need to fork out for a Solid State. We all hit that play button in the IDE 100s of times per day. When I went from a spinner to SSD (post Gradle), my speed and efficiency was literally 2 – 4 times faster and I promise you I’m NOT exaggerating here.
Now I’m not talking about having a machine with a small SSD and a big spinner, I’m talking about 1 big SSD. If you already have a machine with a small SSD and a big spinner you can upgrade the small spinner to say a 500GB SSD and set the SSD as your main OS drive with your developer tools installed on it.
So if you’re working in a fast paced environment please show this post to your boss. A decent 1TB SSD will set you back about £300 (including VAT), or about £160 for a 500GB SSD. Depending on if you are a junior or senior Android developer the drive will pay for itself (in wages expenses) in 1 – 2 working weeks, or about 2 and a half to 5 working days if you invest in a smaller; say 500GB SSD.
A lot of developers may argue that this is not the case, but it is the case for Gradle, as the Gradle system is very hard on the direct disk access. If you work with .NET/C#/VB Net or other development tools you won’t notice much difference but the difference in Gradle is HUGE. If you act on this post I promise you, you won’t be disappointed. Personally I’m using fifth gen i7 with 8GB RAM which originally came with a 1TB Spinner and I upgraded it to a Samsung SSD 840 EVO 1TB and I’ve never looked back since. I bought mine from: https://www.aria.co.uk.
Hope this helps. Also I must state that this is NOT a commercially motivated post, I’m just recommending Aria as I’ve used them many times before and they’ve always been reliable.
few commands we can add to the gradle.properties file:
org.gradle.configureondemand=true - This command will tell gradle to only build the projects that it really needs to build.
Use Daemon — org.gradle.daemon=true - Daemon keeps the instance of the gradle up and running in the background even after your build finishes. This will remove the time required to initialize the gradle and decrease your build timing significantly.
org.gradle.parallel=true - Allow gradle to build your project in parallel. If you have multiple modules in you project, then by enabling this, gradle can run build operations for independent modules parallelly.
Increase Heap Size — org.gradle.jvmargs=-Xmx3072m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 - Since android studio 2.0, gradle uses dex in the process to decrease the build timings for the project. Generally, while building the applications, multiple dx processes runs on different VM instances. But starting from the Android Studio 2.0, all these dx processes runs in the single VM and that VM is also shared with the gradle. This decreases the build time significantly as all the dex process runs on the same VM instances. But this requires larger memory to accommodate all the dex processes and gradle. That means you need to increase the heap size required by the gradle daemon. By default, the heap size for the daemon is about 1GB.
Ensure that dynamic dependency is not used. i.e. do not use
implementation 'com.android.support:appcompat-v7:27.0.+'.
This command means gradle will go online and check for the latest version every time it builds the app.
Instead use fixed versions i.e. 'com.android.support:appcompat-v7:27.0.2'
Open gradle.properties from android folder and uncomment highlighted lines and provide memory values as per your machine configuration.
I have 8gb ram on my machine so I gave maximum 4096mb and 1024mb respectively.
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
# Default value: -Xmx10248m -XX:MaxPermSize=256m
//Uncomment below line for providing your system specific configuration
#org.gradle.jvmargs=-Xmx4096m -XX:MaxPermSize=1024m -XX:+HeapDumpOnOutOfMemoryError - Dfile.encoding=UTF-8
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
#http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
//Uncomment below line to allow parallel process execution.
#org.gradle.parallel=true
My build time was reduced to half after this.
Happy building!!
It often happens when you enabled multidex in you project. This can potentially slow your development process!!
According doc:
multidex configuration requires significantly increased build processing time because the build system must make complex decisions about which classes must be included in the primary DEX file and which classes can be included in secondary DEX files. This means that incremental builds using multidex typically take longer and can potentially slow your development process.
but you can optimize this:
To mitigate longer incremental build times, you should use pre-dexing to reuse multidex output between builds.
If you're using Android Studio 2.3 and higher, the IDE automatically uses this feature when deploying your app to a device running Android 5.0 (API level 21) or higher.
So you need to set the minSdkVersion to 21 or higher!
But if you production version need to support minSdkVersion lower than 21, for example 19
you can use productFlavors to set minSdkVersion 21 for you dev version:
android {
defaultConfig {
...
multiDexEnabled true
// The default minimum API level you want to support.
minSdkVersion 15
}
productFlavors {
// Includes settings you want to keep only while developing your app.
dev{
//the IDE automatically uses pre-dexing feature to mitigate longer incremental when deploying your app to a device running Android 5.0 !
minSdkVersion 21
}
prod {
}
}
buildTypes {
release {
minifyEnabled true
proguardFiles getDefaultProguardFile('proguard-android.txt'),
'proguard-rules.pro'
}
}
}
dependencies {
compile 'com.android.support:multidex:1.0.3'
}
I do prefer building from command line for better build times. If your app code base is large and you have multiple modules then you can try Local AAR approach as described here, it will give you a big boost in Android Studio performance & gradle build times. Its compatible with command line builds as well
https://blog.gojekengineering.com/how-we-improved-performance-and-build-times-in-android-studio-306028166b79
Demo project with integration instructions can be found here: https://github.com/akhgupta/AndroidLocalMavenRepoAARDemo
Right click on Drawable folder and convert png/jpg images to webp format it improved the gradle significantly
From the android developer doc:
Configure JVM garbage collector. It reduced the build time to 2 to 4 seconds
gradle.properties file
org.gradle.jvmargs=-Xmx1536m -XX:+UseParallelGC
Refer to this Developer doc
Follow Android Studio and Gradle performance guidelines
Optimize your build speed
Improving the Performance of Gradle Builds
Configuration cache
Build Cache
Overall
Always keep Gradle version up-to-date to receive new performance improvements
Modularize your project as much as possible to benefit from Gradle parallel work
Know Gradle properties and understand what they do

In a multi-project setup, Gradle evaluates EVERY sub-project. Can I ignore some of the subprojects?

I have the following multi-project setup in Gradle for building multiple Android apps which use a few libraries:
Let's say my root dir is /workspace
The directory structure is the following (it's incomplete):
/workspace/ --> root dir for the gradle build
/workspace/settings.gradle
...
/workspace/myapp1/Android/ --> contains the 1st app (has a build.gradle)
/workspace/myapp2/Android/ --> contains the 2nd app (has a build.gradle)
...
/market_licensing/library/ ---> contains a lib (has a build.gradle)
/play_apk_expansion/downloader_library/ ---> another lib (has a build.gradle)
settings.gradle in /workspace/ looks like this:
include ':market_licensing:library'
include ':play_apk_expansion:downloader_library'
include ':myapp1:Android'
include ':myapp2:Android'
When I run a gradle build in one of the app directories, it works:
For example, /workspace/myapp1/Android>gradle clean runs fine.
However, this will EVALUATE also the build.gradle project for myapp2.
This is expected according to the Gradle manual. In http://www.gradle.org/docs/current/userguide/multi_project_builds.html, in part 56.3. Execution rules for multi-project builds it says "Gradle always evaluates every project of the multi-project build and creates all existing task objects.".
My problem with this is that this slows down my builds unnecessarily. Unnecessarily because, when I'm building myapp1, I don't care about myapp2. I want the libraries to be in a good state, but I don't care about myapp2. This is not terrible for 2 apps, but imagine having 20 apps. I want to be able to build the app that I'm currently working on as fast as possible.
Is there a way to ignore other subprojects , even for the evaluation stage?
Or is there an alternative way to set up multiple projects that depend on the same libraries but don't depend on each other?
as you pointed out the default behaviour in gradle is to evaluate all projects. There are plans (and also initial work started) to make this more fine grained in the future. For now you might checkout the incubating feature "configuration on demand". Maybe this helps you. have a look at http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:cross_project_configuration for details
cheers,
René

Categories

Resources