Android gradle full rebuild on minor module changes - android

We're experiencing very slow build times when modifying code in one of our project modules. Changes are minor, like adding additional logic in pure function, so no interfaces are changed.
In build analyzer we can see the task that takes longest:
:app:mergeProjectDexXXXXXDebug
And reasons:
Reason task ran
Input property 'dexDirs' file /Users/xxxxx/Desktop/xxxxx/beta-temp/xxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_0.jar has been removed.
Input property 'dexDirs' file /Users/xxxxxx/Desktop/xxxxxx/beta-temp/xxxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_1.jar has been removed.
Input property 'dexDirs' file /Users/xxxxxxxx/Desktop/xxxxxxx/beta-temp/xxxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_2.jar has been removed.
How to prevent these files from being removed?

Related

Android builds on Bamboo are hanging for 20+ minutes on lintVital task

For some time now, all of our Android builds (running on Bamboo) are hanging for 20+ minutes at the lintVital task, at which point the build is usually force stopped:
build 28-Jan-2019 09:25:25 > Task :app-bigscreen:lintVitalProdFiretvRelease
build 28-Jan-2019 09:26:36 Calling mockable JAR artifact transform to create file: /appl/bamboo/gradle-home/caches/transforms-2/files-2.1/552f5f21376c4f273769ade73c6fef92/android.jar with input /appl/bamboo/android-sdk-linux/platforms/android-28/android.jar
error 28-Jan-2019 09:47:00 Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.
However, this is only the case on one our two Bamboo build agents - on the other one, the same build is running without any problems, with the lintVital task finishing within a second. (Builds on the development machines are also running fine.) According to our Bamboo admins the setup on the two build agents is "identical" - I also made sure that the Android SDK folder are in sync and since the line before the error references some files in the Gradle cache I also deleted that folder but that didn't do any good, either.
So, I am basically looking for any hints on what could make the lintVital task hang?
Or, more generally: What does the lintVital task do and what tools or other input (besides the content of our Git repository) does it depend on?
Lint is memory intensive. Give your build more memory.
Put this in the <project root>/gradle.properties file:
# Gradle will get 1 GB of memory to play with.
org.gradle.jvmargs=-Xmx1g
(or add the -Xmx argument to a preexisting org.gradle.jvmargs attribute)
Increase it if it's still not enough. But don't give more than it needs, garbage collection gets slower as the available memory grows. The default value is 512m.

what exactly is "configuration on demand" in Gradle?

I recently changed some settings in Gradle to speed up its process and one of them was changing this: org.gradle.configureondemand=true property in gradle.properties file.
I know you can guess a lot from the words "configuration on demand", but I wanna know the exact impact of this feature? Do I have to do something to trigger configuration if I set this argument as true?
Can something go wrong if I set it as true ?
What configuration phase exactly is?
This setting is relevant only for multiple modules projects. Basically, it tells Gradle to configure modules that only are relevant to the requested tasks instead of configuring all of them, which is a default behaviour.
To answer more precisely to your questions:
No, you don't have to trigger configuration manually.
Yes, something could go wrong as stated in the documentation. The
feature should work very well for multi-project builds that have
decoupled projects.
In “configuration on demand” mode, projects are configured as follows:
The root project is always configured. This way the typical common configuration is supported (allprojects or subprojects script blocks).
The project in the directory where the build is executed is also configured, but only when Gradle is executed without any tasks. This way the default tasks behave correctly when projects are configured on demand.
The standard project dependencies are supported and makes relevant projects configured. If project A has a compile dependency on project B then building A causes configuration of both projects.
The task dependencies declared via task path are supported and cause relevant projects to be configured. Example: someTask.dependsOn(:someOtherProject:someOtherTask)
A task requested via task path from the command line (or Tooling API) causes the relevant project to be configured. For example, building projectA:projectB:someTask causes configuration of projectB.
Here is the full documentation.

How does android.enableBuildCache=true functions in new Android Studio 2.2?

I have recently migrated to new Android Studio 2.2 among many other features I also came across android.enableBuildCache=true to decrease build time.
My question is that how is this helping my builds go faster and what exactly is being cached ? Because I have actually noticed an increase in initial build time which has made me question whether I should use it or not. If somebody could explain me the inner functionality and what exactly is cached and in what conditions then maybe I will be able to fine tune this caching and speed up my builds.
this link shows all the details: http://tools.android.com/tech-docs/build-cache
here are some relevant parts:
... build cache feature that can speed up build times (including full
builds, incremental builds, and instant run) by storing and reusing
files/directories that were created in previous builds of the same or
different Android project.
Currently, the build cache contains only pre-dexed libraries; in the
future, we will use it for caching other types of files as well.
when your build time increased, then maybe
I guess this could be normal for the first build after activating the cache
it's a bug of this experimental feature: in this case, please file a bug
the cache was not used at all:
Step 0
Ensure that android.dexOptions.preDexLibraries is either not
set or set to true; otherwise the libraries will not be pre-dexed and
therefore the build cache will not be used.
Step 1
android.enableBuildCache=true
Step 2
Build your Android project and check the following locations to see whether the build cache took
effect. By default the cache-dir is
/.android/build-cache. The final pre-dexed files
are stored in
/build/intermediates/pre-dexed/debug and
/build/intermediates/pre-dexed/release. You
need to run the command line to see the “pre-dexed” directory; if you
click the “Run” button from the Android Studio menu, you will not see
this directory as it will be deleted.
Note: If you are using Multi-dex with minSdk >= 21, the dexed files
will be stored directly into the
/build/intermediates/transforms/dex directory
without being stored in /build/intermediates/pre-dexed.

(Gradle and OrmLite config) How to add resource file after Java has been compiled but before .apk is generated?

NOTE: I have already accepted an answer and awarded the bounty, BUT, have in the end decided that my approach to this issue was far from optimal. After further thought I have come to the conclusion that modifying the .apk during the build process is probably not the safest or most sustainable way to accomplish this and keep it working long term.
I have added an alternative approach to the very bottom of this question, which accomplishes the same thing in the end. This approach I opted to use instead, while not perfect, does not require messing around with the internals of the .apk assembly through hacks.
I want to use OrmLite with its pre-generated configuration file, which is generated by a plain Java class, like this:
public final class DatabaseConfigGenerator extends OrmLiteConfigUtil{
private static final Class<?>[] MODELS = {
Table1.class
};
private static final String ORMLITE_CONFIGURATION_FILE_NAME = "ormlite_config.txt";
public static void main(String[] args) throws Exception {
File configFile = new File(new File("").getAbsolutePath().split("app" +File.separator + "build")[0] +
File.separator +
"app" + File.separator +
"src" + File.separator +
"main" + File.separator +
"res" + File.separator +
"raw" + File.separator +
ORMLITE_CONFIGURATION_FILE_NAME);
if (configFile.exists()){
configFile.delete();
}
writeConfigFile(configFile, MODELS);
}
}
The resulting ormlite_config.txt file will then be placed under res/raw/, and looks something like this:
#
# generated on 2014/06/20 10:30:42
#
# --table-start--
dataClass=com.example.app
tableName=table1
# --table-fields-start--
# --field-start--
fieldName=field1
# --field-end--
# --table-fields-end--
# --table-end--
#################################
This class needs to be run directly via Java every time either itself or one of the Model classes are modified, so that the configuration is up-to-date and the OR mapping can function as expected.
Since I recently switched over to Android Studio and Gradle, and I love the flexibility and customization options for the build process, I would like to automate the generation of the aforementioned ormlite_config.txt via the build.gradle for my app. I already have defined a working task which runs DatabaseConfigGenerator.class from inside the app/build/classes and generates the config, and I have also hooked it up with the compileJava Gradle tasks, so the config is generated after the Java files are compiled and the .class files are up-to-date:
android.applicationVariants.all { variant ->
ext.variantname = "compile" + variant.name.capitalize() + "Java"
def javaTask = project.tasks.findByName("${variantname}")
if (javaTask != null) {
println "Adding post-compile hook to ${variant.name}"
javaTask.finalizedBy runOrmGenTask
}
}
This works well and I can see the ormlite_config.txt change inside the app/src/main/res/raw, but for some reason (I guess the task ordering is not correct), when I extract the .apk, it still contains the outdated ormlite_config.txt from the previous build...
Can anyone tell me or refer me to a link where the build task order of the Android Gradle build system? I've been searching far and wide for a couple of days now and can't find it. I need to find a way to generate the ormlite_config.txt AFTER the Java files are compiled, but BEFORE the .apk is packaged, so it will be included.
It would be really awesome to automated it like this because then it would happen during every build, in one step, because the config would always be up-to-date with the model classes and I would never have to think about it again. I have a gut feeling that his can be done, I just need to figure out how exactly.
DISCLAIMER: I'm still at the very beginning stages of learning how Gradle works, so my understanding of some things I mentioned here could be way off. Please tell me if it is, I want to learn!
EDIT 1:
I figured it would make more sense to have the DatabaseConfigGenerator write the file NOT under:
app/src/main/res/raw
but under
app/build/res/all/<variant_name>/raw
Since, AFAIK, this is where the final resources are placed before they are packaged into the .apk (I could be wrong, so please correct me if I am).
I also updated my build.gradle slightly, according to #pepyakin's answer:
gradle.projectsEvaluated {
android.applicationVariants.all { variant ->
def ormGenTask = project.tasks.findByName("genOrmConfig" + variant.name.capitalize())
def javaCompileTask = project.tasks.findByName("compile" + variant.name.capitalize() + "Java")
def packageTask = project.tasks.findByName("package" + variant.name.capitalize())
ormGenTask.dependsOn(javaCompileTask)
packageTask.dependsOn(ormGenTask)
}
}
Again, this runs fine and outputs the following in the Gradle console:
...
:app:processDebugResources UP-TO-DATE
:app:generateDebugSources UP-TO-DATE
:app:compileDebugJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
:app:preDexDebug UP-TO-DATE
:app:dexDebug
:app:genOrmConfigDebug
Writing configurations to /home/user/development/app/com.example.app.android/app/build/res/all/debug/raw/ormlite_config.txt
Wrote config for class com.example.app.model.Table1
Done.
:app:processDebugJavaRes UP-TO-DATE
:app:validateDebugSigning
:app:packageDebug
:app:assembleDebug
...
So above I see that the app:genOrmConfigDebug task is neatly sandwiched between the Java compilation and the packaging.
HOWEVER, for some reason, the resultant .apk STILL contains an ormlite_config.txt from one build earlier, it's not up-to-date with the changes that I make to the model class (e.g. defining a new #DatabaseField)!
My hunch from this is that either:
I'm writing the ormlite_config.txt to the wrong location (which
would be weird, since it IS picked up into the .apk after a second
build), OR
The contents of app/build/res/all/<variant_name>/raw are picked up
before compile<variant_name>Java is executed
If it's the later, I have no idea how to handle this... Any suggestions are appreciated!
EDIT 2:
It seems that 2. really is the case. I opened the two directories side by side (app/build/apk and app/build/res/all/<variant_name>/raw) and the order of events is:
The up-to-date ormlite_config.txt is generated inside app/build/res/all/<variant_name>/raw
The .apk is created inside app/build/apk
After extracting the .apk, and looking under res/raw, the outdated ormlite_config.txt from one build ago is inside
I would really appreciate it if someone familiar with the internal goings-on of the Gradle .apk generation process could tell me what I am missing here!
EDIT 3
I'm not giving up on this just yet! After some more research, I found a diagram of the android gradle build system workflow.
According to the diagram, the resources (under /res) are merged and collected, and the R class is updated right before compiling the Java code. Which makes sense, because the compilation would fail if the classes reference resource values that are not included in R.
So now I know for sure that the order of execution for the three steps relevant for my case is:
Resources are merged and assembled, R.java is updated
Java is compiled into classes
The .apk is put together
Now, if my ormlite_config.txt is re-generated after the Java compilation (as is defined in the build.gradle snippet I included in EDIT 2), but in the end is not part of the resulting .apk (the earlier version of the file is instead) even though I place it under /build/res/all/<variant name>/raw before step 3., that can only mean that the actual resource files to be included in the .apk are already moved somewhere other than buid/res/all/<variant name>, between steps 1. and 2.
Now I only have to figure out where that is, so I can put the freshly generated ormlite_config.txt there if it's in any way possible or feasible...
As before, I would be extremely thankful if someone could enlighten me on this issue.
AN ALTERNATIVE (SIMPLER) APPROACH
As stated at the very top of the question, in the end I decided to go with an alternative approach which is much simpler and is not a hack of the .apk assembly process, as is what I originally intended to do.
The steps are as follows:
Code your database configuration generator class to write the ormlite_config.txt int your app's src/main/res/raw (You can use the DatabaseConfigGenerator class that I included at the very top of this question as a template). Keep this class withing the package structure of your app, don't make it a separate application or module, there is not reason to do this. So you can put it inside com.your.app.database or whatever.
In the top tool bar in Android Studio, click on the little drop-down box between the "Make" and "Run" buttons:
It will open a menu where you can choose one of your existing run configurations or edit the configurations. Choose the later:
The "Run/Debug Configurations" window will open, where in the top left corner you should click the little green plus sign and select "Application" as a type for your new configuration:
A form will open where you will define a run configuration for your DatabaseConfigGenerator, as it needs to be run as a Java application, separately from your Android app. There are only a few fields you need to modify here. First, give your new Run configuration a name (1), then select the DatabaseConfigGenerator as the main class (2), then under module classpath choose the module of your app wherein your DatabaseConfigGenerator and your model classes reside (3), then remove all entries from the "Before launch" section by selecting them and clicking the red minus sign (4), and finally, click "Apply" (5). Now you can click your app under the "Android Application" section to the left (6).
The very last thing you need to do here is also the most important one, which will put everything together to make your app first build itself, then generate an up-to-date ormlite_config.txt, and then build itself again (albeit much faster than the first time) so that this newly generated config is actually included in the final .apk. In order to accomplish this, you need to modify the "Before launch" section of your app's run configuration (1). Chances are you will already have a "Gradle-aware Make" in here, which is what actually compiles your app and packages it into an .apk during the usual build process. If you don't have it, add it as the first entry. After that, add another entry, but this time for the "Database Configuration Generator" run configuration, which you created a few steps back, as this will ensure that the ormlite_config.txt is generated based on freshly compiled model classes and is up-to-date. And finally, add another "Gradle-aware Make", to make sure that a new .apk is generated, which will now also include this up-to-date ormlite_config.txt. Now click "Apply" (2), and that's it!
From this point on, every time you click the "Run" button in the toolbar at the top of the Android Studio window, while the "app" run configuration is selected, you can be sure that the ormlite_config.txt in the resulting .apk will be up-to-date with whatever changes you made to your model classes or the DatabaseConfigGenerator itself.
For this solution I've taken inspiration from the following two SO answers:
Setup Gradle to run Java executable in Android Studio
Android Studio run configuration for ORMLite config generation
In the end I decided to put together the complete solution in a single place and describe it in detail right here.
There are three small caveats to this approach, and YMMV on whether you can live with them:
This only applies to the "Run" action, not the "Make" action, which means that you will have to initiate a run even in cases when you just want to build an .apk, but not actually run it. The resulting .apk can then be found under app/build/apk/ and is named depending on the variant you are building (for debug builds it will usually be app-debug-unaligned.apk, and for releases, app-release.apk).
This approach in itself means the "Gradle-aware make" will run twice every time you click "Run", which will result in slightly longer build times, but I haven't noticed much of a difference (the android gradle plugin is smart enough to recognize which resources have not changed since the last build and will skip a lot of unnecessary steps the second time around), resulting in maybe a 20% longer build time (don't hold me to the number).
If you are working in a team setting and are using version control, it sucks a little bit that this configuration is not trackable, so every developer in your team will have to go through this process individually and can't simply check it out as part of the repository in, say, .git. This is due to the fact that the run configurations are defined inside your project root, under .idea/workspace.xml, which is universally agreed to be something that should not be tracked in version control as it is machine specific.
There are probably ways to remove some manual steps of the process of defining your run configuration like this on a team level, but it doesn't seem possible to fully automate it in a clean way. I could be wrong though and feel free to let me know if that's the case.
Hope this helps!
The problem is that the resources are compiled by aapt when R.java is produced (and so before javac). Those compiled resources are then used to build the apk. AFAIK those compiled resources is just a kind of zip archive.
So, to achieve your goal you need to modify those compiled resources after javac compilation.
You can try to define a new task (let's call it addOrmToRes) invoking aapt with required arguments to modify the compiled resources.
aapt is located under <ANDROID_SDK_HOME>/build-tools/<version>/
The aapt Usage indicates that :
aapt a[dd] [-v] file.{zip,jar,apk} file1 [file2 ...]
Add specified files to Zip-compatible archive.
So executing something like this:
aapt add /build/apk/myapp.apk ormlite_config.txt
or this
aapt add <the_path_to_compiled_resources> ormlite_config.txt
in your new custom task addOrmToRes should do the trick. So to answer to your edit 3 : aapt add is probably the solution.
To integrate it in the build, something like this should work:
runOrmGenTask.dependsOn(variant.javaCompile)
addOrmToRes.dependsOn(runOrmGenTask)
addOrmToRes.dependsOn(<the task creating the apk>)
<the task signing the apk>.dependsOn(addOrmToRes)
Note that I didn't identify every tasks here. But I guess you get the idea.
Try to use dependsOn mechanism.
runOrmGenTask.dependsOn(variant.javaCompile)
And, then for example:
packageApplication.dependsOn(runOrmGenTask)
More about task manipulation you can get from here.

How to supply Jacoco agent to an Android app?

I am trying to get code coverage using Jacoco for manual testing. I am trying to use offline instrumentation. http://www.eclemma.org/jacoco/trunk/doc/offline.html
In there it mentions:
Configuration File: If a file jacoco-agent.properties is supplied on the classpath options are loaded from this file.
My question is how do I supply this? On some other forum, I saw that jacoco-agent.properties needs to be deployed with the app.
I tried putting jacoco-agent.properties under the res/raw, but it complained about "-" in the file name.
I tried putting this under the root directory it didn't take any effect.
I was wondering how could I setup\run jacoco so that it uses jacoco-agent.properties?
Well, at least for an Ant command line build the trick was to put jacoco-agent.properties in the src directory, which makes it get packaged as-is into the root of the APK file, from where it seems to be properly read (i.e. I can see setting the destfile property having an effect on where the execution data file is being created).
However, the created jacoco.exec file was always empty for me (like also mentioned here), probably because JaCoCo only writes the execution data at JVM termination, but Android's Dalvik VM does not terminate when the app stops.
The way we currently solve this is to explicitly call
org.jacoco.agent.rt.RT.getAgent().getExecutionData(false)
and write the returned byte array to a file when our custom test runner finishes.

Categories

Resources