We're experiencing very slow build times when modifying code in one of our project modules. Changes are minor, like adding additional logic in pure function, so no interfaces are changed.
In build analyzer we can see the task that takes longest:
:app:mergeProjectDexXXXXXDebug
And reasons:
Reason task ran
Input property 'dexDirs' file /Users/xxxxx/Desktop/xxxxx/beta-temp/xxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_0.jar has been removed.
Input property 'dexDirs' file /Users/xxxxxx/Desktop/xxxxxx/beta-temp/xxxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_1.jar has been removed.
Input property 'dexDirs' file /Users/xxxxxxxx/Desktop/xxxxxxx/beta-temp/xxxxxxxx/app/build/intermediates/mixed_scope_dex_archive/xxxxxxxxx/out/1fb9f8265892441074f5335df65343abe421dbddd0581ac36f1164807c91c9aa_2.jar has been removed.
How to prevent these files from being removed?
When i use ViewBinding, it always generate xxxxxBinding.
I know android use Android Gradle Plugin build apk with code and resources。
And they're going to split some task (or transform).
I know task(transform) will build some file into ./build .
But the ViewBinding makes me confused，He doesn't appear to be generating files（Because every time I create layout file, I can access it directly）(Or I can just delete the build folder and it won't be created again，But he can still get it right（No error） in the code)。
Were some files secretly generated?
If so, where will it be generated?
If not, how can I do the same? For example, use a markdown file to generate code in real time.
I am particularly new to the Android Security Realm and trying to understand how Android, during installation(and execution) of an apk file, validates its integrity.
Just to try my hands on, I tried recompiling an apk with three .dex files and changing one byte of of the contents in the header section of only one of the classes.dex using a Hex Editor before signing it.
The App installed on the device with almost no complaints but with much shorter time as it usually does.
On trying to run the App, as expected, it crashed complaining it can't find the Application class and the required .dex file isn't present in the /system/... path.
I want to understand why the App installed successfully in the first place and Android didn't complain of corruption during installation. Also, What are the checks which Android is actually placing during the installation which detects this corruption.
P.S.: I changed one byte in the SHA-1 header of one of the .dex file(classes.dex particularly)
As you found, an invalid dex file won't cause installation to fail. Typically, as part of the installation process, art will optimize and compile the dex file - the result of which is an oat file. It's during this process that the dex file is checked for integrity.
You can take a look at art's source to see what kinds of verification it performs on the dex file. See e.g. here, but elsewhere as well.
NOTE: I have already accepted an answer and awarded the bounty, BUT, have in the end decided that my approach to this issue was far from optimal. After further thought I have come to the conclusion that modifying the .apk during the build process is probably not the safest or most sustainable way to accomplish this and keep it working long term.
I have added an alternative approach to the very bottom of this question, which accomplishes the same thing in the end. This approach I opted to use instead, while not perfect, does not require messing around with the internals of the .apk assembly through hacks.
I want to use OrmLite with its pre-generated configuration file, which is generated by a plain Java class, like this:
public final class DatabaseConfigGenerator extends OrmLiteConfigUtil{
private static final Class<?>[] MODELS = {
Table1.class
};
private static final String ORMLITE_CONFIGURATION_FILE_NAME = "ormlite_config.txt";
public static void main(String[] args) throws Exception {
File configFile = new File(new File("").getAbsolutePath().split("app" +File.separator + "build")[0] +
File.separator +
"app" + File.separator +
"src" + File.separator +
"main" + File.separator +
"res" + File.separator +
"raw" + File.separator +
ORMLITE_CONFIGURATION_FILE_NAME);
if (configFile.exists()){
configFile.delete();
}
writeConfigFile(configFile, MODELS);
}
}
The resulting ormlite_config.txt file will then be placed under res/raw/, and looks something like this:
#
# generated on 2014/06/20 10:30:42
#
# --table-start--
dataClass=com.example.app
tableName=table1
# --table-fields-start--
# --field-start--
fieldName=field1
# --field-end--
# --table-fields-end--
# --table-end--
#################################
This class needs to be run directly via Java every time either itself or one of the Model classes are modified, so that the configuration is up-to-date and the OR mapping can function as expected.
Since I recently switched over to Android Studio and Gradle, and I love the flexibility and customization options for the build process, I would like to automate the generation of the aforementioned ormlite_config.txt via the build.gradle for my app. I already have defined a working task which runs DatabaseConfigGenerator.class from inside the app/build/classes and generates the config, and I have also hooked it up with the compileJava Gradle tasks, so the config is generated after the Java files are compiled and the .class files are up-to-date:
android.applicationVariants.all { variant ->
ext.variantname = "compile" + variant.name.capitalize() + "Java"
def javaTask = project.tasks.findByName("${variantname}")
if (javaTask != null) {
println "Adding post-compile hook to ${variant.name}"
javaTask.finalizedBy runOrmGenTask
}
}
This works well and I can see the ormlite_config.txt change inside the app/src/main/res/raw, but for some reason (I guess the task ordering is not correct), when I extract the .apk, it still contains the outdated ormlite_config.txt from the previous build...
Can anyone tell me or refer me to a link where the build task order of the Android Gradle build system? I've been searching far and wide for a couple of days now and can't find it. I need to find a way to generate the ormlite_config.txt AFTER the Java files are compiled, but BEFORE the .apk is packaged, so it will be included.
It would be really awesome to automated it like this because then it would happen during every build, in one step, because the config would always be up-to-date with the model classes and I would never have to think about it again. I have a gut feeling that his can be done, I just need to figure out how exactly.
DISCLAIMER: I'm still at the very beginning stages of learning how Gradle works, so my understanding of some things I mentioned here could be way off. Please tell me if it is, I want to learn!
EDIT 1:
I figured it would make more sense to have the DatabaseConfigGenerator write the file NOT under:
app/src/main/res/raw
but under
app/build/res/all/<variant_name>/raw
Since, AFAIK, this is where the final resources are placed before they are packaged into the .apk (I could be wrong, so please correct me if I am).
I also updated my build.gradle slightly, according to #pepyakin's answer:
gradle.projectsEvaluated {
android.applicationVariants.all { variant ->
def ormGenTask = project.tasks.findByName("genOrmConfig" + variant.name.capitalize())
def javaCompileTask = project.tasks.findByName("compile" + variant.name.capitalize() + "Java")
def packageTask = project.tasks.findByName("package" + variant.name.capitalize())
ormGenTask.dependsOn(javaCompileTask)
packageTask.dependsOn(ormGenTask)
}
}
Again, this runs fine and outputs the following in the Gradle console:
...
:app:processDebugResources UP-TO-DATE
:app:generateDebugSources UP-TO-DATE
:app:compileDebugJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
:app:preDexDebug UP-TO-DATE
:app:dexDebug
:app:genOrmConfigDebug
Writing configurations to /home/user/development/app/com.example.app.android/app/build/res/all/debug/raw/ormlite_config.txt
Wrote config for class com.example.app.model.Table1
Done.
:app:processDebugJavaRes UP-TO-DATE
:app:validateDebugSigning
:app:packageDebug
:app:assembleDebug
...
So above I see that the app:genOrmConfigDebug task is neatly sandwiched between the Java compilation and the packaging.
HOWEVER, for some reason, the resultant .apk STILL contains an ormlite_config.txt from one build earlier, it's not up-to-date with the changes that I make to the model class (e.g. defining a new #DatabaseField)!
My hunch from this is that either:
I'm writing the ormlite_config.txt to the wrong location (which
would be weird, since it IS picked up into the .apk after a second
build), OR
The contents of app/build/res/all/<variant_name>/raw are picked up
before compile<variant_name>Java is executed
If it's the later, I have no idea how to handle this... Any suggestions are appreciated!
EDIT 2:
It seems that 2. really is the case. I opened the two directories side by side (app/build/apk and app/build/res/all/<variant_name>/raw) and the order of events is:
The up-to-date ormlite_config.txt is generated inside app/build/res/all/<variant_name>/raw
The .apk is created inside app/build/apk
After extracting the .apk, and looking under res/raw, the outdated ormlite_config.txt from one build ago is inside
I would really appreciate it if someone familiar with the internal goings-on of the Gradle .apk generation process could tell me what I am missing here!
EDIT 3
I'm not giving up on this just yet! After some more research, I found a diagram of the android gradle build system workflow.
According to the diagram, the resources (under /res) are merged and collected, and the R class is updated right before compiling the Java code. Which makes sense, because the compilation would fail if the classes reference resource values that are not included in R.
So now I know for sure that the order of execution for the three steps relevant for my case is:
Resources are merged and assembled, R.java is updated
Java is compiled into classes
The .apk is put together
Now, if my ormlite_config.txt is re-generated after the Java compilation (as is defined in the build.gradle snippet I included in EDIT 2), but in the end is not part of the resulting .apk (the earlier version of the file is instead) even though I place it under /build/res/all/<variant name>/raw before step 3., that can only mean that the actual resource files to be included in the .apk are already moved somewhere other than buid/res/all/<variant name>, between steps 1. and 2.
Now I only have to figure out where that is, so I can put the freshly generated ormlite_config.txt there if it's in any way possible or feasible...
As before, I would be extremely thankful if someone could enlighten me on this issue.
AN ALTERNATIVE (SIMPLER) APPROACH
As stated at the very top of the question, in the end I decided to go with an alternative approach which is much simpler and is not a hack of the .apk assembly process, as is what I originally intended to do.
The steps are as follows:
Code your database configuration generator class to write the ormlite_config.txt int your app's src/main/res/raw (You can use the DatabaseConfigGenerator class that I included at the very top of this question as a template). Keep this class withing the package structure of your app, don't make it a separate application or module, there is not reason to do this. So you can put it inside com.your.app.database or whatever.
In the top tool bar in Android Studio, click on the little drop-down box between the "Make" and "Run" buttons:
It will open a menu where you can choose one of your existing run configurations or edit the configurations. Choose the later:
The "Run/Debug Configurations" window will open, where in the top left corner you should click the little green plus sign and select "Application" as a type for your new configuration:
A form will open where you will define a run configuration for your DatabaseConfigGenerator, as it needs to be run as a Java application, separately from your Android app. There are only a few fields you need to modify here. First, give your new Run configuration a name (1), then select the DatabaseConfigGenerator as the main class (2), then under module classpath choose the module of your app wherein your DatabaseConfigGenerator and your model classes reside (3), then remove all entries from the "Before launch" section by selecting them and clicking the red minus sign (4), and finally, click "Apply" (5). Now you can click your app under the "Android Application" section to the left (6).
The very last thing you need to do here is also the most important one, which will put everything together to make your app first build itself, then generate an up-to-date ormlite_config.txt, and then build itself again (albeit much faster than the first time) so that this newly generated config is actually included in the final .apk. In order to accomplish this, you need to modify the "Before launch" section of your app's run configuration (1). Chances are you will already have a "Gradle-aware Make" in here, which is what actually compiles your app and packages it into an .apk during the usual build process. If you don't have it, add it as the first entry. After that, add another entry, but this time for the "Database Configuration Generator" run configuration, which you created a few steps back, as this will ensure that the ormlite_config.txt is generated based on freshly compiled model classes and is up-to-date. And finally, add another "Gradle-aware Make", to make sure that a new .apk is generated, which will now also include this up-to-date ormlite_config.txt. Now click "Apply" (2), and that's it!
From this point on, every time you click the "Run" button in the toolbar at the top of the Android Studio window, while the "app" run configuration is selected, you can be sure that the ormlite_config.txt in the resulting .apk will be up-to-date with whatever changes you made to your model classes or the DatabaseConfigGenerator itself.
For this solution I've taken inspiration from the following two SO answers:
Setup Gradle to run Java executable in Android Studio
Android Studio run configuration for ORMLite config generation
In the end I decided to put together the complete solution in a single place and describe it in detail right here.
There are three small caveats to this approach, and YMMV on whether you can live with them:
This only applies to the "Run" action, not the "Make" action, which means that you will have to initiate a run even in cases when you just want to build an .apk, but not actually run it. The resulting .apk can then be found under app/build/apk/ and is named depending on the variant you are building (for debug builds it will usually be app-debug-unaligned.apk, and for releases, app-release.apk).
This approach in itself means the "Gradle-aware make" will run twice every time you click "Run", which will result in slightly longer build times, but I haven't noticed much of a difference (the android gradle plugin is smart enough to recognize which resources have not changed since the last build and will skip a lot of unnecessary steps the second time around), resulting in maybe a 20% longer build time (don't hold me to the number).
If you are working in a team setting and are using version control, it sucks a little bit that this configuration is not trackable, so every developer in your team will have to go through this process individually and can't simply check it out as part of the repository in, say, .git. This is due to the fact that the run configurations are defined inside your project root, under .idea/workspace.xml, which is universally agreed to be something that should not be tracked in version control as it is machine specific.
There are probably ways to remove some manual steps of the process of defining your run configuration like this on a team level, but it doesn't seem possible to fully automate it in a clean way. I could be wrong though and feel free to let me know if that's the case.
Hope this helps!
The problem is that the resources are compiled by aapt when R.java is produced (and so before javac). Those compiled resources are then used to build the apk. AFAIK those compiled resources is just a kind of zip archive.
So, to achieve your goal you need to modify those compiled resources after javac compilation.
You can try to define a new task (let's call it addOrmToRes) invoking aapt with required arguments to modify the compiled resources.
aapt is located under <ANDROID_SDK_HOME>/build-tools/<version>/
The aapt Usage indicates that :
aapt a[dd] [-v] file.{zip,jar,apk} file1 [file2 ...]
Add specified files to Zip-compatible archive.
So executing something like this:
aapt add /build/apk/myapp.apk ormlite_config.txt
or this
aapt add <the_path_to_compiled_resources> ormlite_config.txt
in your new custom task addOrmToRes should do the trick. So to answer to your edit 3 : aapt add is probably the solution.
To integrate it in the build, something like this should work:
runOrmGenTask.dependsOn(variant.javaCompile)
addOrmToRes.dependsOn(runOrmGenTask)
addOrmToRes.dependsOn(<the task creating the apk>)
<the task signing the apk>.dependsOn(addOrmToRes)
Note that I didn't identify every tasks here. But I guess you get the idea.
Try to use dependsOn mechanism.
runOrmGenTask.dependsOn(variant.javaCompile)
And, then for example:
packageApplication.dependsOn(runOrmGenTask)
More about task manipulation you can get from here.
I wanted to explore the generated AndroidManifest.xml of my .apk and I realised that the format was not readable.
Both debug and release apks contain the encoded manifest file. How is that possible when progaurd is disabled?
Does Gradle modify the encoding during assemble process?
Is there anyway to disable that feature?
How is that possible when progaurd is disabled?
ProGuard has nothing to do with it. XML (manifest and resources) is converted into a "binary XML" format as part of the build process.
Does Gradle modify the encoding during assemble process?
Not specifically. Creating this binary XML is part of the build tools in general, whether invoked from Gradle, Ant, Eclipse, or whatever.
Is there anyway to disable that feature?
No. There are tools available that can decompile an APK, and some of those will convert the binary XML back into a text representation. Or use aapt dump badging. Or, use an on-device inspector, like App Browser.
The App Browser on-device inspector mentioned by Mark doesn't display the Android manifest very well (since it doesn't resolve resource references). There's a much better view given by App Detective