Develop Reference

how to build an APK and separate libraries that the app loads dynamically

The short summary is: How do I build an APK and separate libraries (by which I mean sets of classes (and ideally, resources too) in some form, such as JAR, AAR or DEX files), but not include those libraries in the APK; instead, the app loads them at run time?
Detail
So my main question is how to build such an app (e.g. Gradle configuration). How do I specify which classes go into which JAR or DEX files? Do I create an Android Studio module for each DEX file I want to end up with?
A closely related question is how the Java code should then load the external libraries and access their classes at run time. For the latter, I'm hopeful that the approach shown at accessing to classes of app from dex file by classloader would work.
I've tried the instructions at https://developer.android.com/studio/projects/android-library.html, but that builds an APK that does include the dependency library.
I've also tried Multidex (https://developer.android.com/studio/build/multidex.html), but that doesn't seem to leave the developer any control over which classes go in which DEX file, and furthermore, packages them all into a single APK. AFAICT there is no way to control the loading of these DEX files at run time.
Background
There's a possibility of the "X-Y problem" here, so I'd better explain the background.
I'm building an app for a client. It's not going to be distributed through an app store, so it won't have access to the normal mechanism for updates. Instead, the client wants the app to be able to update itself by downloading new components of itself to replace the old components, without a need to manually sideload a new APK. The primary motive here is that the updates have to be easy for non-technical users. If the app can control the update process, it can make it smooth and guide the user.
Moreover, the app will be used in areas where internet access is scarce and expensive, so the client wants to be able to issue app updates in smaller chunks (e.g. 2MB) rather than forcing the user to re-download the whole app to receive a small update.
One aspect of the requirements I should mention, in case it matters, is that the libraries to be loaded at run time are supposed to live on a microSD card. This can also help with distribution of updates without internet access.
The current status of the app is that it's about 50% written: That is, a couple of earlier versions have been released, but the app now needs to be modified (restructured) to meet the above requirements, as well as others.

This tutorial is a good start for external loading of DEX files.
Only three small files of source (MainActivity.java, LibraryInterface.java, LibraryProvider.java) and it copies secondary_dex.jar from the assets folder, into internal application storage [outdex/dex] (the internet is also stated as possible in the tutorial).
You have to build it with ant, because it uses custom build steps.
I tried it, it works fine. Worth a look.
custom class loading in Dalvik and ART
UPDATE
this code has been ported to Android Studio gradle (no need for ant).
https://github.com/timrae/custom-class-loader
Tested ok. Copies com.example.toastlib.jar from the SDcard into internal application storage [outdex/dex],(not assets folder).
( you must read the README.md file in the project to build it).
Q: How do I add an Activity, I cannot add it to the manifest ? A: Use
Fragments, they don't need entries in the manifest.
Q: A Jar with resources that is meant to be added to an existing
project needs to be able to merge its resources with the project's
own resources (R.). A: Hacks are available, Data file...Packaging Android resource files within a distributable Jar file
Q: The external file has wrong permissions. A: Import it.
Q: I need to add uses-permission. A: Use API23 you can programmatically add uses-permissions (but they still need to be declared in the Manifest, so the new permissions model is probably not much use to us).
This section is for more general users (#LarsH has more specific requirements about updates), The example above is 17kb apk and 1 kb jar. You could put the bulk of you code in the one-off jar, and updates would involve just loading an new Apk (and then importing the bulk code jar, to minimise the data transfer).
When the Apk gets too big, start again with a small Apk and everything migrated to another jar (import 2 jar's). You need to balance coding effort, user experience, maintainability, supportability, bandwidth, android rules, play store rules (if these words even exist ;O)).
NOTE Dalvik is discontinued
The successor of Dalvik is Android Runtime (ART), which uses the same bytecode and .dex files (but not .odex files), with the succession aiming at performance improvements transparent to the end users. The new runtime environment was included for the first time in Android 4.4 "KitKat" as a technology preview, and replaced Dalvik entirely in later versions; Android 5.0 "Lollipop" is the first version in which ART is the only included runtime.

You could try to build multiple apk's with the same sharedUserId and the same process.
This is the plugin mechanism used by Threema
Edit: More about Theema
Threema has one main app and two plugins:
main app: https://play.google.com/store/apps/details?id=ch.threema.app
QR-Code Plugin: https://play.google.com/store/apps/details?id=ch.threema.qrscannerplugin
Voicemessage Plugin: https://play.google.com/store/apps/details?id=ch.threema.voicemessageplugin
Doing so the main app does not need the permissions for accessing the camera or microphone

How can I make Android load dynamically generated dalvik bytecode without writing to a file first?

For the past six months as my final university project, I've been writing a PlayStation 1 emulator in Java to prove it can be performant - part of my strategy involves writing a custom class loader that imports bytecode I have just generated from an array into a new class - in effect a Java bytecode dynarec core which speeds up the emulated CPU orders of magnitude (in theory). All quite possible on an Oracle JVM, and done before by others.
My question is, aside from the fact I would need to generate dalvik bytecode rather than Java bytecode, there doesn't seem to be anyway to dynamically load classes into a running Android app that doesn't involve loading them from a dex file on flash somewhere. I know similar things have been asked before, but as I would eventually like to port this emulator (and have it be quicker than its currently unplayable speed), is there anyway around this? I don't want to be continually writing to flash when a new section of MIPS code is converted to bytecode, as it could wear the flash out and probably isn't very fast either.
A thought I had was maybe mounting a tmpfs using a small JNI lib and storing class files there to be loaded, so in effect storing them in RAM as before - is this even possible for an unprivileged app to do though? I'd appreciate peoples input/thoughts.

No, that might be possible on a jailbroken device but it's not possible in a sandboxed app.
I tried several ways to load dynamic code on Android but the only feasible way is via the DexClassLoader where the dex file must be stored in a privileged region.
You can have a look at my project Byte Buddy where I implemented such class loading: https://github.com/raphw/byte-buddy/blob/master/byte-buddy-android/src/main/java/net/bytebuddy/android/AndroidClassLoadingStrategy.java

What is Smali Code Android

I am going to learn a little bit about Dalvik VM, dex and Smali.
I have read about smali, but still cannot clearly understand where its place in chain of compilers. And what its purpose.
Here some questions:
As I know, dalvik as other Virtual Machines run bytecode, in case of Android it is dex byte code.
What is smali? Does Android OS or Dalvik Vm work with it directly, or it is just the same dex bytecode but more readable for the human?
Is it something like dissasembler for Windows (like OllyDbg) program executable consist of different machines code (D3 , 5F for example) and there is appropriate assembly command to each machine code, but Dalvik Vm also is software, so smali is readable representation of bytecodes
There is new ART enviroment. Is it still use bytecodes or it executes directly native code?
Thank you in advance.

When you create an application code, the apk file contains a .dex file, which contains binary Dalvik bytecode. This is the format that the platform actually understands. However, it's not easy to read or modify binary code, so there are tools out there to convert to and from a human readable representation. The most common human readable format is known as Smali. This is essentially the same as the dissembler you mentioned.
For example, say you have Java code that does something like
int x = 42
Assuming this is the first variable, then the dex code for the method will most likely contain the hexadecimal sequence
13 00 2A 00
If you run baksmali on it, you'd get a text file containing the line
const/16 v0, 42
Which is obviously a lot more readable then the binary code. But the platform doesn't know anything about smali, it's just a tool to make it easier to work with the bytecode.
Dalvik and ART both take .dex files containing dalvik bytecode. It's completely transparent to the application developer, the only difference is what happens behind the scenes when the application is installed and run.

High level language programming include extra tools to make programming easier & save time for the programmer. After compiling the program, if it was to be decompiled, going back to the original source code would need a lot of code analysis, to determine structure & flow of program code, most likely a few more than 1 pass/parse. Then the decompiler would have to structure the source based on the features of the compiler that compiled the code, the version or the compiler, and the operating system it was compiled on eg. if an OS specific features or frameworks or parsers or external libraries were involved, such as .net or dome.dll, and their versions, etc
The next best result would be to output the whole program flow, as if the source code was written in one large file ie. no separate objects, libraries, dependencies, inheritances, classes or api. This is where the decompiler would spit out code which when compiled, would result in errors since there's no access to the source codes & structure of the other files/dependencies. See example here.
The 3rd & best option would be to follow what the operating system is doing based on the programmed instructions, which would be machine code, or dex (in case of Android). Unless you're sitting in the Nebuchadnezzar captained by Morpheus and don't have time to decode every opcode in the instruction set of the architecture your processor is running, you'd want something more readable than unicode characters scrolling on the screen as you monitor the program flow/execution.
This is where assembly code makes the difference; it's almost the direct translation of machine code, in a human readable format. I say "almost" direct because microprocessors have helpers like microcodes, multithreaders for pipelining & hardware accelerators to give a better user experience.
If you have the source code, you'd be editing in the language the code is written in. Similarly, if you don't have the source code, and you're editing the compiled app, you'd still be editing in the language the code is written in; in this case, it's machine code, or the next best thing: smali.
Here's a diagram to illustrate "Dalvik VM, dex and Smali" and "its place in chain of compilers".

How to distribute file collections as separated apk packages in android?

I have a small Android application that uses different sets of files (a couple of images, a small SQLite DB and a couple of XML files) depending on the specific task at hand.
I know I can include my files into the main application APK using resources or assets but I would be happy to distribute them in a separated APK.
How can I create a data-only APK file?
How can I distribute it? In particular, do I have to do anything special for a data-only package (for example for associating it to the main application package in some way)?
(I'm intentioned to give the user a link to the data package and ask him to install it. No automatic installation required.)
How can I install my files into the internal or into the external storage area of my application? Is it possible at all to install files into the internal storage area created by the main application installer? Do I have to set any particular permission for this?

My approach to this would be to create a wrapper app that's nothing but a content-provider and serves up the files per request by your main app. This would allow you to supply different data packages for the user -- you could even have your main app select between those relatively easily.

It looks like that the commonly accepted way to have the same application with different contents (or styles, or configurations) is to use an Android Library Project for the common code (that is: the whole application, the "engine", the "app framework") and a standard Android Application Project for the contents (that is: an application that actually contains just data). A little bit confusing, just because the "library" here is actually the whole "app", but this seems to be the way to go.
More in detail:
Create an Android Library Application and put into it as much code as you can (all of the non-changing stuff). Please note that this library cannot be launched and cannot be distributed alone. It must be included in a hosting application.
Create a standard Android Application. Include your library into this project. Put in /res and in /asset all of your data (files, XML, etc.).
Compile everything and distribute.
Repeat this cycle every time you need a different version. Different because of data, style, configuration or anything else. Publish the resulting app with a new name.
For what regards me, I'm not completely satisfied by this approach.
A possible alternative is preprocessing the source code with Ruby, Python, Perl, GIT, bash, Ant, Maven, Rake or any other tool that is able to read a file from here, make some change here and there, and write the file there.
The general outline is something like this:
Make a "template" application. Leave your /res and /assset empty.
Run a custom-made script. The script reads a configuration file, copy the /res and /asset files from your repository into the project /res and /asset directories, changes some Java source file and creates/changes some XML file.
Compile and distribute (with a new name, of course).
Using GIT or other SCMs, you just make a new branch for every new version and compile it. Not very elegant (because it can strongly interfere with the normal use of the SCM) but...
There are a few example of these approaches on the web. I'm not completely satisfied by them, either.
Frankly, what the Android ecosystem should offer to solve this problem is some kind of "in-app package manager". Something like the Eclipse Update Manager. This would allow us to use the same application framework to handle different scenarios.
As an alternative, a solid, officially-supported, template-based code-generation mechanism would be nice. Something in the spirit of "Software Production Line": https://en.wikipedia.org/wiki/Software_production_line . Have a look at fw4spl, for example: http://code.google.com/p/fw4spl/ .

how does android store and run programs

Android stores it's programs in APK format, which is a modified version of ZIP/JAR.
When these APK files are installed, they are stored in /system/app/$APKNAME.apk.
Some of the apps in this dir also have a $APKNAME.obex file.
These APK files contain some of the fallowing
META-INF
MANIFEST.MF
CERT.RSA
CERT.SF
SHA1-Digest
res
AndroidManifest.xml
classes.dex
resources.arsc
So what I want to know is what are the .obex files and are androids program decompressed from the APK/ZIP/JAR at runtime and how?

The way that this works is pretty interesting, and gives some key insights into Android's runtime model. The first thing I'd recommend watching is Dalvik VM internals, if you plan on doing any serious amount of systems stuff with Android. (Although, it's obviously old.) Now, when the Android package manager gets an intent which requires starting a new app it forks off a new virtual machine from an already running zygote process. This is basically a technique which allows the system to get a lot of nice memory properties (sharing the pages mapped, etc..). Then, the system loads up a (potentially pre optimized and verified) file to load so the vm can start executing it. You should read this document, which will tell you quite a bit about how this works. (Perhaps this thread will also help.) Keep in mind that as all systems are different -- for example, if you're on a new architecture, you won't get JIT support unless you explicitly write it! -- you can't know for sure how Dalvik will load code to run your app.