Can someone explain why ashmem was created?
I'm browsing through mm/ashmem.c right now. As near as I can tell, the kernel is thinking of ashmem as file-backed memory that can be mmap'd. But then, why go to the trouble of implementing ashmem? It seems like the same functionality could be achieved by mounting a RAM fs and then using filemap/mmap to share memory.
I'm sure that ashmem can do more fancy stuff -- from looking at the code, it seems to have something to do with pinning/unpinning pages?
Ashmem allows processes which are not related by ancestry to share memory maps by name, which are cleaned up automatically.
Plain old anonymous mmaps and System V shared memory lack some of these requirements.
System V shared memory segments stick around when no longer referenced by running programs (which is sometimes a feature, sometimes a nuisance).
Anonymous shared mmaps can be passed from a parent to child processes, which is inflexible since sometimes you want processes not related that way to share memory.
Can someone explain why ashmem was created?
David Turner (a regular on Android NDK) answered this in Why was bionic/libc/include/sys/shm.h removed?:
... System V IPCs have been removed for cupcake. See
bionic/libc/docs/SYSV-IPC.TXT for details.
In brief, System V IPCs are leaky by design and do not play well in
Android's runtime environment where killing processes to make room for
other ones is just normal and very common. The end result is that any
code that relies on these IPCs could end up filling up the kernel's
internal table of SysV IPC keys, something that can only safely be
resolved by a reboot.
We want to provide alternative mechanism in the future that don't have
the same problems. One thing we provide at the moment is ashmem, which
was designed specifically for Android to avoid that kind of problem
(though it's not as well documented as it should). We probably need
something similar for semaphores and/or message queues.
Related
On a recent SO question, I explained how calling a RenderScript kernel multiple times will effectively force all threads to be globally synchronized between calls.
I am currently working with multiple convolutions applied in sequence to image data. Since the convolution algorithm requires reading surrounding pixel data of the input image, I have implemented a workflow where my own custom kernel is called multiple times -- to make sure that at every step, all data from the previous convolution is ready and available at the correct coordinates. This technique has worked great for me so far.
However, in my constant quest for optimization, I have noticed that there is much performance to be obtained by keeping intermediate values in local registers for a thread, instead of writing them back to the global memory allocation in between kernel calls. If I were able to chain these convolutions in such a way, things would run much quicker. The problem is obviously that accessing the registers of surrounding threads is not really possible. Furthermore, this would require threads to run in synch to make sure these intermediate values in between stages get calculated in the expected order.
In CUDA and OpenCL, these issues are very common, and are addressed by well-known barrier synchronization + shared memory tiling techniques, which in turn depend on the concept of CUDA thread blocks or OpenCL work groups. I believe these concepts are non-existent in RenderScript, as this issue is very much tied to the wildly different architectures between desktop-class GPU's and mobile SoC's.
So my obvious question here is, are such things possible in RenderScript? That is, better management of threads and possibly thread groups for quicker data sharing among them.
On the Google I/O 2013 RenderScript talk by Jason Sams and Tim Murray, it is discussed how Script Groups might be able to do some behind the scenes optimizations, such as cross-device parallelization, memory tiling, and kernel fusion; all this by analyzing at runtime the dependency DAG in the group, and either automatically creating allocations where needed or possibly optimizing them away. I'm assuming this last bit referes to fusing kernels so that they work off their own local data, kind of how I mentioned above keeping data in local registers and combining separate steps inside a single kernel.
All this seems very much in line with what I'm looking for, especially since my application is indeed a well-defined DAG of inter-dependent operations (for a Convolutional Neural Network). So if Script Groups are indeed a plausible mobile-centric alternative to these mechanisms, I'm wondering if there is any way of influencing how and where these optimizations happen. Or if not, how much can the runtime be trusted to make the correct inference from my data dependencies given the hardware its running on -- in the specific case of "surrounding" pixel data access of the convolutional algorithm.
I realize this might all still be work in pogress, and methods would be highly hardware dependent at this point. So if there is no straight solution for such matters at the present time -- I'd be very much willing to accept a speculative answer on how this kind of workflow might potentially be approached by RenderScript in future releases.
I'd be immensely grateful on some insight about this, as it would greatly affect the development direction of my own project going forward, not to mention there are surely many other people out there wondering how such general parallel computing tasks can be handled in RS.
Thank you very much!
As you've discovered, there's no way in RS to directly share data across threads. However, what you are describing can be done using a ScriptGroup. The catch is that each script in the group has to be unique, so you cannot feed your same script over and over. At least, not as it is written now. You could certainly put the "core" of your script in a RS header and include it from multiple kernels. The ScriptGroup allows you to have the output from one script become the input of another, or the output of one script becomes a global field in another. The documentation states that the kernel to kernel (output to input) is the more efficient use case. Using this approach, your synchronization issue would be resolved as the engine will execute the first script against the entire input data set before starting the second script, etc. The scripts themselves will be parallelized appropriately for the hardware (using either CPU or GPU/DSP). The engine will not have to pop back out to Java between scripts and can also manage the data allocations behind the scenes, if needed.
Something you may notice is the ScriptGroup utilizes Script.KernelID or Script.FieldID in order to identify the exact script or field in which to connect two kernels. Your custom scripts have these things auto-generated as long as you explicitly call out your kernel function using the RS compiler attribute pragma. Then you can call getKernelID_<name> (where 'name' is the kernel function name from your script) to get the kernel ID.
Even a private member/function of my class can be accessed by reflection by using setAccessible(true). Is there a way to prevent this kind of access from outside code?
I read something, here on stack-overflow, that I can use SecurityManager for prevention of reflection in applets(not sure how it works, though), but is there a similar mechanism for Android as well? Maybe an annotation or clever-programming?
Taking a step back, what you're observing is a difference in security philosophy, between the Java execution model as originally embodied in JVMs at Sun and the execution model of Android.
The original Java VM design was intended for a system wherein multiple, mutually-suspicious applications (or "applets" in the Java parlance) would simultaneously inhabit a single address space, running in a single VM. Because the designers didn't want one app to be able to mess with another, they went through great pains to define an intra-VM security model that would disallow things such as one object touching the private fields of another object of a different class.
That said, the Java library ended up having various "escape hatches" out of the security model. One of those is setAccessible() on reflection objects, as you note.
Android's model is different: Android uses processes as the security boundary and unit of application isolation, instead of trying to insinuate it into the process as was done with traditional JVMs. This renders moot the entirety of the Java security model, except in that it helps an application "save it from itself." That is, it's good design to not have an object poke into another object's private parts, and the default Java security model provides just that.
Leaving aside the question of people modifying your code, with Android, as an application author, you control all the code that ends up running inside the process of your app. If you choose to include code that calls setAccessible() that's your business. You might be shooting yourself in the foot, but you certainly won't be shooting any other apps' feet, since the Android security model, running as it as at the layer of processes, inherently doesn't let that happen. Likewise, using native code will totally break you out of the Java object model, which allows for the possibility of things going totally higgledy-piggledy in the process but also allows you to express some things in a more productive manner than you could in Java. It's a trade-off, but it's a per-application-developer tradeoff and not one that particularly impacts anything else that's happening on your phone / device.
I know this doesn't directly answer your question, but I hope it provided some useful context.
Is there a way to prevent this kind of access from outside code?
Not really.
is there a similar mechanism for Android as well?
Even if there is (and I am not aware that such a thing exists), anyone can remove it, by decompiling your code (assuming they do not have your source already), getting rid of the protection, and recompiling the code.
Bear in mind that ProGuard, when used properly, will obfuscate your private classes and methods for your production APK builds. That, plus a lack of documentation, will make it tedious for anyone to gain access to those private classes and methods.
I don't believe that you can ever really 100% protect from users using reflection on your project with malicious intent. You can make it more difficult for users to do it by doing things like obfuscating your code, but it is still possible to reflect on the obfuscated code.
I don't believe SecurityManager can be used for the purpose that you are suggesting, though I could be wrong.
A coworker and I were talking (after a fashion) about an article I read (HTC permission security risk). Basically, the argument came down to whether or not it was possible to log every action that an application was doing. Then someone (an abstract theroetical person) would go through and see if the app was doing what it was supposed to do and not trying to be all malicious like.
I have been programming in Android for a year now, and as far as I know if -- if -- that was possible, you would have to hack Dalvik and output what each process was doing. Even if you were to do that, I think it would be completely indecipherable because of the sheer amount of stuff each process was doing.
Can I get some input one way or the other? Is it completely impractical to even attempt to log what a foriegn application is doing?
I have been programming in Android for a year now, and as far as I know if -- if -- that was possible, you would have to hack Dalvik and output what each process was doing.
Not so much "hack Dalvik" but "hack the android.* class library, and perhaps a few other things (e.g., java.net).
Even if you were to do that, I think it would be completely indecipherable because of the sheer amount of stuff each process was doing.
You might be able to do some fancy pattern matching or something on the output -- given that you have determined patterns of inappropriate actions. Of course, there is also the small matter of having to manually test the app (to generate the output).
Is it completely impractical to even attempt to log what a foriegn application is doing?
From an SDK app? I damn well hope so.
From a device running a modded firmware with the aforementioned changes? I'd say it is impractical unless you have a fairly decent-sized development team, at which point it is merely expensive.
This is both possible and practical if you are compiling your own ROM. Android is based on Linux and I know several projects like this for Linux, like Linux Trace Toolkit. I also know of research into visualizing the results and detecting malicious apps from the results as well.
Another thing functionality like this is often used for is performance and reliability monitoring. You can read about the DTRACE functionality in Solaris to learn more about how this sort of stuff is used in business rather than academia.
Suppose you have an application that spawns a local HTTP server on an Android device.
Will there be any advantage to running it in a separate application instead of spawning a separate thread?
Since the heap size is capped per application, I'm assuming that there is more breathing space for memory when running a separate application.
Apart from this, in terms of performance are there any other benefits (or disadvantages) such as a bigger chunk of CPU time?
The one big disadvantage I see that happens when running application components in different threads is that the two parts will be in different DVMs. This can make sharing Preference changes, Listeners, Observers, ect, not work as you would expect, you would also have to make sure all DB access is synchronized.
To counter this point if you synchronization correctly and don't need preferences you can use bundles or AIDL to communicate back and forth from the 2 applications. The best bet is AIDL for 2-way continuous communication, but be aware AIDL can be expensive. The other option for communication is a socket... but this goes against the SDK that is provided. A trick I have learned when doing this is to create an API jar file to include in both applications that will handle all communications (by way of intent or AIDL - blackbox approach).
Personally I think application components that are similar should stay in the same DVM and application unless they can run as stand alone then you have to be the judge of that.
Have you though about running the HTTP server as on ongoing foreground service? This would uncouple your design as well as make things easy and lighter.
Using separate processes can significantly increase the memory footprint of your app. Not only do you get a multiple of the core Dalvik overhead (figure 2-3MB per process), but none of the RAM used by your app can be shared (such as static symbols and such in your code).
Plus you have more CPU overhead because you now need to do IPC for any interaction between those different parts of your app that cross boundaries. And you have a lot more complexity in implementation because you actually need to implement that IPC and figure out how to correctly manage these different parts of the app that are running in isolated address spaces.
For the vast majority of situations, I don't think it is good to use multiple processes.
There is a remote service example in SDK.
If you go with two apps, you will absolutely need remote service. And that is only way to communicate among apps, as my experience.
From my viewpoint, it is not good with the concept "server" to be used on a cell phone. But perhaps you have your reason...
First, you don't need separate application for that, you define another process for your own application. In most cases doing such thing will boost your performance because there is bigger chance that your process will actually run on separated physical process. How ever the Android OS does not support this yet, not event in Android 4.03.
So the only benefit you will get from this is memory, witch in my opinion should not be a reason for opening another process.
In most cases you will get a boost in the performance of you app if you run your server in a separate process via service but not always.
NOTE: service can run in a separate process of its own but for that you have to provide android:service tag in the xml.
One big flaw in this type of design which I think you already know is that , in android each process runs in its own virtual machine.So if you spawn a new process it will get its own VM. Now you yourself consider which is better.Running one VM for the whole application or running two of them. (In most cases I have heard of at least in android 1 VM is more than sufficient to handle everything you need)
Apart from that another flaw is that when you separate the process from the main process of your app e.x you are running your server service in a separate process then it may not shutdown even though your app comes across an exception or error as its a separate process from that of the main process and is not tied to the life cycle of your app anymore.So it can lead to some unexpected behavior and can cause your app to malfunction.
If you are OK to take the risk than go with it otherwise go with threads(I would recommend Asynctask within a service) for the purpose you are seeking as it will provide you with almost the same functionality while being safe inside the application scope/lifecycle.
hope this helps.
I have implemented a simple shared memeory code which is scattered in the two processes(1 acts writer and other acts as reader). But I want manage this SHM code(just like a memory manager),which works independent of any reader/writer processes. By simply giving some hooks/pointers to out side, Can any one suggests me a way for this. or any related code or links regarding related information to this ? One more this Can I use Zygote process to make it happen please suggest ?
An application cannot "share" its memory using plain pointers on a modern operating system. This is something which requires the assistance of the OS, and is highly dependent on the OS in question. For instance, on Linux the best bet would be to use SysV Shared Memory.
Make sure you understand the overhead of multiple process shared memory and ask yourself if just using threads would not suffice. In most cases, threads will suffice, or if not you should re-think your model to use a message passing/shared nothing model.
Have a look at what Boost.Iterprocess can do for you. Especially have a look at the Managed Memory Segments section.