Develop Reference

How to modifiy the system apps running on my phone

I want to modify the system apps running on my specific phone. I am stumped about some important steps on how to do that - see my questions further below.
N.B.: I am not interested in deploying these modifications to other phones. Instead I want to play with the original code and improve it. Hence "create your own system app" is no sufficient answer.
Background
I am just starting to get into Android development (have sufficient Java knowledge, though). There are plenty of great tutorials out there (e.g. here and here) but they all seem to cover user application development. I'm more interested in getting (a little) "closer to the metal". Unfortunately I am unable to find good sources teaching how to do that.
Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum) or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).
Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?
In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:
some core apps deployed by the vendor
the image binaries (some way down the page)
sources for the Linux kernel and some other libraries
Tools
I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).
Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?
Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?
I'm asking because I assume that AVD already runs an unmodified version of that very app.

I am just starting to get into Android development (have sufficient Java knowledge, though). ... I'm more interested in getting (a little) "closer to the metal".
You can of course follow your own bliss, but you may find it pretty frustrating to learn Android development by hacking on the OS itself. You'll find yourself doing a lot of OS debugging and working with harder-to-use tools than normal.
Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum)
I realize this is just an example, but this could be a tough one, because there are a lot of things that contribute to brightness control -- there's the app with the UI that sets it, but that talks to the framework and ultimately the driver for the hardware that actually deals with brightness curves for the display, in coordination with the ambient light sensor.
or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).
There's a dialer app, but implementing a feature like that will be...difficult, provided it's even feasible. I don't know offhand whether that sort of call state is available to the API.
Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?
If you're replacing system apps, you'll need to run an OS signed with debug keys; you don't have access to the signing keys for the installed system apps on a retail build of the OS that would enable you to replace them. That will mean either getting a debug-keys installation or building it yourself from source, and flashing it to your phone. I would most strongly discourage you from using your daily phone for this sort of work -- you don't want it to be bricked when you need to receive an important call.
If you're not running a Nexus phone, then yes, your vendor has made modifications. The system apps are probably modified, and you almost certainly won't have access to the closed source of those apps. Also, there will be closed-source device drivers that are essential to the operation of the OS that you won't have access to. If you're trying to get AOSP Android up and running on such a device, as an individual hacker it's at best very, very difficult. If you're willing to use something like CyanogenMod, you could look to see if (and to what extent) they support your hardware.
If you are running a Nexus phone, then it's a goal of the AOSP project to try to have the OS bootable on select Nexus devices, though you won't necessarily enjoy the fullest functionality of the retail OS build, depending on the state of drivers -- even Nexus phones can have closed-source drivers and bits that make life more, um, exciting for building with AOSP.
In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:
some core apps deployed by the vendor
Almost certainly
the image binaries (some way down the page)
Not sure what you mean by that
sources for the Linux kernel and some other libraries
It's unlikely there are significant modifications to the kernel itself, though vendors sometimes do...interesting...things with system configuration. I already mentioned device drivers. Vendors also have custom implementations/modifications of some framework APIs.
Tools
Vendor-specific tools are rare.
I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).
Much of what you want to do can probably be implemented in Java, though it depends on how close you're getting to the hardware which can be difficult to know.
Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?
The "system apps" are implemented in Java, but again, you sort of have to know where the dividing line is.
Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?
That's actually different, because you're dealing with an emulator system image instead of a phone image. The emulator system images are completely open source, and you can build them from AOSP.
I'm asking because I assume that AVD already runs an unmodified version of that very app.
That's correct.
To come back to my initial point, if you read between the lines of what I'm saying, I won't tell you it's impossible, but the difficulty ranges from freakin' tough on the easy end to spend-years-of-your-life on the hard end. If you really want to proceed, your best bet may be to buy the phone that's best supported by CyanogenMod and work that angle; there's a more active developer community there to help you. I know you're more excited about trying to customize the OS than you are in making a user-space app, but you'll experience less frustration with the latter than the former.

Tablet development for a dedicated system

I need to make an architectural decision for developing (actually porting) my embedded solution.
I will try to present my case as clearly as possible, and any advice I can get will be appreciated.
Introduction
I have an embedded system, currently developed on ARM11 architecture and ArchLinux OS. It is implemented using all kinds of technologies available under Linux, including C, C++, Bash and Python.
At this time, I would like to port my solution to a tablet, so I am trying to make some decisions about the architecture, based on the requirements of my system.
Requirements
The system is modular, and it runs multiple processes and threads. It also communicates to remote servers and controls the hardware peripherals. These are the basic requirements, at the moment, I will update as discussion develops:
Primary:
Dedicated system (minimum amount of other applications running, even in the bg)
Multiple processes, ability to set priorities
Ability to assign a process to a single CPU Core (cpu affinity)
Inter-process communication mechanisms
Complete hardware control (WiFi, 3G, GSM, mic, speaker, display, ...)
Creating sockets, etc.
Other:
Ability to connect a microphone directly to a 3.5mm plug (TRRS connector)
Mainstream solution to ensure reliability
Future-proof: minimize the porting effort for new tablets and HW
My questions
What tablet and OS combination would meet these requirements?
How to approach the "dedicated solution" requirement?
How to approach the software development, what language and tools to use?
My investigation so far
My investigation so far has been concentrated on the OS choice. The main options seem to be Android and Ubuntu Touch. Here are my thoughts:
Android
Android wins in the mainstream category obviously, but...
I have no experience of Android development, but as far as I can tell, I can either develop a Java application that runs on top of Dalvik, or I can go native via Android NDK. Maybe I can even bypass the whole thing and go native side-by-side of Dalvik, and develop in Python? I guess then I will lose the access to the API for HW access. Not sure how I would access the HW then. But if I go with Java development, this is a sandbox solution, and I am not sure if I can have such a control over the processes, HW and CPU core affinity?
Ubuntu Touch
Developing on Ubuntu Touch would be more like Linux development I am used to, since it uses Qt. The issue here is that the applications are developed using the SDK that restricts me to HTML5 and QML, which I'm not sure can allow me the same control over the system I need. If I use Python and avoid the SDK, same issue arises - how do I control the HW? Of course, there is a way to do it like on a regular embedded system I guess, but I don't want to reinvent the wheel if I don't have to.
Also, it seems that Ubuntu Touch hasn't been ported to many devices yet, only a handful of them are supported.
Finally
I am not sure how well I have presented my case, but I will update the question as needed with further explanations and requirement details. Thank you for your patience, your time and any help you may offer.

I am in the same boat as you, we are developing embedded system on ARM CPU. Currently the system is written in Python on custom built linux distro from hardware vendor.
We have been looking at Android and Ubuntu Touch for a few months. So far we didn't get any decision yet. But here what we found out from our incomplete analysis:
Android is quite heavy system, that also brings JVM into the mix. So you HW and memory requirements will significantly increase.
Android likes to take over the entire system, so sometimes you might be fighting with Android.
Android has very good IPC mechanisms, not available on mainstream linux kernel, such as Binder. Of course you can port it to another system. But this would be considerable effort.
Android has very rich GUI interface, also with things like NDK view you are not confined into the JDK box.
Android doesn't use standard GNU C library, it uses something called BioniC which is a port of C library from BSD. Usually it is not a problem for most software, but if you are using some linux specific low level features, like working with serial ports you will need to tweak your code. See what I had to do to port Java RXTX library to android https://github.com/vladistan/gnu.io.android
You can find a lot of Android developers out there who will do basic GUI programming for you, while you are concentrating on your app specific stuff. Interfacing between GUI and low level things is quite easy using binder. And of course you can bring in whatever mecahnism you want via NDK.
We have considered Ubuntu Touch. But so far it doesn't seem to be mature enough.

Guidance regarding Android Kernel Porting from one device to other

I own a Samsung Galaxy 3 and want to port kernels available for other Android Devices. I have all building environment ready. I have C knowledge also. What exactly I dont understand is how and what all things porting involves, which codes should be modified in what way?
If someone could help me. It would be great.
Thanks in advance.
Ok I got it. But the real problem is I dont understand one thing.
What needs to be changed in the source code so that to make it compatible in other device? Can someone explain me that?

Idolon gives a good suggestion. In theory what is written there is true. In practice, it's a little more involving. I've been porting Android Kernel for months now. My goal was to have a custom S5PV210 processor module on my custom board with my custom I/O. I have the source code for Samsung's galaxy tab and several Samsung's android smartphone as well as a few versions (Froyo, Gingerbread) for the Samsung's Evaluation board (SMDKV210 for S5PV210 CPU). FYI, S5PC110 is a close derivative. Here is what I found out.
Eventhough all the device drivers are supposed to be built in a very modular way and independent from each other and you should be able to replace them with similar devices, the way Samsung did it is not exactly that. To give you one example is about power management. In many low level drivers dealing with hardware devices, it calls some specific routine for other hardware devices like power management chip. So, When you change the choice of drivers in the "make xconfig" or "make menuconfig" it will remove the source code of what you don't want and add the one you want but, there are still other modules that are calling the low level function you removed. When building, you will get tons of unresolved external because you removed a power management chip that your hardware doesn't have but is used in the source you started with. I looked through several Samsung's device source code and they have branched very early on. They have been adapted by different teams and it would be a major amount of work to go from one device to the other from one source kit.
Here is how I succeeded: I bought an S5PV210 evaluation board with an already ported Android Kernel (Gingerbread). All the power management and inter dependant issues where already cleaned-up. Then, form a working kit, I could change the device I needed (the one you can't buy without a 100K unit/yr commitment) without getting stuck with the interdependancy. Then I could even change LCD resolution (from 800x480 to 1024x768), touchscreen, Cell modem etc etc. The whole thing takes about a month worth of work with only one guy (me).
Starting with a Galaxytab or other commercial device made me waste months of headakes with no useful results but the thing I've learned.
There is one requirement to succeed on my approach. You have to know the hardware you are working on. As an example, you need to know what touch screen your hardware has (chipset) to select the driver and were is it connected to hook it up to the right device (USB, Serial etc). That the same thing for all other devices (power managemnt chip, keypad, backlight, LCD etc etc)
Hope that is useful to give you an idea of the work involved and how you can do it.

You should take a look at the cached copy of Android Platform Developer's Guide and at android-porting google group.
There is also an old but useful article about porting Android to Nokia N810, which will give you some hints about Android Linux kernel porting.

I am also into android porting stuff for quite some time I suggest the following route :
Read
You need to have basic knowledge of android porting and AOSP source code, directory stucture hierarchy.
I suggest you to start reading https://books.google.co.in/books/about/Embedded_Android.html?id=plHsngEACAAJ&hl=en
and also refer https://source.android.com/devices/index.html as good guys pointed out you can also google android porting related groups and become a member there, if you are stuck on a issue there are chances similar issue might have been faced by someone else previously.
Observe
AOSP code :
Parallely you need to dive in to the android source e.g from links like https://android.googlesource.com/ observe which components are placed where in the source code, what are the updates from previous android release (like ART replaced dalvik in LL), at top level there would be generic code, code specific to your hardware called HAL in hardware/ and device/ folder.
Kernel code :
In the kernel also you need to observe the directory structure and know which things lie where like SoC specific code will be in arch/ directory defconfigs (used for selecting kernel configuration) will be arch//configs/ directory.
Also there are good books available for linux kernel, you can google them and start reading them also.
While porting kernel you need to take care of following
1.If in the new kernel there is already support of SoC which is used in hardware to be ported then you need to add only device specific changes like suppose you want to port new kernel version to samsung galaxy s3, you can take a reference android kernel having exynos support and then you need to make changes only for adding support for samsung galaxy s3.
For that you need to have reference of some old kernel having support of s3, from there you can study patches which have been added to add support for s3 and port those patches to newer kernel.
Experiment
First you can only port the bare minimum changes required to up the platform, compile and flash it on your platform, then observe the output, if everything looks fine than go on for next changes otherwise try to solve the issue.
Thanks,
Devarsh

Wine like for Android

I am pondering the idea of a Wine-ish compatibility layer on Android.
The idea is to run Symbian apps on it as both OSes share ARM hardware.
I have no knowledge of Symbian but I think that given the hardware capabilities of Android devices this could be done with less effort than Wine's windows emulation.
What would be the most significant difference to overcome in this emulator? (threading, storage, ...)

The real problem is not going to be code execution, but the API's to do things like graphics, interact with hardware, accept input, etc. If you have documentation of the original and android has the capability, API translation layers would be possible.
But android's security model outright prevents a number of things that are possible on other phone platforms, and combined with it's "java apis only" allows only inefficient means of doing things that can be done more efficiently on others.
This is of course all about application-level emulation/api translation. If you are willing to modify the android platform itself, supporting just about anything else for which you have documentation (and licensing?) within the hardware capability of the device should be possible.

Hardware capabilities of a device have nothing to do with complexity of an emulator to be hosted. It depends on Symbian's design and complexity.
And, even more, licencing. Even if one could make a Symbian emulator for Android, its legality would be questioned.
It's difficult to answer your question in detail, but since Symbian is open sourced (and Android too), if you've got the time, it shouldn't be too hard to see what sets them apart.

QT is used for the latest symbian OS, and has been ported to Android, you could write apps in QT build for each platform

the problem for writing an emulatir are variouss.
If the Symbian apps are written in in an interpreter language like Basic or similar then an emulator couldn't be too difficult to write. I did such stuff once to make the same code run on linux and windows, and I used a translation API for all calles coming from the software directed to UI, input/output.
I wound guess that the UI capabilities of Symbian are a subset of the Android functions so it would be not too difficult to write a WINE alike thing or an interpreter that runs the Symbian code on different hardware - IF it is only in high language.
But be aware there could be some machine code in the appps and that is processor specific. Most of the Android tabs nowadays run on Tegra, Tegra2 or (soon) on Tegra3, some may run on StrongArm or Arm, some may run on Intel Atom (x86 architechture), so this might get more or less impossible if the CPU isn't binary compatible like ARM / ATOM. Then you need to emulate the CPU as well and that might eat so much performance that you need a 4-5 times stronger machine to run that stuff smoothly.
It won't be too difficult to crack Android to execute Linux-alike binaries, but for sure this "mod" will affect the ability to use or download stuff from regular appstores.
With some apps you might have even more headache, e.G. my MP3 player from Korea runs on Strongarm, but it also executes Flash games from various sources. When there is no Flash player - and Google announced something like dropping support for Adobe Flash - it won't be usable.
The "most wanted" is obviously the Ovi Maps, probably that stuff could be easily converted to another app having offline navigation capability :-) People wrote "Gaia" some years ago, an open source viewer for Google Earth (and later forced to give up) so it can't bee too difficult to realize at least this.

Is it posible to port Android for a Symbian phone

I have a Symbian based phone with a ARM Cortex-A8 processor (SonyEricsson Vivaz) and was thinking on how hard would it be to try and port the Android OS for it. Obviously Android runs on a myriad of devices with different hardware so I imagine it shouldn't be too hard to adapt it to SE hardware. Could some one give me a clue where to start or if this is even undertakable...

How much information do you have about the hardware in the phone ? Are you starting completely from scratch ?
Porting Android is not simple task. First thing is to have Linux running (preferably 2.6.32 for more compatibility with the latest releases of AOSP.
If you can find a kernel that can run on your phone, that's one big step. After that, you want make sure that the peripherals you want to use also have the right drivers : touch interface, LCD display, SD card, audio, video. The ones that are probably most difficult will be wifi, radio (GSM) and power management module.
If you get that far, you don't have that much to go anymore, a few more changes in the kernel needed for Android, be able to compile Android correctly file system, hook up a few things like buttons and correctly interface with the drivers mentioned above...
But overall, definitely not an easy task (IMHO).