Develop Reference

How to modifiy the system apps running on my phone

I want to modify the system apps running on my specific phone. I am stumped about some important steps on how to do that - see my questions further below.
N.B.: I am not interested in deploying these modifications to other phones. Instead I want to play with the original code and improve it. Hence "create your own system app" is no sufficient answer.
Background
I am just starting to get into Android development (have sufficient Java knowledge, though). There are plenty of great tutorials out there (e.g. here and here) but they all seem to cover user application development. I'm more interested in getting (a little) "closer to the metal". Unfortunately I am unable to find good sources teaching how to do that.
Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum) or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).
Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?
In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:
some core apps deployed by the vendor
the image binaries (some way down the page)
sources for the Linux kernel and some other libraries
Tools
I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).
Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?
Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?
I'm asking because I assume that AVD already runs an unmodified version of that very app.

I am just starting to get into Android development (have sufficient Java knowledge, though). ... I'm more interested in getting (a little) "closer to the metal".
You can of course follow your own bliss, but you may find it pretty frustrating to learn Android development by hacking on the OS itself. You'll find yourself doing a lot of OS debugging and working with harder-to-use tools than normal.
Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum)
I realize this is just an example, but this could be a tough one, because there are a lot of things that contribute to brightness control -- there's the app with the UI that sets it, but that talks to the framework and ultimately the driver for the hardware that actually deals with brightness curves for the display, in coordination with the ambient light sensor.
or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).
There's a dialer app, but implementing a feature like that will be...difficult, provided it's even feasible. I don't know offhand whether that sort of call state is available to the API.
Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?
If you're replacing system apps, you'll need to run an OS signed with debug keys; you don't have access to the signing keys for the installed system apps on a retail build of the OS that would enable you to replace them. That will mean either getting a debug-keys installation or building it yourself from source, and flashing it to your phone. I would most strongly discourage you from using your daily phone for this sort of work -- you don't want it to be bricked when you need to receive an important call.
If you're not running a Nexus phone, then yes, your vendor has made modifications. The system apps are probably modified, and you almost certainly won't have access to the closed source of those apps. Also, there will be closed-source device drivers that are essential to the operation of the OS that you won't have access to. If you're trying to get AOSP Android up and running on such a device, as an individual hacker it's at best very, very difficult. If you're willing to use something like CyanogenMod, you could look to see if (and to what extent) they support your hardware.
If you are running a Nexus phone, then it's a goal of the AOSP project to try to have the OS bootable on select Nexus devices, though you won't necessarily enjoy the fullest functionality of the retail OS build, depending on the state of drivers -- even Nexus phones can have closed-source drivers and bits that make life more, um, exciting for building with AOSP.
In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:
some core apps deployed by the vendor
Almost certainly
the image binaries (some way down the page)
Not sure what you mean by that
sources for the Linux kernel and some other libraries
It's unlikely there are significant modifications to the kernel itself, though vendors sometimes do...interesting...things with system configuration. I already mentioned device drivers. Vendors also have custom implementations/modifications of some framework APIs.
Tools
Vendor-specific tools are rare.
I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).
Much of what you want to do can probably be implemented in Java, though it depends on how close you're getting to the hardware which can be difficult to know.
Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?
The "system apps" are implemented in Java, but again, you sort of have to know where the dividing line is.
Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?
That's actually different, because you're dealing with an emulator system image instead of a phone image. The emulator system images are completely open source, and you can build them from AOSP.
I'm asking because I assume that AVD already runs an unmodified version of that very app.
That's correct.
To come back to my initial point, if you read between the lines of what I'm saying, I won't tell you it's impossible, but the difficulty ranges from freakin' tough on the easy end to spend-years-of-your-life on the hard end. If you really want to proceed, your best bet may be to buy the phone that's best supported by CyanogenMod and work that angle; there's a more active developer community there to help you. I know you're more excited about trying to customize the OS than you are in making a user-space app, but you'll experience less frustration with the latter than the former.

Writing a custom Android OS

I hope this is the right place to ask this.
I would like to work on an Android build with a completely custom OS. I was inspired by this project that ported Inferno to Android.
I would like to do this legally (obeying all terms of service and what-not) if possible, and I have no problem with disclosing my source code to a carrier.
Do any of you have experience with this? What are the considerations for getting a custom build to work with a carrier?
Since I am building on Android, I probably won't have to worry about cell/3g/4g hardware drivers, and I would like to buy a phone from the carrier anyway just to make sure it will be supportable.

I'm one of the creators of the Inferno project you mentioned. Anyway, making a completely custom OS for an Android phone will be difficult; a lot of the hardware will be undocumented or only drivable by Linux binary blobs. What we did (and what Boeing has also done recently for a commercial project) is strip away all the Java from an Android system, leaving a basic Linux upon which you can build your own custom interface. This lets you use all the hardware (since the drivers are in Linux), but everything that the end users see will be your stuff. There are a variety of ways to go from there. Inferno implements a virtual machine, so we can abstract the Linux stuff into our own Inferno-style world.
I don't think there should be any problems with the carriers. We started with a Cyanogenmod ROM and hacked on that; custom ROMs are just fine! I never had any problems, even when I was swapping my little pay-as-you-go SIM card among several phones running Gingerbread, ICS, and Inferno.

Head over to the AOSP source, and clone it. Strip out what you don't need in your branch and begin the kernel patching! You'll need to modify the kernel in portions to make it compatible (fairly) with the device you're on. The Android kernel is in fact a very close cousin of the Linux Kernel, and in Linux 3.3, they've begun merging the two. After this is done, write your interfacing mechanisms (be it a display, web interface, etc) and you can test it out on your platform.

Guidance regarding Android Kernel Porting from one device to other

I own a Samsung Galaxy 3 and want to port kernels available for other Android Devices. I have all building environment ready. I have C knowledge also. What exactly I dont understand is how and what all things porting involves, which codes should be modified in what way?
If someone could help me. It would be great.
Thanks in advance.
Ok I got it. But the real problem is I dont understand one thing.
What needs to be changed in the source code so that to make it compatible in other device? Can someone explain me that?

Idolon gives a good suggestion. In theory what is written there is true. In practice, it's a little more involving. I've been porting Android Kernel for months now. My goal was to have a custom S5PV210 processor module on my custom board with my custom I/O. I have the source code for Samsung's galaxy tab and several Samsung's android smartphone as well as a few versions (Froyo, Gingerbread) for the Samsung's Evaluation board (SMDKV210 for S5PV210 CPU). FYI, S5PC110 is a close derivative. Here is what I found out.
Eventhough all the device drivers are supposed to be built in a very modular way and independent from each other and you should be able to replace them with similar devices, the way Samsung did it is not exactly that. To give you one example is about power management. In many low level drivers dealing with hardware devices, it calls some specific routine for other hardware devices like power management chip. So, When you change the choice of drivers in the "make xconfig" or "make menuconfig" it will remove the source code of what you don't want and add the one you want but, there are still other modules that are calling the low level function you removed. When building, you will get tons of unresolved external because you removed a power management chip that your hardware doesn't have but is used in the source you started with. I looked through several Samsung's device source code and they have branched very early on. They have been adapted by different teams and it would be a major amount of work to go from one device to the other from one source kit.
Here is how I succeeded: I bought an S5PV210 evaluation board with an already ported Android Kernel (Gingerbread). All the power management and inter dependant issues where already cleaned-up. Then, form a working kit, I could change the device I needed (the one you can't buy without a 100K unit/yr commitment) without getting stuck with the interdependancy. Then I could even change LCD resolution (from 800x480 to 1024x768), touchscreen, Cell modem etc etc. The whole thing takes about a month worth of work with only one guy (me).
Starting with a Galaxytab or other commercial device made me waste months of headakes with no useful results but the thing I've learned.
There is one requirement to succeed on my approach. You have to know the hardware you are working on. As an example, you need to know what touch screen your hardware has (chipset) to select the driver and were is it connected to hook it up to the right device (USB, Serial etc). That the same thing for all other devices (power managemnt chip, keypad, backlight, LCD etc etc)
Hope that is useful to give you an idea of the work involved and how you can do it.

You should take a look at the cached copy of Android Platform Developer's Guide and at android-porting google group.
There is also an old but useful article about porting Android to Nokia N810, which will give you some hints about Android Linux kernel porting.

I am also into android porting stuff for quite some time I suggest the following route :
Read
You need to have basic knowledge of android porting and AOSP source code, directory stucture hierarchy.
I suggest you to start reading https://books.google.co.in/books/about/Embedded_Android.html?id=plHsngEACAAJ&hl=en
and also refer https://source.android.com/devices/index.html as good guys pointed out you can also google android porting related groups and become a member there, if you are stuck on a issue there are chances similar issue might have been faced by someone else previously.
Observe
AOSP code :
Parallely you need to dive in to the android source e.g from links like https://android.googlesource.com/ observe which components are placed where in the source code, what are the updates from previous android release (like ART replaced dalvik in LL), at top level there would be generic code, code specific to your hardware called HAL in hardware/ and device/ folder.
Kernel code :
In the kernel also you need to observe the directory structure and know which things lie where like SoC specific code will be in arch/ directory defconfigs (used for selecting kernel configuration) will be arch//configs/ directory.
Also there are good books available for linux kernel, you can google them and start reading them also.
While porting kernel you need to take care of following
1.If in the new kernel there is already support of SoC which is used in hardware to be ported then you need to add only device specific changes like suppose you want to port new kernel version to samsung galaxy s3, you can take a reference android kernel having exynos support and then you need to make changes only for adding support for samsung galaxy s3.
For that you need to have reference of some old kernel having support of s3, from there you can study patches which have been added to add support for s3 and port those patches to newer kernel.
Experiment
First you can only port the bare minimum changes required to up the platform, compile and flash it on your platform, then observe the output, if everything looks fine than go on for next changes otherwise try to solve the issue.
Thanks,
Devarsh

Porting Android on various other mobiles

I am new to the development as well as the prg world. I want to learn android and want to make applications for it. So one possible way is to buy a new android phone to test it live and not on emulator. But I have two spare phones, Nokia N70 and Motorola MotoRokr E6. So I was willing to port the OS on those machines. As a result I wanted to know is that possible then what all do i need for that considering both Software and Hardware Aspects.
ThankYou in Advance...

Its possible however I would strongly recommend against it.
While android is open source and nominally can be run on any system you choose, in reality you would have to start worrying about keyboard incompatibility, drivers for any integrated peripherals your phone has, lacking buttons that are standard to all android phones etc.
Getting android running well on one of these phones would be a large project in its own right. I certainly wouldn't recommend testing your android apps on it as a first port of call as the problems you uncover are as likely to be with your android port as with your app.

Neither phone has anywhere near sufficient memory.
You need 96-128 mb, preferably more like 256 mb of ram.
Something that was already linux-based would be simplest as you would start by adding the android kernel modifications. Alternatively, something that is a winmo cousin to a vendor's existing android phone and has basically comparable hardware so you can hopefully recycle drivers or leverage an existing porting effort.
But if you need to aquire a phone anyway, get one that already runs android.

Wine like for Android

I am pondering the idea of a Wine-ish compatibility layer on Android.
The idea is to run Symbian apps on it as both OSes share ARM hardware.
I have no knowledge of Symbian but I think that given the hardware capabilities of Android devices this could be done with less effort than Wine's windows emulation.
What would be the most significant difference to overcome in this emulator? (threading, storage, ...)

The real problem is not going to be code execution, but the API's to do things like graphics, interact with hardware, accept input, etc. If you have documentation of the original and android has the capability, API translation layers would be possible.
But android's security model outright prevents a number of things that are possible on other phone platforms, and combined with it's "java apis only" allows only inefficient means of doing things that can be done more efficiently on others.
This is of course all about application-level emulation/api translation. If you are willing to modify the android platform itself, supporting just about anything else for which you have documentation (and licensing?) within the hardware capability of the device should be possible.

Hardware capabilities of a device have nothing to do with complexity of an emulator to be hosted. It depends on Symbian's design and complexity.
And, even more, licencing. Even if one could make a Symbian emulator for Android, its legality would be questioned.
It's difficult to answer your question in detail, but since Symbian is open sourced (and Android too), if you've got the time, it shouldn't be too hard to see what sets them apart.

QT is used for the latest symbian OS, and has been ported to Android, you could write apps in QT build for each platform

the problem for writing an emulatir are variouss.
If the Symbian apps are written in in an interpreter language like Basic or similar then an emulator couldn't be too difficult to write. I did such stuff once to make the same code run on linux and windows, and I used a translation API for all calles coming from the software directed to UI, input/output.
I wound guess that the UI capabilities of Symbian are a subset of the Android functions so it would be not too difficult to write a WINE alike thing or an interpreter that runs the Symbian code on different hardware - IF it is only in high language.
But be aware there could be some machine code in the appps and that is processor specific. Most of the Android tabs nowadays run on Tegra, Tegra2 or (soon) on Tegra3, some may run on StrongArm or Arm, some may run on Intel Atom (x86 architechture), so this might get more or less impossible if the CPU isn't binary compatible like ARM / ATOM. Then you need to emulate the CPU as well and that might eat so much performance that you need a 4-5 times stronger machine to run that stuff smoothly.
It won't be too difficult to crack Android to execute Linux-alike binaries, but for sure this "mod" will affect the ability to use or download stuff from regular appstores.
With some apps you might have even more headache, e.G. my MP3 player from Korea runs on Strongarm, but it also executes Flash games from various sources. When there is no Flash player - and Google announced something like dropping support for Adobe Flash - it won't be usable.
The "most wanted" is obviously the Ovi Maps, probably that stuff could be easily converted to another app having offline navigation capability :-) People wrote "Gaia" some years ago, an open source viewer for Google Earth (and later forced to give up) so it can't bee too difficult to realize at least this.