Develop Reference

how to display device address book ordered by custom external data

In my android application I would like to display the mobile user's contacts (names, profile picture ) displaying first the contacts that have already installed and registrered that application (the matching is made by contact's email).
Trying to loop over each contact and match if their email is already registered (in an external SQLITE table) don't seems to be an efficient way.
I would like to directly add (somewhere in the address book ?) the extra data "isRegistered = true/false) and order my addreess book query by this value to scroll the address book.
Is it possible? How to implement this in detail ?

OPTION 1
I think the most efficient way would be what you thought about initially, with a slight improvement:
store the list of registered emails (for the user's contacts) in a local SQLite DB.
read the entire list of emails on application launch, and store them in a HashSet
When sorting the contacts, create a custom Comparator to first check if the contact is an app user or not, and only then fallback to name compare.
OPTION 2
If you still want to check the option of storing the custom value in the Contacts DB, and integrate it into your query, you need to create a SyncAdapter.
This is basically a service that is able to sync contacts to/from a server into your own RawContact, which is then aggregated into one-or-more existing RawContacts, like Google does for Google Contacts.
You can set it to be notified when a new contact is added, and have your SyncAdapter add the needed info to the contact so it'll show links to your app.
If you go to your phone settings > accounts, you can see Whatsapp and Google's SyncAdapters there, where you can turn them off/on.
To create a sync adapter, you can follow the official docs, or this great tutorial.

Should I rely on SOURCE_ID when reading Android Contacts?

I am creating an android contacts app, so I regularly read user's contacts and store them in my app. To do this, I need to rely on some kind of ID so that I know which contact I should update (or add/delete) in my app and Contacts Provider supplies several of them:
CONTACT_ID is the aggregate contact id,
each aggregate contact consists of one or more Raw Contacts, each with its own RAW_CONTACT_ID,
and most important, each Raw Contact has a SOURCE_ID, which is supposed to be the server id, i.e. the id this contact has in this account's server.
I 've chosen to rely on SOURCE_ID, since this sounds like the most stable one. E.g. when user removes and re-adds the same account in their device, I wouldn't want this account's contacts to get different IDs, because I wouldn't be able to match them in my app.
However, only Gmail sync adapter seems to keep the promise documented below. Exchange sync adapters unfortunately do not, SOURCE_ID changes, and it's definitely not any server id, since it has a small number like 23:4.
Question: Any ideas how to overcome this problem? Am I using the right ID for the intended use? Does exchange adapter stores the "permanent server id" in some other field?
Documentation: The SOURCE_ID must be unique for each account type and should be stable across syncs:
Unique: Each raw contact for an account must have its own source id. If you don't enforce this, you'll cause problems in the contacts
application. Notice that two raw contacts for the same account type
may have the same source id. For example, the raw contact "Thomas
Higginson" for the account emily.dickinson#gmail.com is allowed to
have the same source id as the raw contact "Thomas Higginson" for the
account emilyd#gmail.com.
Stable: Source ids are a permanent part of the online service's data for the raw contact. For example, if the user clears Contacts Storage
from the Apps settings and re-syncs, the restored raw contacts should
have the same source ids as before. If you don't enforce this,
shortcuts will stop working.

LOOKUP_KEY is what you're looking for.
LOOKUP_KEY
An opaque value that contains hints on how to find the
contact if its row id changed as a result of a sync or aggregation.
You should use a pair of <CONTACT_ID, LOOKUP_KEY> to keep track of contacts.
In normal use, use the CONTACT_ID value, but if your code gets a hint that the CONTACT_ID has changed (either missing, or unexpected contact name), you can use the LOOKUP_KEY to find the new contact-id.
Or you can use Contacts.getLookupUri() to get a URI you can always use to quickly find a contact no matter what its CONTACT_ID or LOOKUP_KEY actual values are.

First thing, it's not a good idea to store the Id in your application and hoping that the Id won't change over time and be consistent. You are right about 'SOURCE_ID' column that it's more consistent compared to the other two (CONTACT_ID is the most brittle while 'RAW_CONTACT_ID' persists at least until user logs out of an account and logs in again).
We had a syncable account with contacts and we used to keep the unique id in one of the general purpose columns in the 'raw_contacts' table (SYNC1 - SYNC10). So although Google suggests that account providers use the database columns in a certain manner but it's upto the provider totally.
General rule to follow is, never use those id's for long term persistence and if you do so, expect them to change. Also since you are making a contact App, you obviously need some kind of reference key. In such case, don't go by the rule that all account providers will put their key in the same column. It's brittle but that's the way it is.
Edit - You should use ContactsColumns.LOOKUP_KEY (previous answer also cited the same). As per the Google Documentation -
LOOKUP_KEY
Added in API level 5 String LOOKUP_KEY An opaque value that contains
hints on how to find the contact if its row id changed as a result of
a sync or aggregation.
Constant Value: "lookup"
https://developer.android.com/reference/android/provider/ContactsContract.ContactsColumns.html#LOOKUP_KEY
You can fetch the lookup key if you have the contact Id using API's provided. Look here - https://developer.android.com/reference/android/provider/ContactsContract.Contacts.html

difference between RawContacts.CONTENT_URI and ContactsContract.Contacts.CONTENT_URI in android

what is the difference between RawContacts.CONTENT_URI and ContactsContract.Contacts.CONTENT_URI in android?
I am trying to write a service which listens to changes in native address book. So, which one to use?
thanks
Niz

A Contact (as the user perceives it) is an aggregate set of RawContact's
A RawContact being some details associated with a certain account or protocol
The ContactsContract.Contacts.CONTENT_URI is generally the correct one to use as it brings back the correct display name from the entire set of matching RawContacts and can help you join all those accounts that are linked
The RawContacts.CONTENT_URI can be used if you want to match a specific [set of] contact relating to an account/protocol
and then there are various data types / directories to bring back even broader ranges of data.
The question of which one should you use comes down to what data you are watching and if/how you are displaying it, as well as how many rows you are trying to watch. I would say more often than not it's correct to use the ContactsContract.
It also depends how granular you want your 'notifications'. I.e. a content observer callback can give you a URL to show you what changed, but nothing more, which means if you want to only act on the specific row that you know has changed then you're going to have to observe different things than if you just want to know some change has occurred and that it's time to notifyDataSetChanged!

How do I robustly compare phone numbers in Android?

In an Android app I am developing, I have a database which maps:
<Contact ID, Contact Lookup Key> --> <My Custom Data Associated with Contact>
When an incoming SMS message comes in, I want to be able to look up My Custom Data Associated with Contact associated with the contact who sent the SMS message. I want to do the lookup in as few steps as possible.
The only data I can get out of the SMS message is the Phone Number, so the problem is basically one of mapping the Phone Number to My Custom Data Associated with Contact in as few steps as possible.
Now, I am aware that I can look up a contact from a phone number up using ContactsContract.PhoneLookup as described e.g. here. However, this does not get me to my desired result because both the Contact ID and Contact Lookup Key are subject to change. So if either the ID or the Lookup Key has changed for any contact I have persisted to my database, I have to basically load my whole database and refresh the ID/Lookup Key for each row before I can test whether they represent the same contact as the one returned by ContactsContract.PhoneLookup. This is a pretty brutal solution which I would prefer to avoid.
What I would like to do is keep a table mapping phone numbers to my custom data (in an N:1 relationship).
<Phone Number> --> <My Custom Data Associated with Contact>
This way, in theory at least, I can index the phone number column and just look up the data in one step. However, phone numbers come in many formats. My solution will only work if I can create a canonical representation of a given phone number. My plan was to use PhoneNumberUtils.getStrippedReversed to produce an efficiently indexable canonical phone number, but this fails on a trivial case, for example:
PhoneNumberUtils.getStrippedReversed("+1(306)6656320") --> 02365566031
PhoneNumberUtils.getStrippedReversed("(306)6656320") --> 0236556603
They aren't the same number!
So with that by way of long-winded explanation, my question is: Can anyone think of a way of converting a phone number to a canonical phone number so that I can perform an indexed database lookup on the number, or any other solution to the problem I described above?

You can use below method along with getStrippedReversed
public static String toCallerIDMinMatch (String phoneNumber)
*Returns the rightmost MIN_MATCH (5) characters in the network portion in reversed order This can be used to do a database lookup against the column that stores getStrippedReversed() Returns null if phoneNumber == null*
Then your comparison will give you the desired result

Possibility of a custom Contacts field with a set list of values and Contacts lookup performance

I'm pretty sure it's not viable to do what I'd like to based on some initial research, but I figured it couldn't hurt to ask the community of experts here in case someone knows a way.
I'd like to create a custom field for contacts that the user is able to edit from the main Contacts app; however, the user should only be allowed to select from a list of four specific values. A short list of string values would be ideal, but an int with a min/max range would suffice.
I'm interested in knowing if it's possible either way, but also wondering if it make sense to go this route performance wise. More specifically, would it be better to look up a contact (based on a phone number) each time a call or SMS message is received or better to store my own set of data (consisting of name, numbers, and the custom field) and just syncing contact info in a thread every so often? Or syncing contacts the first time the app is run and then registering for changes using ContentObserver?
Here is a similar question with an answer that explains how to add a custom field to a contact.
Thanks in advance.

I don't see the purpose to have your own set of data against contacts stored in your separate database, as you obviously will run into sync issues. You can use the mimetype and store whatever you want against the contact, in the manner you linked.
Whenever you want to looup contacts you can do that by using your custom mimetype. You get the contact id (from ContactsContract.Data.CONTENT_URI), and then you run another query to get the contact details (from ContactsContract.Contacts.CONTENT_URI). Please note these are different tables.
I'd like to create a custom field for contacts that the user is able to edit from the main Contacts app
I don't see that possible, editable from the main app, when you use your custom mimetypes, and you don't have much options here. The main contact app will display only the fields that are in SDK. You can store details against contacts but they won't show up in the inbuilt edit contact screen.