Develop Reference

How to messengers receive notification messages from servers?

Currently I am working on an alert app, which receives push notifications from a server. Reading a lot about that topic on the internet lead me to firebase which can be used to achieve that.
Now I was wondering: Are Whatsapp, Signal and other messengers using firebase or do they have any other solution to receive messages from servers even if the app is not active opened?

They use push messaging. It might be firebase, it might be another provider. MQTT is popular. All any of those things do is open up a persistent socket connection to a server and listen for messages. The trick is that whatever process is doing that needs to be whitelisted from power management restrictions, because you don't want to delay your message notifications for 15 minutes.
Now if you're writing something that's supposed to be robust queueing on the server side and deduping on the client side would be a good idea. But the basics is open socket connection and block on it on a thread.

Building an xmpp server for upstream google gcm

Let's say have an app that has 10s of millions of installs and 10s of thousands of active users at a given point of time. I need to log my users' activity data to my servers. Currently, I make HTTP requests from the device to my servers. I have a bunch of machines running a web server, sitting behind amazon's ELB. They parse the data coming from the devices and put it in mongodb.
Now, I would like to capture device data by using upstream CCS provided by Google' GCM (so that I can piggyback on GCM for more reliable delivery of data) I have written a prototype XMPP server and I can make whole thing work, but I am worried about scaling it up. What will happen if Google starts sending me messages at a rate faster than I can consume? Earlier, I was able to use multiple servers behind load balancer to tackle high request rate. Is there a concept of load balancing here?
If I open multiple connections from my server to Google's server (Google says I can have till 1000 connections for a given sender id), will the incoming requests be load balanced between these connections?
Finally, is there recommended solution which takes care of solving most of the problems above? Will using ejabberd solve some of the problems above?
Thanks a bunch.

What will happen if Google starts sending me messages at a rate faster than I can consume?
At the end https://developers.google.com/cloud-messaging/ccs you may read
Conversely, to avoid overloading the app server, CCS stops sending if there are too many unacknowledged messages. Therefore, the app server should "ACK" upstream messages, received from the client application via CCS, as soon as possible to maintain a constant flow of incoming messages. The aforementioned pending message limit doesn't apply to these ACKs. Even if the pending message count reaches 100, the app server should continue sending ACKs for messages received from CCS to avoid blocking delivery of new upstream messages.
In the same document, you find partial answer to your second and third questions
If at any point the connection fails, you should immediately reconnect. There is no need to back off after a disconnect that happens after authentication.
For me it means, that Google implemented a simple redundancy logic and probably not a fair load balancing system (anyway I hope so). If you have that high volumes, I suggest you to contact them directly.
For the last ones, ejabberd is a good product, there are a lot of deployed systems with a clustered infrastructure and a plenty of documents on how do taht. I suggest you to start from here http://docs.ejabberd.im/admin/guide/clustering/ .
Anyway, for your high volumes I would evaluate RabbitMQ which is another Erlang jewel.

ejabberd can be clustered and placed behind a load balancer to distribute connections. A 3 or 4 server cluster should be able to handle that load fine and give you fail over protection. You can add servers if needed. Once you get close to 10 servers you may want to consider using Redis for the in memory DB rather than mnesia.

Chat - receiving a message

I have a question regarding a simple messaging application which I want to write.
Lets say we have 1000 clients. Now client A writes a message to client B.
Of course, client A sends the message to a server which distributes the messages. But the next step is unclear for me:
Does the server send the message to the specific device of client B? If so, how does it address explicitly this device and not send it to the other devices as well?
Or does client B continuously check for new messages on the server which are addressed to client B?
I am not sure if idea 1 is even possible, and what is the best way. But I just want to ask for how it is normally done.

Those are both valid ways to create a chat platform. The problem with option 2, which is generally referred to as a "polling" model (where the client "polls" the server for new messages on a regular basis) is scaleability. In your example alone if you have 1000 clients you would have 1000 requests to the server for new messages every interval. To give the appearance of real time messaging this interval needs to be pretty short (i.e. a few minutes at most). If you do the math you can see that the volume of requests can balloon very quickly.
The better approach is option 1. Using something like OpenFire and the Smack API what you're describing - the server sending messages to client B - is possible. You can read more about the APIs here. The idea is the same as push notifications. Client A sends a message to the server which then will "push" that message to Client B. This is scalable and eliminates the need for polling (which kills server resources and the phone' battery with constant HTTP requests).

Avoiding a 100ms http request to slow down the REST API where it's called from

I'm developing a multiplayer Android game with push notifications by using Google GCM.
My web server has a REST API. Most of the requests sent to this API send a request to Google GCM server to send a notification to the opponent.
The thing is on average, a call to my API is ~140 ms long, and ~100 ms is due to the http request sent to Google server.
What can I do to speed up this? I was thinking (I have full control of my server, my stack is Bottle/gunicorn/nginx) of creating an independent process with a database that will try to send a queue of GCM requests, but maybe there's a much simpler way to do that directly in bottle or in pure python.

The problem is that your clients are waiting for your server to send the GCM push notifications. There is no logic to this behavior.
You need to change your server-side code to process your API requests, close the connection to your client, and only then send the push notifications.

The best thing you can do is making all networking asynchronous, if you don't do this yet.
The issue is that there will always be users with a slow internet connection and there isn't a generic approach to bring them fast internet :/.
Other than that, ideas are to
send only few small packets or one huge in favor of many small packets (that's faster)
use UDP over TCP, UDP being connectionless and naturally faster

I've solved my problem thanks to this thread:
I'm using Celery to send my notifications through a task queue.
I can't believe how simple it is!
Thanks anyway :)

Ways to guess if C2DM is connected

I'm trying to place a best guess estimate as to whether C2DM messages can be received.
I've created an application that relies on pushing information to a phone while it is physically inaccessible. I understand that C2DM isn't guaranteed delivery, but I'd at least like to know when the delivery of a message is even possible; when it isn't we fall back to our own push service (and can actually tell when we're connected).
I've noticed C2DM on android will still issue auth tokens even when there is not a logged in google account; messages still seem to be delivered in this instance even though it's stated that they shouldn't be. If GTalk isn't connected (firewall or other reasons), no response at all is returned when requesting an auth token. Auth tokens are returned to the application when the phone is in airplane mode. This means it's not as simple as checking if an internet is available. I can't find a reliable way of checking if GTalk is logged in.
Again, I don't need to guarantee the delivery of messages, but I'd at least like to know if delivery is even possible. Does anyone have interesting solutions?

Go watch this video, it's a Google I/O talk about C2DM, how to use it and how it works. AFAIK, you can't know if it's connected or not. Probably most of the time they don't even know (until they have to deliver a message and fail).
However, it is highly recommended (in the video as well) that you do not send important data through C2DM (as messages can get lost). The service should only be used as a "network tickle" (with a footprint as small as possible). Your application should be woken up by this tickle and it should start fetching the information it needs itself.
Now, if you implement it this way, it should be easy to implement a polling mechanism. Since you already separated the "tickle" from the actual information retrieval, you can just trigger the retrieval every once in a while if there's no tickle.
Something you can do to check whether C2DM is connected is something like a ping:
Send message to phone via C2DM
The app receives (or doesn't receive) the message and sends a "pong" back to your server
The server waits for the "pong" for a predetermined amount of time (1-2 minutes, I'd say) before marking the device as "offline".
Edit: relying on GTalk is not feasible. GTalk relies on C2DM just like your app, it doesn't have anything "extra". Also, GTalk is not present on all devices. I'm not sure how the GTalk app determines whether it's offline or not (it's not open source, unfortunately), but I'd guess it just tries to ping a server and fails.

No that is impossible.As you device authenticated once and generate registration ID and send to third party server(As you already know).Now your work is over once the device has been registered.So Wait for message either you got or not(No guarantee of delivering message as C2DM used UDP Protocol ).
Alternative Solution
Although its impossible to check from Google side directly as i mentions above,But if you have any urgency to check connection from your phone
then you can take approach like this
Step 1): Make one Web service to check connection
Step 2): Call this web service from application that will command to server to send push notification for checking purpose.
Step 3): Now from server side,server will immediately send push notification for particular device(FROM which it get command)
Step 4): Now if you got push notification that means you are still connected to C2DM.
this will not take much time.But follow it only when checking connection is urgent and it is on user

This may be a bit naive as I am not an active C2DM user, but wouldn't it be possible to read
/proc/net/netstat
and see if there are any active TCP connections. If there aren't any, then C2DM can't possibly be working. You could also make this technique more versatile by forming a C2DM whitelist that you would expect to find (or maybe its possible to filter on a special C2DM port?)

If the device is inaccessible, even your fallback push messaging system wouldn't work. The C2DM doesn't guarantee that it will deliver your message, but the event of non-delivery would be very rare. So would be the case with any other service. The best workaround that you could have is to poll your server to check if you have any new messages that hasn't been delivered yet. I am assuming that your application is such that it's very important not to miss even a single message in 500 or may be 1000. In that case, you could implement a hybrid of push and pull.

I've worked a little with C2Dm, I've created my own push 3rd party server .
I've implemented a little logic based oh C2DM http response code to know if a push message was sent or not .
Here is some of the code I used :
int responseCode = conn.getResponseCode();
if (responseCode == HttpServletResponse.SC_UNAUTHORIZED || responseCode == HttpServletResponse.SC_FORBIDDEN) {
LOGGER.warn("Unauthorized - need token");
return false;
}
here I'm almost sure that the push message was sent from the c2dm servers because I've got an id ont the response:
if (responseParts[0].equals("id")) {
LOGGER.info("Successfully sent data message to device: " + responseLine);
return true;
}
I've used other methods to get other result codes from Google if you want I can post them.I hope that I've helped you a bit .

I don't think there's any way to determine in ADVANCE whether there's any chance a push attempt will work, but I can think of a fairly straightforward way to verify receipt (but not queueing for future delivery via C2DM) -- just complete the message loop.
Remember, C2DM's main benefit is that it allows notifications when then phone is asleep and nominally offline. Once your application gets the notification, there's little to stop you from waking up the phone at that point, bringing up the network, and sending a confirmation. I don't think you'd even have to request "keep phone awake" permissions, because I believe the mere act of having registered for C2DM notifications and receiving one is sufficient to wake up the phone and allow the app to continue running normally (at least, long enough to bring up the network and send the confirmation).
While you're at it, you should keep track of confirmations that happen LONG after you expected them to be a lost cause. If you see more than a few, you might have to alter the resend strategy.
The only real-world edge case where this might fail is if you had users who bent over backwards to disable data while leaving voice/SMS enabled (I'm pretty sure C2DM uses 4 bytes of the response datagram sent when a phone polls for incoming calls & text messages that were originally set aside for RIM, then later repurposed for Apple and Google).

try to shut down all network connections and reconnect again. if you will get an registration id, then you can receive messages.

Something related to the long running C2DM-connection that is used to deliver the triggers:
On WLAN it sends a heartbeat every 15 minutes.
On mobile networks the timeout is 28 mintes.
28 Minutes might be to long, depening on the hardware your mobile carrier uses, 2g/3g repeaters in garages, etc.
You can get lots of information about the connection by opening the Google Talk Service Monitor Application: http://www.honeytechblog.com/monitor-google-talk-service-android/
Dial: ##8255##
Theres also a button that sends a heartbeat right now and resets the timeout.
If you want to ensure (on the client side) that c2dm-messages can be received at a given time, your best bet is to re-send the heartbeat. This can be done programmatically - only on rooted devices though. I might release an apk to the market sometime that does exactly that.