We are using GCM’s XMPP protocol to deliver push notifications to our customers, the problem we are facing is that when sending xmpp messages with high speeds, we don’t receive 'ack' or 'nack' messages from GCM anymore, These are the results of several tests we did for this matter :
500 XMPP messages - sent every 0.25 seconds:
all messages are acked
500 XMPP messages - sent every 0.1 seconds:
On average, ~4 messages remained un-acked
500 XMPP messages - sent every 0.01 seconds:
72 messages remained un-acked
500 XMPP messages - sent with no sleep time (As fast as possible)
reached the 100 unacked message limit set by GCM in less than 0.5 seconds !
The results are even worse when we go for more than 500 messages, E.g :
4000 XMPP messages - sent every 0.1 seconds:
On average, number of un-acked messages rose to about 16
4000 XMPP messages - sent every 0.01 seconds:
On average, reached the 100 unacked limit in the 800th xmpp message.
——————————————————————————————————
These results are from tests done on Google’s own cloud (Google cloud compute servers), while doing them in anyother place, would yield much worse results ( as we tested, no speed more than 1Msg/0.4S w
We are using GCM’s XMPP protocol to deliver push nould actually survive (!) the 100 unacked limit)
This is too bad for us, since there’s no optimal solution out, what should we do now?
Ignore the 100 unacked limit and continue sending, but that would mean we don’t know whether our messages are received by gcm or not.
We can resend any unacked message after several seconds, but duplicate messages (to same clients) are a result.
Wait to see if they maybe get acked in the near future, but that so far hasn’t worked. When we get to the unacked limit, pending messages are never acked (in our experience)
Limit the speed by which we send XMPP messages, but this solution greatly jeopardizes the main initiative we had to actually use XMPP !
Any help or guidance would be greatly appreciated.
I also had the same problem. I used to send ACKs for gcm's NACKs and when I stopped doing so, everything went perfectly fine. Check to see if you are sending unnecessary ACKs.
Related
I'm doing a little test where I send out a short string(4-8 bytes) to a client every 0.5 seconds from a Node.js server using ws. The client is either using iOS/Android or a web browser. The client does not send anything back to the server, except for TCP-ACKS I suppose. The weird thing is, when I'm debugging the app in iOS using XCode network report, I can only see that the client sends out some bytes(approx 500) when the connections establishes, probably during the HTTP handshake. The remaining time ZERO data is going out from the device, there is only data coming in. The same results is achieved when receiving data in Chrome and tracking the data using Nettop.
The thing that makes so confused is that on the Android, almost the same amount of data that goes in to the device goes out when inspecting the network usage with Android profiler/Battery Historian/TrafficStats. I have tried using different libraries for the Websocket implementation and using different Android devices.
I have a hard time believing the ACKS sent out by the android is as big as the message received, even though it's just a small string of four characters.
So my questions are:
Could the case be that Nettop/XCode network report is simply ignoring all the ACKS, so in reality as much data is sent out in Chrome/iOS as in Android?
Is there something 'Wrong' with the libraries used in Android or could it be something with its operating system?
Could an ACK be as big as a simple TCP-package with 4 characters in it?
The result below when using Websocket
The data received/transmitted when using Android Battery Historian
The data received/transmitted on iOS using Network Report
Could an ACK be as big as a simple TCP-package with 4 characters in it?
An ACK consists of the IP and the TCP header and no payload. With IPv4 this means at least 20 bytes IP header and 20 bytes TCP header, i.e. 40 bytes. A packet with 4 bytes payload is only larger by 4 bytes, i.e. 44 bytes or just 10%.
The network report in Android shows 68350 in vs. 61370 in bytes, which is a difference of 11%. This matches the expected difference.
I'm not familiar with what iOS measure here, but it probably either measures only the application payload (i.e. the 4 bytes) or simply ignores packets with no payload, i.e. the ACK's.
Hi anyone has some experience on scaling GCM XMPP ?
https://developer.android.com/google/gcm/ccs.html
Im reading docs there but Im not sure about this 100 pending msgs on 1 connection. I read somewhere that there is limit to 10 connections on server, is it right ? What f I will run 5 servers, each will open 10 connections, should it work well ?
Regarding 100 pending messages:
So apps can use "messages with payload" " to deliver messages of up to 4 Kb. This would be useful in a chat application, for example. To use this feature, simply omit the collapse_key parameter and messages will not be collapsed. GCM will store up to 100 messages. If you exceed that number, all messages will be discarded but you will receive a special message. If an application receives this message, it needs to sync with the server.
Regarding Server connection limit:
You can allow your server to send up to 4000 messages per second on the persistent connection. Knowing you are allowed up to 10 connections, you can possibly send many notifications fast (up to 40k notifications per second).
So you can speed up the message delivery on a device without eventually increasing the number of connections or number of servers but rather splitting your array of devices.
For faster delivery try these methods: 1. delay_while_idle - set to false 2. time_to_live - set to zero (but we have set to 30 for just in case) 3. Canonical IDs - Make sure Canonical IDs returned by GCM replace the old PushID in database 4. collapse_key - The most important factor - set it to random or TOD to avoid Google to throttle notifications.
In extreme case you can always airpush.
I have an application which depends on sending instant messages. I use google CSS and XMPP to achieve a persistent connection to the GCM server which works well.
But sometimes there is a delay while receiving the messages (10-15 seconds) on the client side (when on 3G), and I need to find a fix. I was reading that TCP connection timeout can occur while mobile operators often kill connection sockets, so I was thinking if there is a way to have a persistent connection with the GCM server on the client side as well?? Or could pinging the server be helpful?
I would be really grateful for any suggestion.
In order to be a battery efficient service , GCM in order to prevent TCP timeout sends a heartbeat every 18 mins on a 3G mobile device and 28 mins on Wifi . That is the reason why messages are delayed . Well one way to fix it is to send that heartbeat every X minutes . Just put the following code in a timer inside a service .
new Timer().scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
Log.e("Trying to save", "GCM or FCM");
getApplicationContext().sendBroadcast(new Intent("com.google.android.intent.action.c"));
getApplicationContext().sendBroadcast(new Intent("com.google.android.intent.action.MCS_HEARTBEAT"));
}
}, 0, 138000{Set this time interval as per your needs;Preferably 5 mins});
CAUTION: Reducing the time interval may result in draining of device battery so
chose that wisely.
This will keep on reviving your connection to GCM servers and you will recieve notifications on time. Good Luck , let me know if it worked
I have an .Net windows service that sends GCM push notifications via https://android.googleapis.com/gcm/send.
The service periodically sends batches of approximately 10,000 messages at a rate of 10 / second. With each batch, 5 - 10 of the messages will cause a "502 Bad Gateway" response. The messages that generate the 502 always go through with a second try.
Is this normal? Could I be sending too many messages too quickly?
According to the GCM guide, all 5xx error codes should be treated as temporary errors, and requests that get these errors should be retried :
5xx Errors in the 500-599 range (such as 500 or 503) indicate that there was an internal error in the GCM server while trying to process the request, or that the server is temporarily unavailable (for example, because of timeouts). Sender must retry later, honoring any Retry-After header included in the response. Application servers must implement exponential back-off.
I don't think 10 messages a second is too quick. As long as your code handles this error and retries, I don't think there should be any problem.
Google Cloud Messaging (Push message) problem.
I have two smartphones(HTC Sensation XE, Desire HD) and a tablet(Samsung Galaxy Note 10.1 Wifi).
Sometimes I need to wait for almost 15 minutes to receive the GCM push messages. Sometimes I receive the GCM push messages immediately(within 30 secs).
Is it normal? Or just because I'm under Wifi connected?
If it is normal, are there any other services like "long-polling" I can use for retrieving messages from server?
I'm currently using GCM+Polling(AlarmManager) to retrieve messages from remote server.
Any suggestions or better ideas?
Below is my test result:
The left side is server send time, while the right side is client mobile receive time.
All the test results are under Wifi connected environment.
==========================
HD
23:10:18, 23:24:XX
XE
23:11:21, 23:22:44
Note
23:10:20, 23:14:54
==========================
HD
00:08:12, 00:08:27
XE
00:07:55, 00:07:58
Note
00:08:04, 00:13:35
==========================
HD
00:40:21, 00:55:22
XE
00:39:56, 00:40:14
Note
00:40:13, 00:40:59
Thanks!!
GCM uses throttling when there's excessive use, see the following document for more details:
http://developer.android.com/google/gcm/adv.html#throttling
Please read my answer here:
Google Cloud Messaging - messages either received instantly or with long delay
It's basically a tcp timeout problem
You can manually trigger the heartbeat to keep the connection alive by the following code. Some network routers will automatically kill the idle socket connections if it is idle for a particular time period (sometimes 5 minutes). Execute the code in every 5 minute so that the connection will be re-established if it is closed. But, obviously reducing the interval will consume bit more battery (I don't know how much it is relevant here because nowadays the smart phones come up with high mAh battery).
getApplicationContext().sendBroadcast(new Intent("com.google.android.intent.action.GTALK_HEARTBEAT"));
getApplicationContext().sendBroadcast(new Intent("com.google.android.intent.action.MCS_HEARTBEAT"));