I am using Firebase (FCM) in a project, where I use the topic feature to send out daily notifications to many different users in various timezones. Notifications are always sent out, when the user's time is 12:00PM, in order to achieve this, I create a unique topic every day, for the batch of users I am going to notify. At the moment our topics could look like: x_daily_notification_1550152751 - Previously I had a system that would create static topics based on the timezone, so users could be subscribed to a topic like: x_daily_notification_europe_london, but this proved to be too unreliable and hard to manage, due to users moving between timezones.
The way the system works is based on research and advise from this question I made late last year: FCM topic limits and expiration/invalidation of old unused topics?
So to sum it up:
Bulk subscribe users to a topic
5 minutes later, send out the topic. The delay is due to not knowing if FCM waits for all the bulk subscriptions to process, before sending out the notifications
10 mins after sending the topic, unsubscribe all tokens from the topic again, deleting the topic from FCM.
The reason for these delays, is that I have noticed that sometimes we do not get the notifications. The only reason I can think of, is because we send and unsubscribe the topic too fast, so if 100K users were to be notified and we deleted it straight away, only a few would get it. It seems that FCM does not wait for one request to finish before another is processed.
Even though I wait 10 minutes before I unsubscribe, it seems that sometimes not all users get notified, which could mean that they were not processed before I unsubscribed them again, so I would like to wait longer before I unsubscribe everyone from the topic, to ensure that they have received the notification. The first solution I thought of was to just wait, until the next batch is ready to be processed and then delete the previous topic, which would be about an hour later, where I can be more or less sure that everyone has received the notification at that point.
My real question is, do I need to unsubscribe at all? It could get messy on Firebase' end, but surely they have systems in place to handle this, as it does not mention anything in the documentation about cleaning up.
Related
I Will describe my use case with figures attached and everything i can do to be clear so we may get up with a better idea ...
General idea:
Whatsapp chat application with firebase
Use case:
As we know one of the features of whatsapp is the last seen, when the user did (exited the app, log out, lost wifi connection, etc ..)
I tried to use:
onDisconnect, but onDisconnect gives bad result when losing wifi connection (because of the socket latency to be timed out)
the one I am using now, is every user updates its timestamp every 3 seconds (update document every 3 seconds), when the user loses connection,
he won't be able to update his timestamp, right? So, if another user wants to chat with this offline user, I can show him the user's last seen. Hope this
this is clear...
Developed using Flutter framework
Redux to manage app state
Firebase, cloud firestore
The code below is dispatching an action every three seconds, this dispatched action will update the last seen in firebase...
timer = Timer.periodic(Duration(seconds: 3), (Timer t) {
// store.dispatch(updateUserOnline());
});
As you can see in the figure below my data structure of how I am updating last seen for this user every 3 seconds ...
This implementation is very expensive to get satisfactory results for a last seen for a user, if we have million users and these million users are updating their
last seen every 3 seconds it will cost a lot $ per month, as we are doing a write operation, no?
So, my other solution is to implement a socket connection to my own server and let all the users listen to the onDisconnect socket event on my server instead of Firebase server, is this doable to avoid the huge cost of writing operations?
image attached: here
Firebase writes would indeed be a bit costlier since you would be sending in a lot of writes, which, apparently are just for the job of "last seen".
Instead, as you mentioned, having a socket connection with your own server will help reduce the number of queries you make. i.e. As soon as the socket disconnects from the server, you can send a write operation to Firebase. "Every 3 seconds vs Only when the user disconnects".
Plus (not something that you asked for), if you would be setting up a socket server of your own, then it shall help in the following scenarios as well:
Typing events (The indication we get when the other person is typing a message)
Quicker way to know if the person at the other end is online/offline (because of sockets)
Our app with more than a million subscribers is facing huge delivery issues with FCM. It has become worse lately and the service is hardly working anymore. We are receiving errors like:
{ code: 'messaging/message-rate-exceeded',
message: 'Topic quota exceeded.' },
codePrefix: 'messaging' }
We get this error a lot. And it seems to be worse during EU / US evenings. In some cases over 90% of the notifications are failing.
We are in contact with the firebase support team, but so far there seems to be no solution. The gave us lots of information with some useful facts though:
resources are shared between developers. So the max message rate can be different because of other developers taking up resources.
OR queries should be converted to multiple AND queries because OR queries actually generate messages to all of user base, and then the filtering condition is applied
240 messages/minute and 5,000 messages/hour to a single device.
limit upstream messages at 15,000/minute per project (we don't understand this one)
limit upstream messages per device at 1,000/minute
They also updated their docs at https://firebase.google.com/docs/cloud-messaging/concept-options#topics_throttling
So we are aware of message rate limits and fanouts mechanism. In our case we have an approximate of 6000 different topic send requests per hour and on average 10k subscribers per topic.
A single user will never get more than 50-100 notifications per hour.
We believe we are not hitting the limits set by FCM.
Back in the GCM time everything worked fine. So we are quite unhappy about the current situation. The core functionality of the app is really bad right now. And a solution seems to be not there.
We are considering switching to a SSE solution.
There is a story about someone who succesfully moved away from FCM
https://f-droid.org/en/2018/09/03/replacing-gcm-in-tutanota.html
But since Google has made it very difficult lately to have background processes running, I wonder what other people with similar experience did.
Or can we still fix this situation?
One such alternative is Cloud Alert - it can replace FCM, provides high throughput and unlimited messages. It uses a background job and maintains its own connection to its dedicated servers. While a free plan is present, your 1 million connection requirement would put you into the paid bracket.
Disclosure: I work for Cloud Alert.
I'm using Firestore event listeners https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentReference.html#addSnapshotListener(com.google.firebase.firestore.EventListener%3Ccom.google.firebase.firestore.DocumentSnapshot%3E) to show notifications in NotificationManager. But in Doze mode the notifications are delayed for about 10-15 minutes. Is my understanding correct that Firestore uses something like FCM normal priority messages for data synchronizations, causing the listeners to stall?
Per my requirements I need to show the notifications as soon as possible (the app is some kind of POS) and 10 minutes is too long.
I'd like to ask if following approach is feasible: I plan to use Firebase Cloud Functions triggered on changes in Firestore (Google example is for Datastore, is it true for Firestore too?) and send FCM high priority message to client, and then it will show notification. Is my assumption correct that Firestore listeners on client will be updated then, since the device woke up on the high priority message?
I plan to defer the FCM messages for about 30 seconds, to process the changes in batch. The listeners will send to Firestore DB the last time when they were activated, and the Firestore Cloud Function will check if the client's listeners were not called for some time (30 secs) and changes exist, then the FCM high priority message will be sent.
I'm worried if I'm not "inventing a wheel", as the use case seems to be quite common. Is there any better solution? For my approach I'll have to keep a table with entries per connected device, which may be large, in general. And most important that it goes against Firestore "sense" - all Google tutorials say how great it is just to register listeners and get instant updates (and it works smart I agree, at least for UI changes which obviously not important for doze mode, under the black screen) for status bar notifications (when user expects notification by sound etc) I'll have to use FCM push notifications anyway.
If go further, I can add relevant payload to the messages, to receive updates without listeners, and then what the fun is to use listeners at all? If it is correct then Firestore doesn't seem to be so sweet as it sells, at least for my case, when the Doze mode delays matter...
Please advice, may be somebody already dealed with such use case and Firestore.
Is my understanding correct that Firestore uses something like FCM normal priority messages for data synchronizations, causing the listeners to stall?
Firestore does not use FCM for data synchronization. It has its own wire protocol, separate from FCM.
Everybody tells me that polling server for new data is stupid if the server is yours and you should implement push with GCM instead. Well, I agree and have done so but, I was wondering, how often can or should you perform push?
I have a simple app where people post stuff to the server and have a feed of everyone else's posts. The method on the server that does the saving to the database also triggers the GCM Push at the end, so Google sends out the push notifications to everybody and everybody then requeries the server for new posts, containing the post weve just made.
But, what if I have, lets say milion users and a new post is created lets say every minute. This will mean that app will connect to server every minutes and will kill the battery. So my question is, how often you perform Push? Wouldnt in this case polling every 10 minutes be actually better for battery, right?
I know this example is kind of crazy, having milion people in your feed, but its just to make a point, because I cant imagine how this would scale. I figure Google will handle sending million gcm messages at once no problem.
So the question is, is there a limit after which Push is contraproductive? I guess its kind of a first world problem :D but cant imagine Facebook handles stuff like this.
Thanks!
Push notifications are welcome when the server has important new data to show to the user (and by important I mean important to the user).
I believe that if any app sent me a notification every minute I would either disable notifications from that app or uninstall that app. You should be very carefull when deciding what updates to send to your users as push notifications, in order not to antagonize them. If your server can send push notifications very often to the same device, you must have settings options in your app that can reduce the frequency of those notifications.
Push notifications are usually relevant when the app is not running (or is running in the background). When it is running in the foreground, polling the server is probably a better solution (given the disclaimer that you can't rely on the push notifications being delivered every time).
In your example, I'm not sure I'd want to be notified automatically about each new post in the DB. Even while I'm using the app (i.e. it's in the foreground) I wouldn't want it to update with new posts automatically. Even facebook don't update the view automatically (they show you on top of the screen that you have new posts, and you have to pull the list view down in order to load them. And when the app is not running, I'd want to be alerted only about important posts (for example, posts from specific users). When an app sends me too many push notifications, I go to its settings to reduce them.
I can't give you a numeric figure of a reasonable frequency of push notifications, but I hope my answer helps. In short, you don't have to worry about GCM's technical limits. You should worry about the user experience.
we've been trying to develop an android application which uses the c2dm service of Google.
When we start the application after clearing all data, the application receives the c2dm messages just fine, but after some time (maybe 2 minutes) the messages refuse to arrive.
We also checked the code we received after pushing the c2dm messages from the server, and the code was successful (code number 200 without error).
After searching relevant posts on Stack Overflow, we came across this post:
Why do Android C2DM push messages not always arrive?
but we verified that we don't register to the c2dm service each time the application starts.
What seems to be the problem in our case?
We use android 2.2 API 8 version .
Thanks in advance,
Mark.
You should always have in mind that Google's C2DM allows a certain limit of messages/day. I'm thinking that sending a large number of messages in 2-3 minutes (a client-chat, or something like that) could be the source of your problem.
And also, have in mind that there is no guarantee whatsoever that messages will arrive. Per Google's C2DM Introduction: C2DM makes no guarantees about delivery or the order of messages. But you probably already know this.
I am thinking that if your 2-3 minute average is a rule, then probably the limitation of the messages could be the cause. Try sending fewer messages and see if the interval doesn't get larger.
"maybe 2 minutes" - you should confirm that first of all. You must clarify:
Is this issue related to this one device?
Does it happen consistently? If not, what triggers it?
Has it happened once, or does it happen every time?
Do bear in mind that C2DM messages are not guaranteed. Some will not arrive.
Also be aware that sometimes Android devices "fall off" c2dm and don't receive messages for a period of time. You will see similar effects on some networks (e.g. in my experience some C2DM messages are not delivered over wifi networks, so try 3G too).