I am using firebase for a school dashboard application. The application is built using android. It logs in via google credentials to firebase. Disk persistence is enabled on firebase. There are no other services or connections that the app uses. Firebase version set in build is 'com.firebase:firebase-client-android:2.4.0+'
I opened the app - it logs in and gets the data - then put it in background. The data usage looks like the following -
Foreground data usage stays steady at - 38.87kb
Background data usage keeps increasing every few minutes - 2.53kb, 2.94kb, 11.49kb, 11.9kb, 12.18kb, 12.59kb, 13.90kb, 67kb, 92kb. It is consuming about 1.5kb of data every minute.
I've made sure all firebase listeners are disconnected on pause. Why is the application background data usage continuously increasing over time?
The Firebase client keeps a network connection open to its servers. As part of keeping this connection, the client and server occasionally exchange keep-alive messages. These will likely use some memory, which should be (largely) reclaimed upon a garbage collection.
You might want to upgrade to the latest version of the Firebase SDK for Android btw. It's now on version 2.5.2.
I just found 2 methods on Firebase class - goOffline() and goOnline() that might help me reduce the background data usage. I am going to try them in onPause, and onResume for all the activities in the app.
What I've done to reduce data usage is create a service that mantain a reference counted state of when firebase is needed to be online and use goOnline()/goOffline() according that state:
class OnlineTracker {
void onlineNeeded(); //Increase reference count
void releaseOnlineNeeded(); //Decrease reference count
}
And when you need to do operations that requiere Firebase to be online you do:
try {
OnlineTracker.onlineNeeded();
...
Firebase operations that need online status
...
} finally {
OnlineTracker.releaseOnline()
}
That way when OnlineTracker has reference count>0 it calls goOnline() and when it returns to 0 it calls goOffline();
To prevent too much switching you can also delay when you goOffline and wait some time to see if online needed status will be needed in very small time.
See this gist for complete real code
Related
I've made Android app that uses offline-mode. Also it has button "Sync", on click - syncronization with server is performed (server is not Firebase-service DB).
I want to do the same sync without this button once a minute when my app is on foreground and when network connection is on.
WorkManager seems the best solution for the usecase, but according to this article minimum interval for WorkManager is 15 minutes.
Other ways described in this article are: ForegroundService, AlarmManager and background Thread.
Also I found solution Sync Adapter
What way would be the most efficient for the case?
P.S. I understand that this scenario is not very clean and probably the best would be add online-mode and switch between two modes without frequent syncs. But I have some reasons at this time not to do that
Finally, I chose next way:
To invoke one-minute sync I used ThreadPool. This discussion helped me to choose.
Inside my Application class I put that code:
val scheduler = Executor.newSingleThreadScheduledExecutor()
scheduler.scheduleAtFixedRate({
.... <My Sync Block> ....
}, 0, 1, TimeUnit.MINUTES)
To prevent getting the same data from the server (there could be huge pieces of data in my case), I had to use MD-5 alghoritm on the server-side. It works as follows:
server emits data with hashes (for each piece od data)
mobile app gets data and saved both data and hashes in SQLite. In next sync app sends hashes back in request
server looks if requested data has different hashes and includes only updated data in respond
I'm considering the use of keepSynced() for some data from Firebase Realtime Database. I understand that it will automatically sync those paths. But how does that relate to Android lifecycle? If the user leaves all activities (and all normal listeners disconnect), will it stop syncing? I don't want the app to become data or battery hog.
On the other hand, I would like to update cached data when FCM notification arrives. I can launch some service which will connect to Firebase. I would like to sync all paths which are in keepSynced() and stop it when it's synced. I'm not sure how to achieve that. Create a listener to one of the paths and keep the service running for some time? After the service is finished, will it stop syncing?
firebaser here
Great question!
When there is no active activity, the operating system may close the connection to the Firebase database at any time. Our SDKs don't try to prevent that, but will reconnect when the app becomes active again.
What you're describing in your second paragraph is what we call "push to sync", where you send a push notification (typically a silent FCM data message) to trigger synchronizing of the data.
We did something like that in last year's I/O app and, while it was a bit more complex than we wanted it to be, it worked great. We explicitly managed the connection in that case, calling goOnline() and goOffline() (after 5 minutes iirc). The main sync code can be found in the IOSched github repo.
I have a app working offline. It is assumed that 1000+ records are created with images in each record during this period and whenever connectivity is established. What should be the approach to send all the 1000+ records to server that also handles any interruption between the network calls or API failure response.
I assume I have to send records in batches but how to handle the interruption and maintain consistency and prevent any kind of data loss.
I guess the best way here is to send each record separetely (if they are not related to each other).
If you have media attachments, sending of each record will take 2 seconds in average, if you uploading via mobile internet with speed ~2 MB/s. If you will send the large batch of records via each request, you must have stable connection for a long period.
You can send each record as multipart request, where parts are record's body and media attachments.
Also you have no need to check for internet connection, or use receiver for catching changes of connection state. You can simply use this libraries for triggering sync requests:
JobScheduler
Firebase JobDispatcher
Evernote android-job
I would suggest to use Firebase database API.
It has got nice offline/online/sync implementations.
https://firebase.google.com/docs/database/
And it is possible to read/write the data using Admin SDK for your NodeJS server:
https://firebase.google.com/docs/admin/setup
You can use divide and conquer approach means divide the task into small task and upload the data to the server.
1. take a boolean flag "isFinishData" starting with false.
2. starting upload the data on server from 0 to 100 records.
3. next record send from 100 to 200.
4. this process run until last record (1000) is not send .
5. in last record update set boolean variable true and exit from loop .
this logic would be work fine in IOS/android both.
Save your records in local Db and use ORMs for it. Use Retrofit which provide onSuccess and onFailure method for Webservice calling. To send data to server at regular interval you can use sync adapter.
1st I need to know how did you save image in local db ?
You need to create a service to catch connection status. Each time when connection is established, you submit your record as Multipart kind. You can you Retrofit/Asynctask.
Just submit 1 record per one Retrofit/Asynctask, it makes you ez to handle success/fail of each record.
You can run a single or multi retrofit/asynctask to submit one or more record, it's up to you.
If ur data has image, on server side, you have to handle process from ur server to 3rd server ( server to save image ).
This is a very broad question and it relates to Architecture, UI Experience, limitations, etc.
It seems to be a synchronization pattern where the user can interact with the data locally and offline but at some point, you'd need to synchronize the local data with server-side and vice-versa.
I believe the best place to start is with a background service (Android, not sure if there's a similar approach on iOS). Essentially, regardless of whether the Android app is running or not, the service must handle all the synchronization, interruption, and failure in the background.
If it's a local db, then you'd need to manage opening and closing the database appropriately and I'd suggest using a field to mark any sync'd records so if some records did fail, you can retry them at another point.
Also, you can convert the records to json array, then do a post request.
As for uploading images, definitely needs to be in batch if there's a lot of them but also making sure to keep track of which ones are uploaded and which ones aren't.
The one problem that you will run into if you're supporting synchronization from different devices and platforms, is you'll have conflicting data being synchronized against the backend. You'll need to handle this case otherwise, it could be very messy and most likely cause a lot of weird issues.
Hope this helps on a high level :)
To take on simple approach ,have 1 flag in your data objects [NSManagedObject] classes as sync.While creating new object / modifying an existing object change sync flag to false .
Filter data objects with sync value as false.
let unsyncedFilter = NSPredicate(format: "sync = %#", #(false))
Now you will have an array of objects which you want to sync with server.If you are sending objects one by one in requests.
On success change sync flag to true else whenever your function gets executed again on app launch/reachability status update, it will filter out unsynced data again & start synch.
As others have mentioned this is a rather broad question. A lot depends on both the architecture of the server that will receive the data as well as the architecture of the app.
If you have any control over the implementation of your backend I would recommend implementing a storage solution that allows for pausing and resuming of transfers. Both Google Cloud Storage and Amazon S3 offer a similar functionality.
The idea behind this approach is to be able to pick up the upload from where it stopped. In case of app crash or issues with internet connection you don't have to restart all from the beginning.
In your case I would still start separate uploads for each one of the records and store their upload progress.
Here you can find an example of how to use the pause / resume approach using the mobile SDK with Amazon https://aws.amazon.com/blogs/mobile/pause-and-resume-amazon-s3-transfers-using-the-aws-mobile-sdk-for-android/.
Editing adding reference to Amazon iOS SDK , http://docs.aws.amazon.com/mobile/sdkforios/developerguide/s3transfermanager.html
Best way is to break the files into chunks of 100s and upload at intervals or when app is idle.
I'm considering the use of keepSynced() for some data from Firebase Realtime Database. I understand that it will automatically sync those paths. But how does that relate to Android lifecycle? If the user leaves all activities (and all normal listeners disconnect), will it stop syncing? I don't want the app to become data or battery hog.
On the other hand, I would like to update cached data when FCM notification arrives. I can launch some service which will connect to Firebase. I would like to sync all paths which are in keepSynced() and stop it when it's synced. I'm not sure how to achieve that. Create a listener to one of the paths and keep the service running for some time? After the service is finished, will it stop syncing?
firebaser here
Great question!
When there is no active activity, the operating system may close the connection to the Firebase database at any time. Our SDKs don't try to prevent that, but will reconnect when the app becomes active again.
What you're describing in your second paragraph is what we call "push to sync", where you send a push notification (typically a silent FCM data message) to trigger synchronizing of the data.
We did something like that in last year's I/O app and, while it was a bit more complex than we wanted it to be, it worked great. We explicitly managed the connection in that case, calling goOnline() and goOffline() (after 5 minutes iirc). The main sync code can be found in the IOSched github repo.
My Firebase database has setPersistenceEnabled=true. When offline all the listeners work fine and fetch data from the local cache.
But, when online, Firebase is making a network call each time I use a addListenerForSingleValueEvent on the same node, even though I have a local cache and nothing has changed on the server.
I have two items at the node I'm attaching the listener to, and I see the following reported after setting log level to debug. This network activity is repeated every time I use the listener while online, even within sub second delays. My understanding is that if Firebase has local data, then server calls won't be made. Any sync checks might happen in the background and with efficient network usage.
conn_18 - received data message: {r=20, b={s=ok, d={}}}
conn_18 - received data message: {r=21, b={s=ok, d=}}
I am trying to aggressively reduce network usage on my app, and any suggestion to stop/reduce Firebase network activity would be very helpful.
Been digging into this myself.
As far as I can tell, the second call you are experiencing is simply the network saying "Hey, there's no updates", and you're not charged for any reads.
As far as reducing the network activity, I think this is just how it works, and how you want it to work (its the actual listening part of the listener). If you don't want that, use a getDocument call, rather than a listener.
This is based on a few things:
When you look at snapshot.metadata.isFromCache (iOS SDK) for a query, the call will come twice. first from cache and then from the server. However, if you print a statement under snapshot.documentChanges, ONLY the cached call will print.
I did an experiment re-loading a view in iOS ~20 times. I also made on small change to my data. I then waited 5-10 minutes and looked at the usage on: https://console.cloud.google.com/firestore/usage. My total read count went up by 3, which means I was only charged for the update. Reloading the view a bunch of times was NOT charged.