I have a app working offline. It is assumed that 1000+ records are created with images in each record during this period and whenever connectivity is established. What should be the approach to send all the 1000+ records to server that also handles any interruption between the network calls or API failure response.
I assume I have to send records in batches but how to handle the interruption and maintain consistency and prevent any kind of data loss.
I guess the best way here is to send each record separetely (if they are not related to each other).
If you have media attachments, sending of each record will take 2 seconds in average, if you uploading via mobile internet with speed ~2 MB/s. If you will send the large batch of records via each request, you must have stable connection for a long period.
You can send each record as multipart request, where parts are record's body and media attachments.
Also you have no need to check for internet connection, or use receiver for catching changes of connection state. You can simply use this libraries for triggering sync requests:
JobScheduler
Firebase JobDispatcher
Evernote android-job
I would suggest to use Firebase database API.
It has got nice offline/online/sync implementations.
https://firebase.google.com/docs/database/
And it is possible to read/write the data using Admin SDK for your NodeJS server:
https://firebase.google.com/docs/admin/setup
You can use divide and conquer approach means divide the task into small task and upload the data to the server.
1. take a boolean flag "isFinishData" starting with false.
2. starting upload the data on server from 0 to 100 records.
3. next record send from 100 to 200.
4. this process run until last record (1000) is not send .
5. in last record update set boolean variable true and exit from loop .
this logic would be work fine in IOS/android both.
Save your records in local Db and use ORMs for it. Use Retrofit which provide onSuccess and onFailure method for Webservice calling. To send data to server at regular interval you can use sync adapter.
1st I need to know how did you save image in local db ?
You need to create a service to catch connection status. Each time when connection is established, you submit your record as Multipart kind. You can you Retrofit/Asynctask.
Just submit 1 record per one Retrofit/Asynctask, it makes you ez to handle success/fail of each record.
You can run a single or multi retrofit/asynctask to submit one or more record, it's up to you.
If ur data has image, on server side, you have to handle process from ur server to 3rd server ( server to save image ).
This is a very broad question and it relates to Architecture, UI Experience, limitations, etc.
It seems to be a synchronization pattern where the user can interact with the data locally and offline but at some point, you'd need to synchronize the local data with server-side and vice-versa.
I believe the best place to start is with a background service (Android, not sure if there's a similar approach on iOS). Essentially, regardless of whether the Android app is running or not, the service must handle all the synchronization, interruption, and failure in the background.
If it's a local db, then you'd need to manage opening and closing the database appropriately and I'd suggest using a field to mark any sync'd records so if some records did fail, you can retry them at another point.
Also, you can convert the records to json array, then do a post request.
As for uploading images, definitely needs to be in batch if there's a lot of them but also making sure to keep track of which ones are uploaded and which ones aren't.
The one problem that you will run into if you're supporting synchronization from different devices and platforms, is you'll have conflicting data being synchronized against the backend. You'll need to handle this case otherwise, it could be very messy and most likely cause a lot of weird issues.
Hope this helps on a high level :)
To take on simple approach ,have 1 flag in your data objects [NSManagedObject] classes as sync.While creating new object / modifying an existing object change sync flag to false .
Filter data objects with sync value as false.
let unsyncedFilter = NSPredicate(format: "sync = %#", #(false))
Now you will have an array of objects which you want to sync with server.If you are sending objects one by one in requests.
On success change sync flag to true else whenever your function gets executed again on app launch/reachability status update, it will filter out unsynced data again & start synch.
As others have mentioned this is a rather broad question. A lot depends on both the architecture of the server that will receive the data as well as the architecture of the app.
If you have any control over the implementation of your backend I would recommend implementing a storage solution that allows for pausing and resuming of transfers. Both Google Cloud Storage and Amazon S3 offer a similar functionality.
The idea behind this approach is to be able to pick up the upload from where it stopped. In case of app crash or issues with internet connection you don't have to restart all from the beginning.
In your case I would still start separate uploads for each one of the records and store their upload progress.
Here you can find an example of how to use the pause / resume approach using the mobile SDK with Amazon https://aws.amazon.com/blogs/mobile/pause-and-resume-amazon-s3-transfers-using-the-aws-mobile-sdk-for-android/.
Editing adding reference to Amazon iOS SDK , http://docs.aws.amazon.com/mobile/sdkforios/developerguide/s3transfermanager.html
Best way is to break the files into chunks of 100s and upload at intervals or when app is idle.
Is there anyway that I can read the data from the real time database of firebase and can save it while the app is not opened, I am using Firebase.getInstance.goOnline();
in the code whenever Network connectivity changes so that it listens the data.
I am able to write the data in the database if app is not in recents or is not opened but in the same way I am unable to read the data when app is not opened. Is there any possible way ? Or I am going for an wrong approach ?
I just wants that whenever network connection will be available it will download the updated data automatically. , so that later on I can see the updated data being offline.
Any help will be highly appreciated. Please.
Android makes no guarantees about whether or not your app process is running when it's not visible on screen. It could be killed in favor of other processes that are a higher priority for the user. So, in general it's not a good idea to assume that you can do networking any time you want.
Also, newer Android devices will go into doze mode to help save the user's battery. In that case, your app will definitely not be running.
If you want to ensure that your app process is running, you can schedule some time with JobScheduler or AlarmManager. Or you can ping your app from your server with Firebase Cloud Messaging.
The Situation
I've been banging my head on this one for awhile. I have a local DB using Realm with a back end SQL Server DB. There are a few areas in my app where data can be changed very quickly (a checkbox for instance). The user has the ability to toggle data back and forth many times a second. So lets say the user in the matter of a second makes 10 changes. What I am trying to avoid is POSTing out to SQL Server 10 times in that second. Instead what I'd like to happen is have some sort of batch process that executes every 5~10 seconds to sync up the data thus reducing network calls.
The Caveat
How can I ensure the sync process will fire even if the user kills the app before the 5~10 second mark or while the POST call is being executed.
Call a service in the onPause() of the current activity to run after the fact? Is this
guaranteed?
Cache the data and try again on the next app launch?
Use GCM or some form of push service to initiate a scheduled sync?
I understand there is not a silver bullet for this but if anyone has ran into a similar concern any suggestions/best practices are appreciated, thanks in advance!
try to used onchangesListener on realm. to sync your updated or new data to sql server..
I'm currently developing an mobile application and rest service. The mobile application executes lots of calls to the service even if no update is required and data didn't changed. In order to remove this overhead of rest calls I'm planning to implement GCM (Google Cloud Messaging).
My strategy would be the following:
Load all required data on application startup. When data change was recognized on server side a push notification will be sent via GCM to affected devices to make partial refreshes of data (via specific rest calls). Advantages of this would be less overhead at service side, because there are no unnecessary rest calls and a more fluid user experience in my opinion. Disadvantage is that the app is dependent on GCM Messages and that they arrive in time.
I'm unsure if this is the right strategy for this. Could someone maybe point me in the right direction and tell me if this is a good practice?
We could use more information before answering details...
I'm unsure if this is the right strategy for this. Could someone maybe point me in the right direction and tell me if this is a good practise?
I will consider for the sake of giving an overall answer that:
A - User is not always with the application "online", neither has network, not even a desire to have updated info at all times.
B - User is eletronicaly litterate enough to understand difficulties with the program.
With those in mind, then what would be a good approach is:
Poll relevant data, store them locally. At this stage, one would consider the relevant informations that user would have and store them, with a date flag.
Once a flag goes "old" (below your threshold), re-query that data.
Operations follow 2 directives... When observing a data, check its state, show the user if its recent or not, and if its not, poll it. If it is, if the user selects operations on it (POST mostly), re query the data.
This way, you have no static overhead, if users dont have the app on foreground. Also, should they use your "always online app", they understand that network is a necessity.
I'm getting hung up on how to handle the data for an app I'm designing. I want to pull a list of items from the net. The data is updated routinely, and I think it would be good to store all the data on the device so the app can load quickly and refresh the data in a background thread rather than have to wait for the network on every start-up.
I think I should make the data available in an XML and have a thread parse and save into a SQLite DB, but I'm not sure if that's the "best practice." Are there other ways that people go about handling this?
The cleanest way (or at least, I think it is the cleanest way) is to implement a custom ContentProvider class that interfaces with your server. You can query the contenprovider and if it doesn't have the data in a local cache (for example a SQLite db as you said) it downloads it from your server and adds it to the local data. Why a content provider? because then you can easily access your data across apps and you have a nice and clean way to get your data when using intents.
Also, I personally prefer not to download data when the app is not running, because it will cost battery life while the user does not actively requests the data.
Sounds reasonable.
If the download of the new data takes more than some seconds, you might want to use a Service. This will allow you to continue the update even when the user has left your app.
Also, think about how you will notify the user that something is going on. Displaying some progress indicator is always a good idea. Otherwise, users might just think the data is not up to date because the app is broken.