I am currently storing my transactions in SqLite DB if there is no internet connection at the moment and when there is internet available i send the pending transactions with the new one made so making it quite a lot of data and a lot of API hits which chokes the device and it becomes unresponsive. So need help with a proper way to sync these transactions to the server. Also these are being sended to a Socket as well as a Server.
I tried using AsyncTask for it but it also causes problems if transactions are above 200. Tried Retrofit for it and to some extent the count exceeded from 200 to almost 350 but the issue and unresponsivness remains.
You should give a try to PriorityJobScheduler lib or WorkManager from JetPack.
When there is no network connection you can queue those request and those request will be send ones network connection is available. (So you dont need to wait until someone made new transaction to send old queued data too)
Also, in your current scenario, rather than sending single request for each transaction, ask your API Guy to accept request in List of object format. So you just need to create list of request body object and send to server
I have a app working offline. It is assumed that 1000+ records are created with images in each record during this period and whenever connectivity is established. What should be the approach to send all the 1000+ records to server that also handles any interruption between the network calls or API failure response.
I assume I have to send records in batches but how to handle the interruption and maintain consistency and prevent any kind of data loss.
I guess the best way here is to send each record separetely (if they are not related to each other).
If you have media attachments, sending of each record will take 2 seconds in average, if you uploading via mobile internet with speed ~2 MB/s. If you will send the large batch of records via each request, you must have stable connection for a long period.
You can send each record as multipart request, where parts are record's body and media attachments.
Also you have no need to check for internet connection, or use receiver for catching changes of connection state. You can simply use this libraries for triggering sync requests:
JobScheduler
Firebase JobDispatcher
Evernote android-job
I would suggest to use Firebase database API.
It has got nice offline/online/sync implementations.
https://firebase.google.com/docs/database/
And it is possible to read/write the data using Admin SDK for your NodeJS server:
https://firebase.google.com/docs/admin/setup
You can use divide and conquer approach means divide the task into small task and upload the data to the server.
1. take a boolean flag "isFinishData" starting with false.
2. starting upload the data on server from 0 to 100 records.
3. next record send from 100 to 200.
4. this process run until last record (1000) is not send .
5. in last record update set boolean variable true and exit from loop .
this logic would be work fine in IOS/android both.
Save your records in local Db and use ORMs for it. Use Retrofit which provide onSuccess and onFailure method for Webservice calling. To send data to server at regular interval you can use sync adapter.
1st I need to know how did you save image in local db ?
You need to create a service to catch connection status. Each time when connection is established, you submit your record as Multipart kind. You can you Retrofit/Asynctask.
Just submit 1 record per one Retrofit/Asynctask, it makes you ez to handle success/fail of each record.
You can run a single or multi retrofit/asynctask to submit one or more record, it's up to you.
If ur data has image, on server side, you have to handle process from ur server to 3rd server ( server to save image ).
This is a very broad question and it relates to Architecture, UI Experience, limitations, etc.
It seems to be a synchronization pattern where the user can interact with the data locally and offline but at some point, you'd need to synchronize the local data with server-side and vice-versa.
I believe the best place to start is with a background service (Android, not sure if there's a similar approach on iOS). Essentially, regardless of whether the Android app is running or not, the service must handle all the synchronization, interruption, and failure in the background.
If it's a local db, then you'd need to manage opening and closing the database appropriately and I'd suggest using a field to mark any sync'd records so if some records did fail, you can retry them at another point.
Also, you can convert the records to json array, then do a post request.
As for uploading images, definitely needs to be in batch if there's a lot of them but also making sure to keep track of which ones are uploaded and which ones aren't.
The one problem that you will run into if you're supporting synchronization from different devices and platforms, is you'll have conflicting data being synchronized against the backend. You'll need to handle this case otherwise, it could be very messy and most likely cause a lot of weird issues.
Hope this helps on a high level :)
To take on simple approach ,have 1 flag in your data objects [NSManagedObject] classes as sync.While creating new object / modifying an existing object change sync flag to false .
Filter data objects with sync value as false.
let unsyncedFilter = NSPredicate(format: "sync = %#", #(false))
Now you will have an array of objects which you want to sync with server.If you are sending objects one by one in requests.
On success change sync flag to true else whenever your function gets executed again on app launch/reachability status update, it will filter out unsynced data again & start synch.
As others have mentioned this is a rather broad question. A lot depends on both the architecture of the server that will receive the data as well as the architecture of the app.
If you have any control over the implementation of your backend I would recommend implementing a storage solution that allows for pausing and resuming of transfers. Both Google Cloud Storage and Amazon S3 offer a similar functionality.
The idea behind this approach is to be able to pick up the upload from where it stopped. In case of app crash or issues with internet connection you don't have to restart all from the beginning.
In your case I would still start separate uploads for each one of the records and store their upload progress.
Here you can find an example of how to use the pause / resume approach using the mobile SDK with Amazon https://aws.amazon.com/blogs/mobile/pause-and-resume-amazon-s3-transfers-using-the-aws-mobile-sdk-for-android/.
Editing adding reference to Amazon iOS SDK , http://docs.aws.amazon.com/mobile/sdkforios/developerguide/s3transfermanager.html
Best way is to break the files into chunks of 100s and upload at intervals or when app is idle.
Scenario :
1. I have a list view, fetching data on the fly. Due to scrolling what data it might get could be stale.
2. When I implemented HTTP client using a thread , on scroll i would cancel all requests threadHandler.removeMessages(intwhat) so that no stale data was returned.
3. The autobahn Websocket API gives me option to sendmessage and receive its response asynchronously.
4. Can anyone advice me on how to go about cancelling the requests from the WebsocketWriter ? or a work around.
5. If the above description is not sufficient , I can elaborate more.
The OPs question has been taken to the Autobahn mailing list here.
Extract:
sendMessage() cannot be canceled. Upon invoking, it'll forward the
message to the background writer thread, and that thread will send out
the bytes on wire.
What you can do (at app level) is ignore any responses to the sent
message the application expects.
My architecture will use ActiveMQ on the server and have Android clients send and receive messages.The network situation will be very unreliable; possibly hours of missing connection. Is there a framework that will allow me to queue up the messages on the android client and deliver them reliably once the connection is back?
You can efficiently implement one yourself, I don't think anyone will provide you this service, and if they do they will certainly charge, Here is what I can suggest for an optimal solution.
Design a db using SQLITE to hold you message, once a message is ready for deliver from android client, you can perform the following
a. If network is avaibale, then you can directly deliver message to your web clinet
b. If network in not present, then cache it directly to you local android db
Design a Sync logic, you can achieve it by network listener, so when user device comes back into network,
you can write a logic to query from databse and posting to your webclient, deleting local data subsequently
upon successful posting into server
You can strengthen you logic, by caching message everytime into local db first, then a Sync logic which will commit your local changes to web server in bulk, thus improving upon processing time.
Hope this answer your problem.
I'm currently working on an Android project that will need connecting to a Bluetooth device that will dispatch messages to different nodes. This means that I will have to pass the right messages to the appropriate nodes (many micro-controllers).
At the moment, I can send a string or receive a string from the master micro-controller and I think the best way to solve my problem will be that the master micro-controller node simply repeats and broadcasts the message to all the others nodes. For the Android part, I was wondering if it was a good practice to make an array that will contain the id of the receiver and after the data I want to send. The ID will be on 8 bits and the data will be a string. After I will cast the int to a string and concatenate both strings to send my id+data.
Is this a good way to solve my problem or there is a more elegant way to do so?
It would be more efficient to cast the string to bytes and send it all as an array of bytes. Serious network protocols would never use text data like that. If you're just doing a for-fun trial that's ok though.
Here's the real problem I see with your mesh- infinite exponential propagation. Lets say I send a message to someone, and do it by sending it to all of my neighbors. They'll forward it to all their neighbors. Who'll forward it to all their neighbors. Which if there's ever any loop in the graph will cause it to get sent back to someone who's already seen it, who will forward it again. And it will never die. Unless you have no loops, in which case you don't have a mesh and you're very fragile and will likely fragment. You need some way of preventing retransit of the same message- possibly as simple as a message id field and not retransmiting the same message id again. You'd need a large pool of message numbers for that though- something like a 128 bit UUID.