By accident, I had created an endless loop in my android studio app which has added over 100k+ records to my Realtime Database. After closing everything I wanted to delete this massive table using Firebase CLI but every time I try it i get an error:
> firebase database:remove /matches
? You are about to remove all data at https://XXX.firebaseio.com/matches. Are you sure? Yes
Error: Task index 0 failed: retries exhausted after 4 attempts
How am I supposed to delete it then?
The most likely cause is that other clients are updating the data while your code is trying to remove it. If that is the case, the only way to work around it, is to remove the data in smaller chunks, to reduce the contention during the operation.
To do this, you'll:
Read the data from /matches either through the API, or from an automatic backup.
Determine a reasonable subset of the nodes to remove in one go.
Make an API call to remove that subset of the data.
Repeat this process until all data is gone.
Related
I'm currently working on an app with React Native using Firebase as backend. I won't be too specific on the details, but, basically, the app has to constantly fetch for new data uploaded, so I added this part in my main code:
useEffect(() => {
db.ref('rooms').on('value', (data) => {
if (startRoom.id) {
ping = (data.val());
setRoom(() => (data.val())[startRoom.id]);
}
});
},[ping]);
This way, every time new data are added in the object "rooms" in my db, the app is able to update all variables linked to it. This process however can cause some huge slowdowns and performance issues: indeed, when there are a lot of data uploaded by different users, or when data are uploaded very quickly (let's say, uploads are handled by user clicks, so very quick clicks can cause the db to go nuts), the app is constantly refreshing the page even several times per second, causing the slowdowns I was talking about.
So my idea is: is there anyway I can limit the number of fetches the firebase ref method does over time? Like, if the database is uploaded seven times in a second, it only catches the last one, ignoring the uploads which, for example, happened in a very short time. In other words I wonder if there is a way to limit the "sensitivity" of the ref method or the useEffect method of React Native.
Thank you very much!
If updates are happening multiple times a second, I think you should change your approach.
I think you should instead of updating your app with the data the moment it changes, store the changes in to an array changes, and set one interval that run for every, says 3 seconds, check the changes array. If there is data in it, use it to update your app and then clear the array. This way you can make sure your app only update at a fixed rate even if there is a lot of changes
what's the data type of ping? If it's an object, your useEffect will run every time because of the way JavaScript checks for equality on objects (they check reference I believe).
A way around this is to use useDeepCompareEffect or lodash's isEqual. Or just use a primitive value as the dependency for your useEffect.
I have a requirement, if a record inserted in local DB(Sqlite) as soon as possible app triggers API call to send offline record to server. I don't want to use Timer and Alarm service because those will run continuously in background even there is no offline records in local DB. Any other alternative solutions?
Note: Target SDK level is Andorid 8 version.
As I understood your task (in general) - is how to invoke some function (some sendNewRecord() for instance) when new row(s) were inserted in some Sqlite table.
I suppose there are 3 alternative ways to implement such a task in Sqlite:
To invoke your function at the same place in code where you put your insert-new-data code. I think that's the most obvious and the simplest choice. I hope in your app there is only one place where you do it (Repository pattern or its analogue).
To put some observer(callback function) to your table and process changes in table. That more complicated task, and as far as I know you'll get this callback after every change in your table and it would be another task to identify what rows in your table were changed. Using modern Room framework and LiveData as a query result, you can observe changes in the table in more strait-forward way, but then nevertheless you have to detect inserted rows.
To use some scheduling mechanism for syncing (Executor, Timer and so on). You've wrote that it's not your case.
I would recommend you to choose first way. At least I don't see its disadvantages.
When my app is offline and I am adding or updating a document the memory increases. Also, showing lists of documents take longer to load. If I run the same code when the device is online the memory stays consistent as well as the speed of the activities that have lists of documents.
I'm currently doing saves like the following:
collectionRef.document(id).set(obj, SetOptions.merge());
Or for batching a couple of records:
batch.set(docRef1, obj1);
batch.set(docRef2, obj2);
batch.commit();
I had listeners for onComplete but in the accepted answer for this question, it seems to indicate that listeners are unnecessary in most situations and that you can't wait for it to complete anyway when you're offline.
In another question they indicate in code that a "Snapshot" is required to properly do online and offline saving: Offline issue with Firestore vs Firebase. But I can't find anywhere else indicating if that will make a difference. I think of Snapshots as being something you attach to a document or query when you want to be notified of changes to it and attaching a listener like that will result in a memory leak if it isn't removed.
Another part of all of this is how this slowness might affect data integrity. When I watch in the profiler in Android Studio I see that the FirestoreWorker can get to a point where it is constantly working even if I don't do anything in the app. I'm not just talking a few seconds, more like a minute. There isn't any ordering guarantee of writes when it is offline that I can find. Trying to stop and restart the app doesn't seem to have any effect on the slowness (although it will reset the memory).
So all of this leads to the question: what is the proper way to add/update data in Firestore when offline so that the app's memory doesn't grow unbounded and slow down?
Cloud Firestore uses SQLite for its persistence mechanism. So for intermittent periods of offline activity, you shouldn't have problems with performance or durability.
However, if you intend to use a Firestore database for very long periods of time, there are some things you should be aware of. Cloud Firestore was not built as an offline database, is an online database that continues to work when you're offline for short or longer periods of time. When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations.
So I suggest use this database for its online capabilities. As one of the Firebase engineers said and I quote, "It is impossible to build a slow query in Firestore". So, the performance comes from the new indexing capabilities on the backend and these optimizations don't exist when you're offline.
One more thing, if you have many offline clients who are trying to write to the same document, only the last one will be actually be written to servers when the state is changed.
So, to answer your question, there is no proper way to add/update data in Firestore when offline, to have a less memory usage. Just go online and that's it!
When my app is offline and I am adding or updating a document the memory increases. Also, showing lists of documents take longer to load. If I run the same code when the device is online the memory stays consistent as well as the speed of the activities that have lists of documents.
I'm currently doing saves like the following:
collectionRef.document(id).set(obj, SetOptions.merge());
Or for batching a couple of records:
batch.set(docRef1, obj1);
batch.set(docRef2, obj2);
batch.commit();
I had listeners for onComplete but in the accepted answer for this question, it seems to indicate that listeners are unnecessary in most situations and that you can't wait for it to complete anyway when you're offline.
In another question they indicate in code that a "Snapshot" is required to properly do online and offline saving: Offline issue with Firestore vs Firebase. But I can't find anywhere else indicating if that will make a difference. I think of Snapshots as being something you attach to a document or query when you want to be notified of changes to it and attaching a listener like that will result in a memory leak if it isn't removed.
Another part of all of this is how this slowness might affect data integrity. When I watch in the profiler in Android Studio I see that the FirestoreWorker can get to a point where it is constantly working even if I don't do anything in the app. I'm not just talking a few seconds, more like a minute. There isn't any ordering guarantee of writes when it is offline that I can find. Trying to stop and restart the app doesn't seem to have any effect on the slowness (although it will reset the memory).
So all of this leads to the question: what is the proper way to add/update data in Firestore when offline so that the app's memory doesn't grow unbounded and slow down?
Cloud Firestore uses SQLite for its persistence mechanism. So for intermittent periods of offline activity, you shouldn't have problems with performance or durability.
However, if you intend to use a Firestore database for very long periods of time, there are some things you should be aware of. Cloud Firestore was not built as an offline database, is an online database that continues to work when you're offline for short or longer periods of time. When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations.
So I suggest use this database for its online capabilities. As one of the Firebase engineers said and I quote, "It is impossible to build a slow query in Firestore". So, the performance comes from the new indexing capabilities on the backend and these optimizations don't exist when you're offline.
One more thing, if you have many offline clients who are trying to write to the same document, only the last one will be actually be written to servers when the state is changed.
So, to answer your question, there is no proper way to add/update data in Firestore when offline, to have a less memory usage. Just go online and that's it!
I have a app working offline. It is assumed that 1000+ records are created with images in each record during this period and whenever connectivity is established. What should be the approach to send all the 1000+ records to server that also handles any interruption between the network calls or API failure response.
I assume I have to send records in batches but how to handle the interruption and maintain consistency and prevent any kind of data loss.
I guess the best way here is to send each record separetely (if they are not related to each other).
If you have media attachments, sending of each record will take 2 seconds in average, if you uploading via mobile internet with speed ~2 MB/s. If you will send the large batch of records via each request, you must have stable connection for a long period.
You can send each record as multipart request, where parts are record's body and media attachments.
Also you have no need to check for internet connection, or use receiver for catching changes of connection state. You can simply use this libraries for triggering sync requests:
JobScheduler
Firebase JobDispatcher
Evernote android-job
I would suggest to use Firebase database API.
It has got nice offline/online/sync implementations.
https://firebase.google.com/docs/database/
And it is possible to read/write the data using Admin SDK for your NodeJS server:
https://firebase.google.com/docs/admin/setup
You can use divide and conquer approach means divide the task into small task and upload the data to the server.
1. take a boolean flag "isFinishData" starting with false.
2. starting upload the data on server from 0 to 100 records.
3. next record send from 100 to 200.
4. this process run until last record (1000) is not send .
5. in last record update set boolean variable true and exit from loop .
this logic would be work fine in IOS/android both.
Save your records in local Db and use ORMs for it. Use Retrofit which provide onSuccess and onFailure method for Webservice calling. To send data to server at regular interval you can use sync adapter.
1st I need to know how did you save image in local db ?
You need to create a service to catch connection status. Each time when connection is established, you submit your record as Multipart kind. You can you Retrofit/Asynctask.
Just submit 1 record per one Retrofit/Asynctask, it makes you ez to handle success/fail of each record.
You can run a single or multi retrofit/asynctask to submit one or more record, it's up to you.
If ur data has image, on server side, you have to handle process from ur server to 3rd server ( server to save image ).
This is a very broad question and it relates to Architecture, UI Experience, limitations, etc.
It seems to be a synchronization pattern where the user can interact with the data locally and offline but at some point, you'd need to synchronize the local data with server-side and vice-versa.
I believe the best place to start is with a background service (Android, not sure if there's a similar approach on iOS). Essentially, regardless of whether the Android app is running or not, the service must handle all the synchronization, interruption, and failure in the background.
If it's a local db, then you'd need to manage opening and closing the database appropriately and I'd suggest using a field to mark any sync'd records so if some records did fail, you can retry them at another point.
Also, you can convert the records to json array, then do a post request.
As for uploading images, definitely needs to be in batch if there's a lot of them but also making sure to keep track of which ones are uploaded and which ones aren't.
The one problem that you will run into if you're supporting synchronization from different devices and platforms, is you'll have conflicting data being synchronized against the backend. You'll need to handle this case otherwise, it could be very messy and most likely cause a lot of weird issues.
Hope this helps on a high level :)
To take on simple approach ,have 1 flag in your data objects [NSManagedObject] classes as sync.While creating new object / modifying an existing object change sync flag to false .
Filter data objects with sync value as false.
let unsyncedFilter = NSPredicate(format: "sync = %#", #(false))
Now you will have an array of objects which you want to sync with server.If you are sending objects one by one in requests.
On success change sync flag to true else whenever your function gets executed again on app launch/reachability status update, it will filter out unsynced data again & start synch.
As others have mentioned this is a rather broad question. A lot depends on both the architecture of the server that will receive the data as well as the architecture of the app.
If you have any control over the implementation of your backend I would recommend implementing a storage solution that allows for pausing and resuming of transfers. Both Google Cloud Storage and Amazon S3 offer a similar functionality.
The idea behind this approach is to be able to pick up the upload from where it stopped. In case of app crash or issues with internet connection you don't have to restart all from the beginning.
In your case I would still start separate uploads for each one of the records and store their upload progress.
Here you can find an example of how to use the pause / resume approach using the mobile SDK with Amazon https://aws.amazon.com/blogs/mobile/pause-and-resume-amazon-s3-transfers-using-the-aws-mobile-sdk-for-android/.
Editing adding reference to Amazon iOS SDK , http://docs.aws.amazon.com/mobile/sdkforios/developerguide/s3transfermanager.html
Best way is to break the files into chunks of 100s and upload at intervals or when app is idle.