Develop Reference

Cache HTTP response or store in database in mobile apps?

For better UX mobile apps store data on the client side (on the device) to provide immediate information when loading an app without having to wait for data from the internet and providing data even when the device is offline. Of course data is updated/fetched whenever possible later on.
I am building an app (in flutter) which is a social network/information feed like app: there are users, profiles, feed, posts etc. When the user opens the app I would like to show data that was available the last time the app ran.
My question is what is the right way to implement cache? There are two main ways I can think of, an easier/uncertain way and a more difficult/stable way, and I would like your opinion about them. I have time/resource constraints ofc. Most information is through HTTP requests, so:
The easier way: HTTP Cache interception
I would use an out-of-the-box cache plugin for my HTTP client. I think I can just cache the response for each request I make (for some time) and rely on the cached info. Images are also cached based on url. When I make a request on application load I return the cached result if there's any, and if it was a cache hit I fire the request again, so when you open the app you will see immediate information, but after a sec or so you will get the fresh data too. Usability of this solution ofc depends on how well I design my API.
Harder: Store data in a structured database.
This is the option I try to avoid, because it's more time implementing this. It could be either a SQL or document store, and I would have to implement the cache look up/save/update mechanism. Since I am just building the app, I think this would slow me down because data types/ architecture might still change. But is this the ultimate way to go with mobile side caching?
Thank you

I think the easier way is your best bet the only time i can think of that cache could be a problem is if you need critical data that has to be correct and not a old cache value, but you can avoid this problem by not caching the critical values.
Also if you use firebase it does some automatic caching which might be useful.

Data is still downloading from firebase even using firebase offline capabilities

I am using firebase offline capabilities to fetch data from cache once it load all data in local disk even if my internet is on.I successfully managed to do it but when ever I read data in my app, my firebase data download size increases. I am confused: when I get data from local cache, my data download size increases every time I am reading data in my app. If anyone knows please help me out.

Even when the data is already in the cache, the Firebase client will connect to the server and check if the cached data is up to date. This check is typically significantly smaller than downloading the actual data, but you'll still be charged for the bandwidth it consumes.
If you want to work completely offline, and don't want the client to check with the server at all, you can tell the Firebase client to goOffline when the app starts.

Sync large amounts of data between mobile app and webserver

The Setup
I have native iOS and Android apps which sync data to and from my webserver. A requirement of the apps is that they work offline so data is stored on the apps in sqlite databases.
The apps communicate with the server with a series of REST calls which send JSON from the server for the apps to store in their databases.
My Problem
The scale of this data is very large, some tables can have a million records, and the final size of the phone databases can approach 100mb.
The REST endpoints must limit their data and have to be called many times with different offsets for a whole sync to be achieved.
So I'm looking for ways to improve the efficiency of this process.
My Idea
An idea I had was to create a script which would run on the server which would create an sqlite file from the servers database, compress it and put it somewhere for the apps to download. Effectively creating a snapshot of the server's current data.
The apps would download this snapshot but still have to call their REST methods in case something had changed since the snapshot happened.
The Question
This would add another level of complexity to my webapp and I'm wondering if this is the right approach. Are there other techniques that people use when syncing large amounts of data?

This is a complex question, as the answer should depend on your constraints:
How often will data change? If it is too often, then the snapshot will get out of date really fast, thus apps will be effectively updating data a lot. Also, with the big volume of data, an application will waste CPU time on synchronization (even if user is not actively using all of that data!), or may become quickly out of sync with the server - this is especially true for iOS where Applications have very limited background capabilities (only small window, which is throttled) compared to Android apps.
Is that DB read-only? Are you sending updates to the server? If so, then you need to prepare conflict resolution techniques and cover cases, in which data is modified, but not immediately posted to the server.
You need to support cases when DB scheme changes. Effectively in your approach, you need to have multiple (initial) databases ready for different versions of your application.
Your idea is good in case there are not too many updates done to the database and regular means of download are not efficient (which is what you generally described: sending millions of records through multiple REST calls is quite a pain).
But, beware of hitting a wall: in case data changes a lot, and you are forced to update tens/hundreds of thousands of records every day, on every device, then you probably need to consider a completely different approach: one that may require your application to support only partial offline mode (for most recent/important items) or hybrid approach to data model (so live requests performed for most recent data in case user wants to edit something).

100mb is not so big. My apps have been synching many GBs at this point. If your data can be statically generated and upated , then one thing you can do is write everything to the server, (json, images, etc...) and then sync all on your local filesystem. In my case I use S3. At a select time or when the user wants to, they sync and it only pulls/updates what's changed. AWS actually has an API call called sync on a local/remote folder or bucket. A single call. I do mine custom, but essentially it's the same, check the last update date and file size locally and if it's different, you add that to the download queue.

Export data to preload into a Firebase instance in a mobile app

The use case is a dual platform mobile app for an event. There is a schedule with photos, links, bios of the speakers and talk descriptions. If all of the attendees happen to download and open the app at the same time and in the same place, they might not get the best experience -> the WiFi might slow the calls into the data server, calls into the FireBase server side will spike.
Is it possible to export a database from the server side and preload the event schedule data into the mobile app download? Upon launch the app can sync any last minute updates, as needed, with a connection and a short sync to Firebase.
If this type of architecture is not available, is there an alternative that the Firebase team would recommend?

There is no way within the Firebase Database API to preload data into the disk cache.
Two things I can think of (neither of them very nice):
Have the client read the JSON file from your app's resources and write it to the location. The end result of this will be that the data on the server stays unmodified. But it does result in each client writing the same data to the server, so the inverse of your original problem (and likely worse performing).
Have a wrapper around the Firebase API calls that loads from the JSON file and then have them later attach listeners after a random delay (to reduce the rush on the app).
As said, neither of them is very good. For both of them, you can download the JSON from the Firebase Database console.
In my experience the usage of conference apps is a lot lower than most developers/organizers imagine. It's also typically quite well spread out over the duration of the conference. So reducing the amount of data you load might be enough to make things work.

On android you can ship a sql database in the assets directory with the app and then reconcile it with the updates when the users open the app. The Firebase database is a json file. You could also ship that in the assets directory and then reconcile on first load.

Firebase offline capabilities causing memory problems

I am developing a chat app that uses firebase database to store data. The usual approach while developing a chat app is to keep the database nodes synced so that you access the messages offline. So the problem rises when I implement the firebase offline capabilities to keep the data nodes synced. Firebase suggests two required steps for accessing data offline:
Enabling disk persistance
this is enabled according to the documentation by using this line of code (in my case I add it in application class):
FirebaseDatabase.getInstance().setPersistanceEnabled(true);
and
Keeping a node synced
this is enabled by simply adding keepsynced(true) to any databasereference that you wish to keep synced, like this:
ChatNode.keepSynced(true);
What is the difference between the two?
According to the firebase team answers on this site, I deduced that:
1) (Disk persistance) stores the data on the device disk to use them when needed, and data is stored wether you write data or read data.
a) If you write data offline: data is stored on disk and is sent to database when you go online again.
b) If you read data offline: the listener that was read online and was kept in disk and stored, you will be able to read it offline from disk.
2) (keep synced true) will keep a database reference synced in 2 ways:
a) If you are also using (disk persistence) with (keep synced) you will be able to keep data synced on disk ... which seems to be the default behavior of (disk persistence).
b) If you are using (keep synced) alone then you only store to what is known as the app memory.
The problem
I did set both of the methods, but my app is now very laggy and slow and sometimes stops on its own.
The question
If all the things that I said above are true, then would this method of offline capability be a heavy load on my app?
If I kept many listeners synced and set persistence enabled, then would the disk become full of data? Should I clean the data? Is the data on disk cleaned by itself in both methods? Is data cleaned by itself from memory?
I want to avoid the lagging and slow response in my app, thanks for your help.

You are right about your assumptions. If you are using FirebaseDatabase.getInstance().setPersistenceEnabled(true); means that Firebase will create a local copy of your database which also means that every change that is made while you are offline, will be added to a queue. So, as this queue grows, local operations and application startup will slow down. So the speed depends on the dimension of that queue. But rememeber, Firebase is designed as an online database that can work for short to intermediate periods of being disconnected and not as an offline database.
Second, if are using many listeners, don't forget to remove the listener accordingly to the life-cycle of your activity like this:
databaseReference.removeEventListener(valueEventListener);