My Android app is fetching data from the web (node.js server).
The user create a list of items (usually 20-30 but it can be up to 60+). For each item I query the server to get information for this item. Once this info is fetched (per item), it won't change anymore but new records will be added as time go by (another server call not related to the previous one).
My question is about either storing this info locally (sqlite?) or fetching this info from the server every time the user asks for it (I remind you the amount of calls).
What should be my guidelines whether to store it locally or not other than "speed"?
You should read about the "offline first" principles.
To summarize, mobile users won't always have a stable internet connection (even no connection at all) and the use of your application should not be dependant on a fulltime internet access.
You should decide which data is elligible for offline storage.
It will mainly depend on what the user is supposed to access most often.
If your Items don't vary, you should persist them locally to act as a cache. Despite the fact that the data mayn't be really big, users will welcome it, as your app will need less Internet usage, which may lead to long waits, timeouts, etc.
You could make use of Retrofit to make the calls to the web service.
When it comes to persisting data locally within an Android application, you can store it in several ways.
First one, the easiest, is to use Shared Preferences. I wouldn't suggest you this time, as you're using some objects.
The second one is to use a raw SQLite database.
However, I'd avoid making SQL queries and give a try to ORM frameworks. In Android, you can find several, such as GreenDAO, ORMLite, and so on. This is the choice you should take. And believe me, initially you might find ORMs quite difficult to understand but, when you learn how do they work and the benefits they provide us, you'll love them.
Related
Is there a way to make a Firestore collection available offline on Android?
I know there's setPersistenceEnabled(true), but I think it only caches data after I first access it. There's no upfront caching.
I would like to have certain collections to be always fully local stored, so that if device is offline it can still query for data that have never been queried before. (Of course it would require internet connection at some point in time so that data can be synched)
I thought on doing a get on the whole collection at some point, and then attaching a listener so that the app gets new data. Is it a good practice? Is there anything better?
Yes, you have to query the data before it's available locally. There are no other options.
It's a good practice is it meets the requirements of your app, and you're willing to pay for the cost of that query, and all the changes to the doucments that happen over time (each changed document while the listener is attached costs 1 read).
The Setup
I have native iOS and Android apps which sync data to and from my webserver. A requirement of the apps is that they work offline so data is stored on the apps in sqlite databases.
The apps communicate with the server with a series of REST calls which send JSON from the server for the apps to store in their databases.
My Problem
The scale of this data is very large, some tables can have a million records, and the final size of the phone databases can approach 100mb.
The REST endpoints must limit their data and have to be called many times with different offsets for a whole sync to be achieved.
So I'm looking for ways to improve the efficiency of this process.
My Idea
An idea I had was to create a script which would run on the server which would create an sqlite file from the servers database, compress it and put it somewhere for the apps to download. Effectively creating a snapshot of the server's current data.
The apps would download this snapshot but still have to call their REST methods in case something had changed since the snapshot happened.
The Question
This would add another level of complexity to my webapp and I'm wondering if this is the right approach. Are there other techniques that people use when syncing large amounts of data?
This is a complex question, as the answer should depend on your constraints:
How often will data change? If it is too often, then the snapshot will get out of date really fast, thus apps will be effectively updating data a lot. Also, with the big volume of data, an application will waste CPU time on synchronization (even if user is not actively using all of that data!), or may become quickly out of sync with the server - this is especially true for iOS where Applications have very limited background capabilities (only small window, which is throttled) compared to Android apps.
Is that DB read-only? Are you sending updates to the server? If so, then you need to prepare conflict resolution techniques and cover cases, in which data is modified, but not immediately posted to the server.
You need to support cases when DB scheme changes. Effectively in your approach, you need to have multiple (initial) databases ready for different versions of your application.
Your idea is good in case there are not too many updates done to the database and regular means of download are not efficient (which is what you generally described: sending millions of records through multiple REST calls is quite a pain).
But, beware of hitting a wall: in case data changes a lot, and you are forced to update tens/hundreds of thousands of records every day, on every device, then you probably need to consider a completely different approach: one that may require your application to support only partial offline mode (for most recent/important items) or hybrid approach to data model (so live requests performed for most recent data in case user wants to edit something).
100mb is not so big. My apps have been synching many GBs at this point. If your data can be statically generated and upated , then one thing you can do is write everything to the server, (json, images, etc...) and then sync all on your local filesystem. In my case I use S3. At a select time or when the user wants to, they sync and it only pulls/updates what's changed. AWS actually has an API call called sync on a local/remote folder or bucket. A single call. I do mine custom, but essentially it's the same, check the last update date and file size locally and if it's different, you add that to the download queue.
I'm building a listview and getting it's data from Parse.com. At the moment, every time the app loads up it queries for new data from Parse.com, causing the whole listview to load.
I'd like a situation where the listview references a local datasource and only go to Parse.com if new data is available. Somewhat similar to what the instagram app does whereby when you load it up, the list view is already populated and would get updated if needed.
I have tried ParseQuery Cache policies but the behavior still stays the same. What would be the most efficient way of implementing this feature?
Thanks in advance.
Sync Adapters can help you with your your problem. It is generally used for account and cloud synchronization. But there is no limitation using it.
http://developer.android.com/training/sync-adapters/index.html
Synchronizing data between an Android device and web servers can make
your application significantly more useful and compelling for your
users. For example, transferring data to a web server makes a useful
backup, and transferring data from a server makes it available to the
user even when the device is offline. In some cases, users may find it
easier to enter and edit their data in a web interface and then have
that data available on their device, or they may want to collect data
over time and then upload it to a central storage area.
Although you can design your own system for doing data transfers in
your app, you should consider using Android's sync adapter framework.
This framework helps manage and automate data transfers, and
coordinates synchronization operations across different apps. When you
use this framework, you can take advantage of several features that
aren't available to data transfer schemes you design yourself
You can access sample project here: http://developer.android.com/shareables/training/BasicSyncAdapter.zip
Note: Sync adapters run asynchronously, so you should use them with the
expectation that they transfer data regularly and efficiently, but not
instantaneously. If you need to do real-time data transfer, you should
do it in an AsyncTask or an IntentService.
I am working on an application that will, basically, allow people to create, join and manage groups of other people. The people within the groups can also message each other.
I have been wondering which path would be better:
Keep a remote database with all the information, including messages sent to and from users. And have the app query the server every time it needs information. Even information it has seen before.
Keep a remote database with all the information, including messages sent to and from users. Also keep a local copy of the remote database and just keep it synced with the remote database. Whenever the app needs to query for information, it does a query to see if the local table is up to date. If it is not up to date, it updates the table and runs the query on the local table. This way it will keep a local copy and the app will have fast queries when there is not an update to the remote table.
What is generally done with mobile applications and remote databases?
Would it be "bad practice" if i just did number 1?
From my point of view, in most cases, the database in the mobile is just a cache of the real database, the one in the server. So, my suggestion will be to keep locally all data that you need syncing with the server. This allows you to show information even when no connection and show something to the user while the info is updated.
Also, this approach makes the local data volatile without risk, as it's stored in the server. So:
All info is in the server
With a background process (service, thread, intentservice, whatever best suits you) you sync this information with the local database
UI is always showing info from local database
Of course, this is a very general approach, and needs to be examined for each case as different situations may need different approaches.
My base response is that I would keep the data in one place and access it remotely unless there is a major reason to keep it locally. There would have to be extenuating circumstances to mandate that I keep a copy of the data locally. Just make sure your queries are accurate and concise. Don't pull over more data than you need to.However, you can have a subset of data kept locally. Items that are specific to the user (like messages), but keeping data that is not relevant just adds overhead and bloat.
I would ask a semi-theorical question about web services and client-server architecture.
I have a server with a database with about 50 tables. This server holds even one table which contains information about users associated with the clients.
Each client has associations with a subset of all data in db.
The defined architecture implies that each client (running on Android app) calling ,with a predefined frequency, a certain number of Web Services, fill the local database, creating a copy of all the tables residing on server sb, containing the only information related to the requesting client.
Furthermore, the data in db server side can change, so frequent synchronization is required.
Considering that the client application can be imaginated as a shop online application, so should be possible browsing through providers, articles, make a order,remove orders etc.
So, when I talk about fill the local database I mean store providers, articles, ...that is, all information realated to the requesting client.
Can make sense fill the local database with this information?
I think that is more reasonable call a web service only exactly when the information is required, and not store information in local db.
So, synchronization is no more required.
Tall me what you think about ? thank you.
I know NOTHING about Android development, but have tons of experience with using web services and SOA.
In my experience, especially when the client device has limited storage and processing power, all the business logic and data logic should be in the web services, and the client app used only for display and calling those services. How you implement that is something that we can't answer for you. It's different on every project.
The only exception is when you absolutely have to have the app running while not connected. In that case, be sure you know your specific requirements, and be very dilligent to only persist at the client the data that you absoutely need, and only provide the business logic that you absolutely need while disconnected.
(This helps to keep security simpler as well - lost or stolen devices are one of the highest sources of data breaches, so the less potentially sensitive data on the device the better.)
I don't know if that's helpful, but I throught I'd throw it out there.
If your application is going to work only online, then YES. You don't need to worry to store the data into the database and retrieve it later. If your requirement that needs to support both online and offline, then you have to go withe the database.
You can run a service or Async task or Handlers to invoke the Webservice from the Android Application, that way your UI thread won't be blocked.
If your response is too big, then probably you have to think about middle tier, that way you can convert the SOAP response to JSON that would be easier for the device to process the response and network transmission is faster.
I believe based on your requirements, its good enough to support only online mode... Because the data might vary if the user is not using the app for two days and logs in back and show offline data that might be irrelevant